Apple study evaluates LLM judges for personalized writing style assessment

Apple researchers tested how well LLM judges assess whether generated text matches a user's preferred writing style, generating text with four LLMs across three long-form writing tasks and comparing judge annotations against human pairwise labels. Judge quality correlated only weakly with general LLM ability on MMLU (Pearson's ρ ≤ 0.2) and varied by writing task and evaluation criteria, and paired comparisons proved more consistent than independent scoring. The strongest LLMs did not most reliably reproduce human evaluations, possibly due to exaggerated sensitivity to subtle details.