What a benchmark score actually measures: leakage, shortcuts and extraction effects, on public data. 13 projects, each with its code on GitHub.
Twenty measurements of a local coder model: where model size pays, when self-debugging stops helping, what model-written tests actually catch, and how much of a benchmark score is the prompt.
The same generations score 0% or 85.4% on HumanEval depending only on how the code is extracted from the model output.
MBPP gives each problem three assert statements. Measuring how much wrong code three asserts accept.
51.3% of SWE-bench Lite never names the file you must fix, so a single score cannot separate retrieval failure from reasoning failure.
Cross-split overlap is smaller than assumed, but every exact duplicate inside the test set carries conflicting labels.
The weaker model is enough 53.6% of the time - and a character count predicts human preference 61.7%, on 57,477 Arena votes
Four ways a summarisation faithfulness checker looks better than it is — measured on 10,066 human-labelled summaries, and on a 14B judge that invents its own evidence in 6% of its flags.
An exact bound on how well a fact-verification model can score while ignoring the evidence: 0.502 on VitaminC, 0.999 on FEVER
The ECtHR outcome benchmark scores 0.859 micro-F1 if you ignore the facts and repeat the complaint; reading them tops out at 0.309
7.6% of FinQA questions require a number that appears nowhere in the filing they were written against
An 8k context window sees 41% of CUAD's answers - and the 41% is almost entirely the four fields on the first page
Why you cannot parse an age floor out of clinical trial eligibility prose: 89.6% of apparent registry contradictions are the checker's fault
9.2% of BBC Urdu articles are typed with Arabic letters standing in for Urdu ones - and Unicode NFC will not merge them