How the Benchmark runs
The Benchmark measures whether a model, given an ordinary writing request from an organization, describes people with dignity, and how much that changes when a standard is loaded as a system prompt. It doesn't measure general writing quality, factual accuracy or safety. This page is a preview; the full methodology page is templated from each run's manifest.
This run
- Run
- November 2026, completed Nov 1, 2026
- Standard
- v0.1.1, commit 432ddd4
- Models
- 16
- Requests
- 45, all fictional, across nine sectors and four difficulty levels
- Conditions
- Without guidance, Human-centered guidance, Trauma-informed guidance
- Samples
- 2 per request per condition, temperature 0 where the provider honors it
- Drafts scored
- 4,320
Judges
Three judges from three labs score every draft against each practice the request tests: flagged or not, with a quoted span as evidence. Two of three decide. Judges never see the model name or the condition, and no judge is a model on the roster.
| Seat | Judge | Served as | Same-lab contestants | Flag ratio, own lab / others |
|---|---|---|---|---|
| A | Claude Opus 4.6Anthropic | claude-opus-4-6Identity check passed | Claude Opus 5.5, Claude Sonnet 5.5 | 0.99 / 1.36 |
| B | GPT-5.5OpenAI | gpt-5.5-2026-04-23Identity check passed | GPT-6 Astra, GPT-6 Luna (free-ChatGPT default), GPT-6.1 Sol, GPT-6.1 Terra | 0.78 / 1.14 |
| C | Muse Spark 1.3Meta | muse-spark-1.3Identity check passed | Muse Wonder | 0.67 / 0.66 |
Flag ratio compares how often a judge flags drafts against how often the other two judges flag the same drafts. Below 1 on its own lab's drafts means the judge was gentler than its peers there. Panel agreement (Fleiss kappa) was 0.639 across 8,160 practice ratings; the judges agreed unanimously on 90.3%. Muse Spark 1.3 now runs on Meta’s own API.
What changed since October
- The standard and graders are unchanged from October (v0.1.1).
- OpenAI changed its free ChatGPT default; the roster followed.
- New models: Kimi K3.1 Pro, run beside Kimi K3, which retires next month, and GPT-6.1 Terra. Both Kimi models run at their provider’s default reasoning setting, as shipped, and their reasoning tokens are reported beside their scores.
- Kimi K3.1 Pro and GPT-6.1 Terra don’t accept a temperature setting from their provider, so they ran at the provider’s default.
- Muse Spark 1.3, our third judge, moved from a router pinned to Meta’s servers to Meta’s own API. Before the switch we rescored 150 October drafts on both routes: the direct route matched October’s verdicts on 98.97% of practice ratings, the router on 98.63%.
- 9 DeepSeek drafts were regenerated after the original host returned no response; that host was excluded.
- Open-weight models like DeepSeek, GLM, Kimi and Nemotron were served by 9 to 19 providers each. From December, each model runs on one fixed provider.
- The human voice request set rotated to a new 15 requests, so human voice results compare within this month only.
- The judge calibration check is unchanged.
Intervals and verdicts
- Single rates carry Wilson 95% intervals. Changes carry prompt-clustered bootstrap 95% intervals, resampling requests.
- A change counts as improved or slipped only when its interval clears zero and it's at least 5 points with guidance or 10 without. Otherwise it reads as no detectable change.
- Lab pages check each model's guided interval against a 10% line: holds, falls short, or can't tell yet.
- Generation verdicts come only from a successor and predecessor run in the same pass.
- Models tested within 30 days of their snapshot date are labeled first look.
Credits
Table design inspired by EQ-Bench, by Sam Paech. Judges were chosen with Judgemark v4, also by Sam Paech. Benchmark engineered by MKTNG.