How the Benchmark runs
The Benchmark measures whether a model, given an ordinary writing request from an organization, describes people with dignity, and how much that changes when a standard is loaded as a system prompt. It doesn't measure general writing quality, factual accuracy or safety. This page is a preview; the full methodology page is templated from each run's manifest.
This run
- Run
- October 2026 baseline, completed Oct 20, 2026
- Standard
- v0.1.1, commit 432ddd4
- Models
- 15
- Requests
- 45, all fictional, across nine sectors and four difficulty levels
- Conditions
- Without guidance, Human-centered guidance, Trauma-informed guidance
- Samples
- 2 per request per condition, temperature 0 where the provider honors it
- Drafts scored
- 4,050
Judges
Three judges from three labs score every draft against each practice the request tests: flagged or not, with a quoted span as evidence. Two of three decide. Judges never see the model name or the condition, and no judge is a model on the roster.
| Seat | Judge | Served as | Same-lab contestants | Flag ratio, own lab / others |
|---|---|---|---|---|
| A | Claude Opus 4.6Anthropic | claude-opus-4-6Identity check passed | Claude Opus 5.5, Claude Sonnet 5.5 | 1.07 / 1.32 |
| B | GPT-5.5OpenAI | gpt-5.5-2026-04-23Identity check passed | GPT-5.6 Luna (free-ChatGPT default), GPT-6 Astra, GPT-6 Luna, GPT-6.1 Sol | 0.75 / 1.19 |
| C | Muse Spark 1.3Meta | meta/muse-spark-1.3Identity check passed | Muse Wonder | 0.76 / 0.65 |
Flag ratio compares how often a judge flags drafts against how often the other two judges flag the same drafts. Below 1 on its own lab's drafts means the judge was gentler than its peers there. Panel agreement (Fleiss kappa) was 0.625 across 7,650 practice ratings; the judges agreed unanimously on 89.7%. Muse Spark 1.3 runs through a router pinned to Meta's own servers, a disclosed exception until Meta's direct API serves it.
Intervals and verdicts
- Single rates carry Wilson 95% intervals. Changes carry prompt-clustered bootstrap 95% intervals, resampling requests.
- A change counts as improved or slipped only when its interval clears zero and it's at least 5 points with guidance or 10 without. Otherwise it reads as no detectable change.
- Lab pages check each model's guided interval against a 10% line: holds, falls short, or can't tell yet.
- Generation verdicts come only from a successor and predecessor run in the same pass.
- Models tested within 30 days of their snapshot date are labeled first look.
Credits
Table design inspired by EQ-Bench, by Sam Paech. Judges were chosen with Judgemark v4, also by Sam Paech. Benchmark engineered by MKTNG.