Overview
Most model leaderboards measure reasoning, coding, or trivia. Fiction Eval measures something those boards ignore: how well a model actually writes fiction — and it refuses to compress the answer into one number, because fiction ability is genre-dependent. A model that produces sharp, dry thrillers can be mediocre at cozy small-town warmth, and a blended average hides exactly the thing a reader wants to know.
How it works
- Genre-by-genre scoring — models are prompted with matched briefs across genres and scored per genre, so the board reads like a skills matrix rather than a ranking.
- Re-run on every launch — when a major model ships, the eval re-runs and the board updates. Standing infrastructure, not a one-off blog post.
- Rubric-driven judging — outputs are scored against genre-specific rubrics (pacing, voice, trope handling) rather than a generic "quality" prompt, which keeps comparisons stable across runs.
Stack
Next.js front end over a generation-and-judging pipeline. Each eval run is stored so the board can show movement over time as models and versions change.