Methodology · Last reviewed Sep 2026

Benchmark methodology: how we score AI tools

How we plan to score AI tools on profession-specific tasks: rubric, task selection, run protocol, limitations and versioning. Published before any results.

Why we publish a methodology

AI tool comparisons are only useful if you can see how the scores were produced. This page sets out how we plan to score AI tools against profession-specific tasks, what the scores mean, and what they do not. It is published ahead of our first results so that the method can be criticized before any numbers exist.

Status: as of September 2026 we have not published a benchmark run. We will publish the first one only when every score is backed by saved model outputs and a written rubric. Until then, the recommendations on this site come from editorial review of vendor documentation, pricing pages and hands-on use, not from benchmark scores.

What we score

Each output is rated from 0 to 10 on three dimensions, and the composite is the simple mean of the three.

  • Accuracy: are the facts, figures and reasoning correct for the task?
  • Completeness: does the answer address every part of the prompt, including the constraints?
  • Tone: does it match the professional register the role needs, such as a clinical note versus a client email?

Cost and response time are recorded separately and are never blended into the quality score, because they depend on plan, region and load.

How tasks are chosen

Tasks are written from the workflows described on our profession pages: drafting, summarizing, analysis, research and communication. Each task has a fixed prompt, a written rubric that lists what a strong answer must contain, and a note on common failure modes. Tasks are fixed before a run starts so results cannot be tuned to a particular model.

Run protocol

  1. Every tool receives the identical prompt with default settings and no special tuning.
  2. Each task is run three times and the scores are averaged.
  3. The exact model name, version and run date are recorded for every output.
  4. Outputs are saved in full and scored against the rubric by a human reviewer.

Limitations

Small task sets cannot represent an entire profession, human scoring is subjective even with a rubric, and models change without notice. Treat any score as a snapshot for the listed date, not a permanent ranking. Scores should support your own testing on your own work, not replace it.

Versioning and data

Any change to tasks, rubrics or scoring will be recorded here with a new version number. When results are published, each run will be released as a CSV under the CC-BY-4.0 licence so others can check and reuse the data with attribution. Questions or corrections are welcome through our contact page.

Independent methodology by the AI by Career editorial team. Last reviewed Sep 2026. No paid placement or affiliate influence; see our editorial policy.