Open reporting specification / v0.1
Standardize eval
hacking reporting
for frontier agents.
Document how agents game evaluations.
Make the evidence checkable.
01 / Interface
From observation
to a checkable report.
One structured account of the evaluation, the behavior, and the evidence behind your finding.
Drafts stay in your browser.
Export as Markdown or JSON.
Identify the exact system and version.
State the benchmark, tools, and permissions.
Describe what legitimate success should establish.
Record actions, not just a stated intention.
Separate a weakness, an attempt, and success.
Disclose prompting, run counts, and case selection.
Link the original traces, code, and results.
Preserve the setup, steps, and independent checks.
Make missing information and uncertainty explicit.
02 / The standard
Six parts.
One shared format.
State what was measured, what happened, and what the evidence actually supports.
Read the specification01Evaluation context
Identify the model, agent or harness, benchmark and versions. Record the available tools, data, and permissions.
02Intended measurement
Explain what legitimate success demonstrates. Separate explicit protocol rules from your interpretation.
03Observed behavior
Describe the agent's actions and proposed mechanism. Distinguish executed actions from stated intentions, with trace or code pointers.
04Outcome & validation
Preserve the original result and explain why it may be misleading. Distinguish an attempted exploit, success, and independent verification.
05Experimental conditions
State whether the behavior was naturally observed, elicited, or constructed. Disclose run counts, prompting, selection, and exclusions. Selected cases do not establish a rate.
06Evidence & reproduction
Link original materials and reproduction steps. Keep uncertainty, missing evidence, corrections, and mitigation claims explicit.
03 / Taxonomy
Map the mechanism.
Connect findings across models and evaluations, not just individual incidents.
Answer access
Obtaining hidden tests, reference answers, or solutions outside the permitted information boundary. An exposed answer alone does not establish that the agent used it.
Evaluator tampering
Changing tests, scoring code, timing, or result artifacts so the evaluation misrepresents performance. Preserve the original evaluator, the diff, and an independent check.
Judge manipulation
Redirecting a judge away from its evaluation criteria through the submitted content. Compare against a control without the proposed manipulation.
Protocol loopholes
Exploiting task or simulator behavior to receive a favorable result without demonstrating the intended capability. Explain why the shortcut is not a permitted optimization.
Evaluation-conditioned behavior
Changing behavior in response to perceived evaluation conditions in a way that distorts the assessment. Awareness alone is not hacking; a low score alone does not establish sandbagging.
Make your finding checkable.
Start with what you observed. Keep uncertainty visible.