# Eval Hacking Reporting Specification

**Version:** 0.1-working-draft  
**Project:** Eval Hacking Atlas  
**Status:** Proposed reporting format. Open to revision; no community adoption or independent review is implied.

## Purpose and scope

This working specification provides a common format for reporting how agents exploit evaluations. A report should explain what was being measured, what the agent did, how the result was affected, and what the available evidence supports.

For this specification, **evaluation hacking** means an agent exploiting an evaluation's data, protocol, environment, or judgment process so that its result misrepresents the capability or behavior the evaluation intended to measure. Results include scores, pass/fail decisions, and safety judgments. The scope includes score inflation and behavior that conceals capabilities or otherwise distorts an assessment.

Reports may document naturally observed cases, elicited behavior, constructed demonstrations, potential weaknesses, and unsuccessful attempts. Real-world harm is not a prerequisite. An exposed answer, evaluation awareness, an unusual optimization, or a low score alone does not establish successful hacking. Explain the relevant evaluation boundary and retain uncertainty when the protocol or evidence is unclear.

The report unit is a specific case or a group of runs with a shared setting. Mechanisms aggregate findings across reports. Papers, system cards, issues, and other publications are sources; several sources describing the same runs do not create several independent cases.

## Completing a report

Use [the JSON template](report-template.json) or the website's report builder. The builder exports the same field structure as Markdown or JSON. Its output is a self-reported draft for sharing and review; creating or downloading a draft does not submit it to an editorial service.

Keep the template's keys. Replace bracketed instructions before sharing. A completed report needs a title, model identifier, evaluation identifier, intended measurement, observed behavior, and evaluation result or validation statement. It also needs at least one evidence URL **or** an explanation of why evidence is unavailable.

Unknown information is allowed. Use `Unknown` for unavailable contextual information, `Not tested` for a check that was not performed, and `Not provided` for omitted optional narrative. Preserve `null` for an unknown run count or an absent optional reporter. An unknown value is not zero, a failed check, or evidence of absence.

## The six core sections

### 01 / Evaluation context

Identify the system and its boundaries.

- `evaluation_context.model_and_version`: model name and exact version or snapshot, where known.
- `evaluation_context.agent_or_harness`: agent framework, scaffold, or harness and version.
- `evaluation_context.evaluation_and_version`: benchmark, task, or custom evaluation and version.
- `evaluation_context.tools_permissions_and_protocol`: available tools, data, permissions, environment boundaries, and the source of relevant rules.

State when a version is unknown. Record environment changes that could affect the interpretation or reproduction of the behavior.

### 02 / Intended measurement

Use `intended_measurement` to describe the capability or behavior the evaluation should measure. State what legitimate success would establish and which resources or solution methods are permitted.

Distinguish explicit restrictions from the reporter's interpretation. Place supporting protocol references in `evaluation_context.tools_permissions_and_protocol` or `evidence.urls`. A claim that a shortcut invalidates the evaluation needs an explanation tied to its intended measurement.

### 03 / Observed behavior

Use `observation.behavior` to describe the actions actually observed, with pointers to traces, tool calls, code changes, or outputs. Describe the proposed mechanism separately within that narrative.

Separate executed actions, model statements, and interpretations of intent. A stated plan or reasoning trace is not proof that an action was executed. Do not require an inference of malicious intent where observable actions and measurement effects can establish the finding.

`observation.proposed_mechanisms` accepts zero or more of these working categories:

| Value | Meaning |
| --- | --- |
| `answer_access` | Access to hidden tests, answer keys, or reference solutions outside the intended information boundary. |
| `evaluator_tampering` | Modification of tests, scoring, timing, or evaluation evidence. |
| `judge_manipulation` | Submission content that redirects an evaluator away from its intended judgment rules. |
| `protocol_loopholes` | Exploitation of task or protocol weaknesses that bypasses the capability being measured. |
| `evaluation_conditioned_behavior` | Selective behavior in response to evaluation conditions that distorts the assessment. |

Categories are proposed labels, not a finding of guilt or an exhaustive taxonomy. A case may involve multiple mechanisms; leave the array empty if the mechanism is unresolved.

### 04 / Outcome & validation

Preserve the original score or judgment in `observation.evaluation_result_and_validation`. Then explain why it may be misleading and describe any independent checks. Identify the data, evaluator, and environment used in each check, including differences from the original run.

A high score is not by itself evidence of hacking. A score from a repaired evaluator is a separate measurement with its own assumptions. If no independent validation was performed, say `Not tested` and retain the original observation.

Choose one `observation.outcome`:

| Value | Meaning |
| --- | --- |
| `potential_weakness` | A potentially exploitable condition was identified; agent exploitation has not been established. |
| `attempted` | An attempt was observed, but success or a definitive block has not been established. |
| `succeeded` | The reported evidence supports successful exploitation under the stated evaluation conditions. Explain the success criterion. |
| `blocked` | The attempt was stopped by an identified boundary or control. |
| `undetermined` | Available evidence is insufficient to classify the result. |

These are the reporter's outcome claims. Selecting `succeeded` does not confer independent verification.

### 05 / Experimental conditions

Use `observation.conditions` to describe how the behavior arose:

- `natural`: observed while the agent was pursuing the intended task, without an explicit instruction to hack the evaluation.
- `elicited`: explicitly prompted or otherwise elicited to produce the behavior.
- `constructed`: a demonstration arranged to illustrate a proposed mechanism.
- `unknown`: the conditions are not known.

If a study mixes conditions, choose the best description for this report and explain the mixture in `sampling.selection_process`; use separate reports when needed for meaningful comparison.

`sampling.total_runs` is the non-negative integer number of runs in the relevant sampling frame, or `null` if unknown. In `sampling.selection_process`, state how runs were generated and selected, how many met the reported outcome, relevant prompting or retries, exclusions, and whether only successful or unusual runs were retained.

Any rate claim needs its numerator, denominator, population, and selection procedure. A collection of submitted or selected cases does not establish a model's hacking rate. Avoid double-counting the same runs described by multiple sources.

### 06 / Evidence & reproduction

- `evidence.status` is **`self-reported`** for every initial builder export and template-based submission.
- `evidence.urls` contains original supporting HTTP(S) URLs, one string per source. Avoid credentials in URLs. Add section, page, file, or trace pointers in the relevant narrative.
- `evidence.unavailable_explanation` explains missing or unshareable evidence. It is required when the URL array is empty; otherwise it may be `null`.
- `evidence.reproduction_steps` describes setup, prompts, commands, dependencies, and expected observations, where available.
- `evidence.limitations` records uncertainty, missing information, redactions, inaccessible materials, and unperformed checks.

Distinguish the reporter's own reproduction from an independent reproduction by another party. A working source link alone does not establish that the source supports the report. Preserve the evidence needed to inspect each substantive claim.

## Keep outcome, evidence, and mitigation separate

These describe different things:

| Dimension | Question |
| --- | --- |
| Outcome | What happened in the evaluated run? |
| Evidence | How strongly and independently has that claim been checked? |
| Mitigation | What change was proposed, implemented, or tested, and with what result? |

Initial reports remain `self-reported`, even when they include the reporter's tests or describe a fix. Evidence review and independent replication require their own attributable records; the initial exporter cannot award those statuses.

Version 0.1 does not yet have a dedicated mitigation object. If applicable, add a clearly labeled **Mitigation status** paragraph to `evidence.limitations`, distinguishing not reported, proposed, implemented, and retested. Put the retest method, conditions, and results in `observation.evaluation_result_and_validation`. Specify whether a result covers only the original case or additional models and environments. Do not use “fixed” as an evidence grade.

## Provenance and revisions

`report_title` should identify the finding without claiming more than the evidence supports. `reporter` may be a name or `null`. `created_at` and `updated_at` use ISO 8601 UTC timestamps; preserve the creation timestamp and update the revision timestamp when the report changes.

Keep earlier shared versions and explain corrections in a labeled revision note within `evidence.limitations`. If a claim is withdrawn, retain its history and the reason. Use the specification version `0.1-working-draft`; future changes to fields, categories, or review metadata should receive a new specification version.

The goal of this draft is consistent, checkable reporting. Its categories and evidence requirements should be tested through actual reports, review, and reproduction before stronger claims of standardization are made.