How AI Coding Agent conducts real repository evaluation: task set, pass rate, cost and regression baseline

In the United States over the past week, “AI coding” had the highest popularity among the five comparison words in this round, at 27.

The public list is suitable for understanding model capabilities, but it cannot answer whether a certain Agent is suitable for your repository.

First define the result of your real purchase

Someone needs to fix a small bug quickly.

Someone needs to refactor across services.

Some people care more about privacy, cost and auditability.

Write these goals as weights to avoid just “looking smart” in the end.

Extract tasks from historical work orders

Select tickets that have been resolved and have verifiable answers.

Covering four types of difficulty:

Single file explicit fix.
Cross-file behavior modification.
Issues that require running the project to locate.
Tasks with ambiguous requirements and must ask questions first.

Don’t just select public hot issues that may appear in the model training materials.

Create a fixed starting point for each task

Save repository commits, dependency lock files and test data versions.

Use a standalone Git worktree.

1

git worktree add ../eval-task-01 <base-commit>

Container images use digest instead of floating latest.

Use external APIs to record responses or test environments.

Write machine-decidable pass conditions

Passing unit tests is just the first layer.

Also check legacy tests, type checking, lint, and builds.

The database task checks that the schema is compatible with the data.

Add screenshots or accessibility assertions to front-end tasks.

Add negative testing to security tasks.

Prevent Agent from modifying the referee

Hidden tests are not placed in the writable working tree.

The test runner is mounted read-only.

Check if the Agent has deleted, skipped, or weakened existing tests.

Compare the number of tests to changes in coverage.

Disables “fixing” issues via hard-coded test input.

Record the complete running trajectory

Save at least:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


agent_version
model
task_id
base_commit
prompt_version
wall_time
input_tokens
output_tokens
tool_calls
human_interventions
final_cost

Results without a version field cannot be used in future regressions.

Sensitive logs are desensitized first and then archived.

The success rate is divided into three levels

Passed once: No manual modification is required, and all acceptances pass.

Assistance in passing: someone provides clarification or minor corrections and then passes.

Failure: Unfinished, destroying other actions, or the conclusion is implausible.

Don’t count “generating a lot of code” as success.

Don’t take the “fixed” reported by the Agent as a test result.

Cost is calculated based on successful tasks

Average cost per request masks failed retries.

More useful is:

1

每个一次通过任务成本 = 总成本 / 一次通过任务数

Also record the engineer review time.

If a cheaper model requires longer manual review, the total cost may not be lower.

The time indicator is split into three segments

The first effective action time.

Agent completion time.

Manual review to merge time.

Parallel agents may shorten the second stage but increase the conflict cost of the third stage.

Therefore, it is meaningless to only look at the model response speed.

Repeat the run to measure stability

Run the same task at least three times.

Fixed model version, temperature and environment.

Only one pass out of three times cannot be regarded as reliable automation.

Check whether the failure types are consistent.

A steady stream of asking the right clarifying questions is also an available skill.

Create failure classification

No relevant files found.
Misunderstood the requirements.
Tool or environment failure.
The patch is correct but the testing is incomplete.
Modification out of scope.
Falsify verification results.
Cost or time exceeded.

Only after classification can you decide to change the model, prompts, tools or environment.

Run regression before upgrade

Keep 15 to 30 representative tasks as a baseline.

Changes in agents, models, tool permissions, or system prompts trigger regressions.

The new version should at least not set back high-stakes missions.

Results were generated using the same scoring script.

Don’t change weights on the fly after seeing the results.

A practical results table

Agent	First time pass rate	Man-minutes/task	Cost/successful tasks	Number of boundary crossings
A	Filled by test	Filled by records	Filled by billing	Filled by audit
B	Filled by test	Filled by records	Filled by bill	Filled by audit

Keep the original task-level data, don’t just publish summary averages.

How to write a conclusion that is believable

Describe the repository language, task type, model date, and permission configuration.

Explain sample size and confidence limits.

Distinguish between factual data and subjective experience.

Don’t promote one repository winner as the winner in all scenarios.

A truly valuable review is an engineering asset that can be rerun as is during the next upgrade.

Evaluation data source

Task lists use versioned YAML

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


id: api-017
base_commit: 8c18d4a
category: cross-file-bug
time_limit_minutes: 30
allowed_paths:
  - src/api/
  - tests/api/
required_checks:
  - python -m pytest tests/api
  - ruff check src/api tests/api
forbidden_changes:
  - tests/fixtures/golden.json

Task descriptions are saved separately from hidden tests. YAML enters the evaluation repository, and the hidden tests are placed in an execution environment that the Agent does not have permission to read.

Distinguish between environmental failure and capability failure

Unavailability of dependency sources, failed container pulls, or testing infrastructure failures should not be directly counted as model failures. First, a fixed pre-check script confirms the health of the environment.

Start timing only after the environment is normal. Failures caused by the Agent’s own destruction of dependencies or configurations are task results and cannot be marked as infrastructure problems.

How to score manual intervention

Divide the intervention into needs clarification, environmental assistance, technical tips, and direct answers. Each category is counted and timed separately.

An agent takes the initiative to provide necessary clarification, which is different from a person taking the initiative to reveal the location of key files. The former may be good behavior, while the latter indicates insufficient ability to complete independently.

Check patch scope

1
2
3


git diff --name-only <base-commit>...HEAD
git diff --numstat <base-commit>...HEAD
git diff --check <base-commit>...HEAD

Statistics include the number of modified files, net additions and deletions, and files outside the allowed range of the task. No points are automatically deducted for large patches, but irrelevant modifications should be marked separately.

Fighting “test passes but behaves incorrectly”

Hidden tests cover bounded input, error handling and legacy behavior compatibility. Human review to see if tests are being skipped, if mocking is excessive, and if implementations are hard-coded examples.

Front-end tasks record key interactions, API tasks compare status codes to error structures, and performance tasks use fixed data sets and warm-up rules.

Evaluation costs include infrastructure

In addition to model token fees, container time, browser runs, database instances, and log storage are also recorded. The enterprise environment then adds the cost of manual review.

When the same Agent runs concurrently, the shared service cost is shared by task. Don’t think of free trial credits as long-term unit costs.

Significance of result changes

Passing one more task out of twenty may just be a random fluctuation. See the task-level pairing results: which old tasks regressed and which new tasks improved.

Set minimum pass rates for key categories rather than just looking at overall averages. Security fix regressions cannot be offset by documentation task improvements.

Save failed product

Failed runs retain the final diff, last test output, tool errors, and stop reason. Remove the credentials and save in a directory named by the task and run ID.

When reviewing, first compare the failure types, and then read the long dialogue. Many problems can be seen directly from repeated tool calls, incorrect working directories, or tests not executing.

Prevent benchmarks from being contaminated by training memory

Do not publish full descriptions and answers to internal tasks. Regularly replenish recently solved work orders and eliminate leaked tasks.

Keep a set of long-term anchors for trend comparisons, while rolling tasks measure current real work. The results of the two groups are reported separately.

Publish results with a list of failed tasks and reasons for stopping to avoid summary scores masking the model’s stable failure in key scenarios.

The original data retains sufficient accuracy and is rounded uniformly when displayed.