DEVELOPER NOTES

An AI coding eval should catch the bug, not admire the answer

Build a compact evaluation set with hidden cases, independent checks, and honest first-attempt results.

Edwin··3 min·425 words
AI developmentTestingSRE
An AI coding eval should catch the bug, not admire the answer
ENNATURAL ARTICLE

A persuasive explanation can make a wrong patch feel finished. That is why I prefer coding evaluations where the application gets a vote. The agent can explain its reasoning, but the changed behavior needs independent evidence. This is a practical evaluation recipe, not a leaderboard or a claim about one model's performance.

Start from failures you care about

Pick several small tasks that represent actual work: an import boundary, a permission check, a timezone conversion, a missing empty state, and an expensive query. Store a sanitized fixture and expected behavior for each. Include a task where the correct response is to explain insufficient information rather than invent a fix.

Before running an agent, verify that the target failure exists. A regression test that already passes cannot prove the proposed change repaired it. Keep some checks hidden from the prompt so the worker cannot merely imitate the visible assertion. Hidden does not mean unfair: the expected behavior should follow from the stated requirement.

Freeze the evaluation contract

Record repository revision, environment, task text, model configuration, allowed tools, and timeout. Keep the test harness outside the files the worker may casually edit. If the task explicitly requires changing tests, retain an independent acceptance check as well. Otherwise deleting the failing assertion can become an accidental “success.”

Score dimensions separately: required behavior, unrelated regressions, permission boundaries, patch scope, and evidence quality. A single number can conceal a serious issue. A correct output obtained by exposing a secret is not an acceptable pass.

Count attempts honestly

Measure first-attempt success separately from success after feedback. Include all retries and reviewer repair in time and cost. If a human supplies the missing algorithm, record that assistance. It is still useful to know that a model becomes productive with guidance; it is misleading to call the result fully independent.

Repeat cases where nondeterminism matters and retain failures. Review false failures too: a flaky test or environment outage should not be silently counted as a reasoning defect. Categorize it and report the denominator you actually measured.

Turn a failure into a better fixture

After a real escape, add the smallest reproduction that would have caught it. Keep the set compact enough to run before changes to the model, harness, or dependency environment. Rotate some held-out cases so optimization does not collapse into memorizing one suite.

The best eval report lets another developer inspect a failing input, the patch, and the check. A nice explanation is welcome. A repeatable behavioral result is what tells us whether the assistant is making the engineering work better.

JP日本語
SubjectObjectVerbParticleTechnicalConnectorAdjectiveTime / context
CONTINUE IN MANABI KŌBŌ

Put this idea into practice

Reading Nook

Practice bilingual reading with Japanese scaffolding.

Open tool →
Guided Japanese Course

Continue with structured lessons and practice.

Open tool →
Adaptive Recall

Turn useful phrases into spaced practice.

Open tool →
RELATED NOTES
← Back to all articles
Share this note Email