Critical Gate 2.7.2 Evidence Report: Is Diff Integrity Useful for AI Agent Development?

Critical Gate 2.7.2 is not interesting because it adds one dramatic feature.

It is interesting because the project is now being shaped by evidence from real AI-agent development loops.

That matters more than a feature checklist. A diff integrity gate for AI-generated code has to earn trust in the same place it will be used: inside actual repositories, after an agent has made a change, when the final diff has to be judged against the task that produced it.

So this post is not a launch announcement. It is a status report.

The question is simple:

Does Critical Gate appear useful in real agentic development work?

The short answer is yes, with an important constraint:

Critical Gate has enough evidence to show that the product is useful as a repository-aware diff integrity gate in my current dogfood projects. It does not yet have enough evidence to claim broad proof across many teams, ecosystems, and workflows.

That distinction is important. The point of the evidence phase is not to create marketing confidence. It is to make the package better.

What Critical Gate Is Measuring

Critical Gate is not trying to be a generic AI code reviewer.

It checks whether an agent-produced diff is acceptable for the task and repository around it. The focus is not style commentary. The focus is diff integrity:

• Did the agent stay inside the requested scope?
• Did it touch unrelated files?
• Did it add dependencies without justification?
• Did it weaken tests?
• Did it change public API surface silently?
• Did it introduce config drift, hardcoded paths, secrets, or environment leaks?
• Did it ignore repository conventions or known companion files?

The tool is deterministic-first. It starts with the task intent, the git diff, package metadata, file roles, repository history, and detector-specific evidence. Optional model interpretation can help explain ambiguous findings, but the model is not the detector of record.

The 2.7.x line has been mostly about calibration: reducing false positives, improving expected companion behavior, making package-upgrade runs quieter, and turning real dogfood failures into fixture-backed package improvements.

The Current Evidence Snapshot

The latest aggregated dogfood summary covers three real repositories:

• critical-components
• criticaldeveloper-blog
• criticaldeveloper-ft

Across those projects, the current evidence set contains:

Metric	Count
Labeled reports	54
Useful findings	108
False-positive finding instances	128
Missed findings recorded	0
Fixture-needed reports	17
Fixture-created reports	16
Missed-finding reviews captured	12
Repair attempts captured	0

The controlled evaluation corpus now has 34 cases and currently passes at 100% precision and 100% recall.

Those numbers need careful interpretation.

The good news is that Critical Gate is producing many useful signals in real agentic work. It has found risky or review-worthy changes around scope, blast radius, expected companions, repository intelligence, dependencies, intent coverage, and rewrites.

The second good sign is that false positives are not being ignored. They are being labeled, grouped, and promoted into fixtures when the pattern is general enough.

The unresolved part is repair-loop evidence. Right now the aggregate has zero captured repair attempts. That means the project has evidence that Critical Gate detects and calibrates, but not enough structured proof yet that it reliably improves automated repair loops end to end.

Why False Positives Are Not Automatically Bad

Seeing 128 false-positive finding instances can look bad at first.

It would be bad if those findings were being hidden, dismissed, or shipped as permanent noise.

But in this phase, false positives are part of the development loop. The important question is what happens after they appear.

In the current evidence process, a false positive can become:

• a label sidecar beside the generated report;
• an aggregate count in the dogfood summary;
• a generalized regression fixture;
• a detector or policy calibration change;
• a cleaner rerun in a later package version.

That loop is what makes the evidence valuable.

For example, package-only Critical Gate upgrades were producing noise in earlier versions. The tool saw package and lockfile changes and treated some normal upgrade diffs as suspicious. That was useful evidence, but it was not useful product behavior. The fix was to make explicit package upgrade diffs quieter and then replay the same style of upgrade across the dogfood repositories.

By 2.7.1 and 2.7.2, those package-upgrade replays were clean in the relevant projects. That is exactly the kind of improvement dogfood should create.

Another example came from article and UI styling work. A small responsive stylesheet fix for an article hero title overflow triggered scope and expected-companion noise because historical blog files had often changed together. The lesson was general: selector-local visual fixes should not be forced to touch route, listing, or template companions unless the diff changes markup, routing, imports, exports, or data hooks.

That became a package fixture instead of a blog-specific exception.

This is the standard I want for Critical Gate improvements:

Do not tune the tool for my personal workflow.
Tune it only when the evidence describes a pattern that other repositories can hit too.

What The Evidence Says Critical Gate Is Good At

The strongest current evidence is around diff-shape awareness.

Critical Gate is useful when an AI agent produces a patch that looks plausible, but the file set or change pattern deserves scrutiny. In real dogfood runs, the useful findings have clustered around:

• broad changes during small tasks;
• unexpected clusters across unrelated files;
• repository-intelligence observations from historical co-change patterns;
• package and lockfile behavior;
• expected support-file relationships;
• style and UI presentation changes that needed better calibration;
• agent workflow and documentation changes that touched more than one surface.

This is the category where Critical Gate adds the most value today. It does not replace tests, type checking, linters, or human review. It answers a different question:

Does this diff look like the task it claims to complete?

That question is especially important for AI agents because agents can produce coherent code while still widening the blast radius.

What The Evidence Does Not Prove Yet

The evidence does not yet prove that Critical Gate is broadly reliable across the whole software ecosystem.

The current dogfood repositories are real projects, but they are still my projects. They share my workflow, my agent instructions, and my tolerance for specific kinds of evidence collection.

The evidence also does not yet prove repair-loop value. A repair-loop proof would look like this:

1. Critical Gate reports a finding.
2. Codex or another agent receives the repair guidance.
3. The agent changes the diff.
4. Critical Gate reruns.
5. The finding disappears.
6. The repair stays inside the original task scope.
7. A human accepts the result as useful.

That is the next important evidence class.

Today, the project can say:

Critical Gate is producing useful evidence-backed findings in real AI-agent development loops, and false positives are being converted into generalized fixtures and package improvements.

It should not yet say:

Critical Gate is proven across all teams and repositories.

That would be overclaiming.

Why 2.7.2 Is A Better Tool Than Earlier Versions

The latest version is better mostly because it is quieter in places where the earlier versions were too eager.

Recent calibration improved:

• package-only Critical Gate upgrade diffs;
• package and lockfile companion behavior;
• focused UI presentation tasks;
• selector-local stylesheet fixes;
• historical expected companion noise;
• repository-intelligence observations for explicitly named UI surfaces.

That matters because a gate that reports too much becomes another thing developers ignore. The product only works if the signal remains reviewable.

The goal is not to make Critical Gate silent. The goal is to make it calibrated:

• high-confidence risks should be visible;
• ambiguous repository-intelligence findings should usually start as observations;
• false positives should become fixtures;
• clean reruns should be documented;
• repo-specific policies should not replace package-level improvements when the pattern is general.

The Most Useful Pattern So Far

The most useful pattern is not a single detector.

It is the evidence loop:

real dogfood run
-> generated report
-> human label sidecar
-> aggregate summary
-> fixture candidate
-> detector calibration
-> replay on real repositories

This has changed how I think about developer-tool quality.

It is not enough to say “the detector works” because a unit test passes. It is also not enough to say “the tool feels useful” because a report looks convincing.

The useful middle ground is labeled evidence:

• Was the finding useful?
• Was it a false positive?
• Did the tool miss something the human noticed?
• Should this become a fixture?
• Did a later version reduce the noise?
• Did the fix generalize beyond one repository?

That evidence loop is becoming part of Critical Gate’s product design.

What Comes Next

The next phase is not simply “collect more reports.”

More reports help, but only if they improve the package.

The immediate next steps are:

1. Close or obsolete the last unresolved fixture-needed reports.
2. Add structured repair-loop evidence capture.
3. Run controlled repair-loop scenarios.
4. Require missed-finding review discipline across all dogfood repos, not only one.
5. Add more true-positive controlled scenarios: bad diffs that should be caught, not only clean and false-positive regressions.
6. Test Critical Gate outside my own repository style.

The repair-loop part is especially important. If Critical Gate can not only find risk but also guide an AI agent into a smaller, safer repaired diff, then the value becomes much easier to explain.

My Current Confidence Level

Internally, I would call Critical Gate useful.

The evidence is strong enough to keep investing. The tool has already caught meaningful patterns, and the calibration work has made later versions cleaner.

Publicly, I would describe the evidence more carefully:

Critical Gate has early real-repository dogfood evidence showing utility in AI-agent development workflows. It is not yet a broad benchmark. It is a growing proof trail.

That is the right place to be.

A tool like this should not ask for trust before it has evidence. It should produce evidence, absorb the uncomfortable parts of that evidence, and get better.

That is what Critical Gate 2.7.2 represents.

Not final proof.

Proof under construction.