Case Studies.

Method

Every run follows the same five steps.

1. Pick a repository cold. No coordination with maintainers and no pre-cleaning. Both runs so far are published; with two of them, that is a description of what happened, not evidence of a policy.
2. Record the starting board. sm scour before anything is touched, artifact committed.
3. Run sm refit gate by gate. Fix what can be fixed. Where the result is to accept the existing state instead, record it as a baseline.
4. Record the finishing board, and separate the fixes from the baselines in a table.
5. File tool friction against our own repo as a barnacle, and publish the count. See below.

The reason step 5 exists: a green board proves nothing on its own. It can mean the repo was already in good shape, or that we disabled gates and lowered thresholds until the screen turned green — and those look identical from the outside. So every study carries a fixed-versus-baselined table, a suppression count, and the anti-gaming meta-gates left switched on.

The register

Two runs so far, at opposite ends of the size range, most recent first.

Barnacles · what each run cost us

A barnacle is friction in slop-mop itself — a false positive, a misleading message, a gate that fails for the wrong reason. We file them against our own repo and publish the count per run.

• rulebook-ai · 10 barnacles · #331, #332

  The worst was a security scanner that never ran being reported as a finding: on a checkout without pip-audit, the gate announced “1 security scanner(s) found issues” — naming a vulnerability that did not exist while concealing that nothing had been audited. The guard for exactly that already existed in our code and was never called from anywhere.

  Also: a gate that could not tell code from prose, refit blaming the wrong gate, and two path bugs. All ten fixed. Our first attempt at fixing the scanner bug was worse than the bug, and review caught it.

• OpenHands · 2 barnacles · #263, #264

  refit reformatted 94 applied database migrations — historical records that must never be rewritten — and separately re-formatted 417 files in a repo that already pinned its own formatter, which then disagreed and demanded the work be redone.

  One fixed the next day; #263 is still open at the time of writing. The lower count is not a better result. This run drove fewer gates to green, so there were fewer opportunities to hit a defect — though we have not measured that, and two runs cannot separate the two explanations.

Twelve barnacles across two runs. We have no baseline to compare that against — no other run, and no equivalent figure published by comparable tools — so we are reporting the count, not characterising it as good or bad.

The loop · runs feed back into the tool

Each run is also a test of slop-mop. When the tool gets in the way, that friction is filed against our own repo as a barnacle and fixed there, so the next run uses a tool that no longer has that defect. The table records what that has actually cost so far.

The intended effect of that loop is that later runs hit less friction. These two runs do not show it. The count went from 2 in June to 10 in August, and the remediation churn went up by roughly a factor of seven.

We can think of three explanations and cannot currently separate them. The June run stopped at a baseline with a number of gates disabled or scoped out, while the August run drove all 21 gates to green — more gates exercised is more surface on which to hit a defect. The two repositories are different enough that the runs may not be comparable. And the August run was more diligent about filing friction rather than working around it, which would raise the count without the tool being any worse.

That last one matters most: a barnacle count measures friction noticed and recorded, not defects present. It can rise because we looked harder. A third run, or a repeat run against a repository already onboarded, would say more than these two do — until then the trend line is one segment pointing the wrong way, and we are not going to describe that as the process working.