rulebook-ai, cold.

Running sm refit on a repo that had never heard of us, and publishing whatever fell out.

The first run covered a 322k-line codebase and produced mostly baselining. So we ran the opposite experiment: a small repo, picked from the wild, that had never heard of us — botingw/rulebook-ai, ~8.5k lines of Python. No coordination with the maintainer, no cherry-picking, nothing pre-cleaned. Run sm refit and publish whatever falls out.

Repository	botingw/rulebook-ai — a CLI for composable AI rule packs
Scanned scope	63 files · 8,473 lines (plus 201 Markdown files of distributed rule packs)
Language	Python
Starting grade	8 gates failing · 155 raw findings · 6 passing
Final grade	`A+ — shipshape` · 0 findings · 21 gates passing
Their test suite	56 passing at the merge-base · 81 passing at the end (25 added). Three were failing in between, from a regression of ours; see the ledger.
Remediation	22 commits · 69 files · +3,431/−1,871, on a branch we control. Not offered upstream, so none of it has been reviewed by the maintainer.
Barnacles filed (against ourselves)	4 more, all fixed in PR #332 — plus 6 in #331 from the same run

Two latent crashes

The type gates surfaced two latent crashes. Both raise at runtime given the right input, rather than being style complaints.

Crash 1 · Path / None

Single-file assistant specs did target_path / spec.filename, but filename is declared Optional[str]. A spec that omitted it raised a bare TypeError deep inside the copy strategy, naming neither the spec nor the field.

Fixed: the single-file branch now fails fast with a message naming the spec. Every spec that ships today sets a filename, so the path is not reachable through the shipped configuration — it is reachable through a custom or future one.
Crash 2 · the dict.get default that never fires

entry.get("path", "").strip("/"). The default applies when the key is absent — not when it is present and null. A manifest with an explicit path: null returned None and blew up on .strip().

Fixed: (entry.get("path") or ""). Neither the formatter nor the linters flagged it; the type checker did.

A retired CI action, and two setup blockers

github-actions-hygiene opened with 23 findings, and one of them was not hygiene at all: the workflows still called actions/upload-artifact@v3, which GitHub retired. Their pipeline had been failing on that since September 2025. We pinned the actions to SHAs and moved to v4 on our branch, which fixes it there. Their CI is still failing, because we have not offered the branch upstream — the fix exists only in our copy.

The same pass found grpcio==1.71.0 pinned exactly, which has no wheel for Python 3.14 and fails to compile from source, and a [tool.pytest] block whose addopts was a string where current pytest requires a list — the suite would not start until that was fixed. Both block a fresh checkout.

What was actually fixed — versus baselined

The equivalent table for OpenHands records mostly baselining. This run's split is different.

Genuinely fixed	Two latent crashes; 13 mypy errors; 18 pyright unknown-type errors resolved by giving `yaml.safe_load` a typed boundary rather than silencing it; 12 broken documentation links; a 111-line `create_parser()` split along the seams its own comments already marked; the retired CI action; the uninstallable pin; the config that stopped pytest booting; stubs declared for a dependency that was already required at runtime; and 25 new tests.
Suppressed	9 markers total. Seven are `pragma: allowlist secret` on the literal string `"not-needed"` — a placeholder API key in distributed template files, copied verbatim into seven packs. One `# nosec`, attached to a URL-scheme check that actually validates. One `# noqa` for reaching at a private `argparse` type. For contrast, case study #1 added 61.
Baselined	Coverage, and only coverage. `sm init` wrote an 80% threshold; the repo measures 40%. Rather than gate on a bar it has never met, the threshold records where it actually stands and `just-this-once` requires ≥80% coverage on changed lines — so every future PR has to move the number up rather than sideways.
Scoped out	The `tool_starters` and `vscode-extension` trees, excluded from six gates. These are template packs copied byte-for-byte into user projects, so the same file gets analysed once per copy — pyright reported 555 errors that were really 18, repeated. This is a judgement call: the excluded code is real code that ships to users, and it is no longer checked.
Not compromised	`strict_typing` and pyright `strict` are both still `true` and both pass, so the typing gates were satisfied rather than downgraded. `gate-dodging` and `silenced-gates` are enabled and passing.

rulebook-ai — sm scour

$ sm scour --no-cache

   ✅ myopia:dependency-risk.py         (passed)
   ✅ myopia:github-actions-hygiene     (passed)
   ✅ myopia:ambiguity-mines.py         (passed)
   ✅ myopia:string-duplication.py      (passed)
   ✅ myopia:code-sprawl                (passed)
   ✅ myopia:just-this-once.py          (passed)
   ✅ myopia:interactive-assumptions    (passed)
   ✅ overconfidence:missing-annotations.py  (passed)
   ✅ overconfidence:type-blindness.py  (passed)
   ✅ overconfidence:untested-code.py   (passed)
   ✅ overconfidence:coverage-gaps.py   (passed)
   ✅ overconfidence:dangling-references (passed)
   ✅ laziness:sloppy-formatting.py     (passed)
   ✅ laziness:repeated-code            (passed)
   ✅ laziness:dead-code.py             (passed)
   ✅ laziness:complexity-creep.py      (passed)
   ✅ laziness:debugger-artifacts       (passed)
   ✅ laziness:broken-templates.py      (passed)
   ✅ laziness:silenced-gates           (passed)
   ✅ deceptiveness:bogus-tests.py      (passed)
   ✅ deceptiveness:gate-dodging        (passed)

✨ NO SLOP DETECTED · 21 checks passed
   ⚓ hull rating: A+ — shipshape · 0 findings

The finishing board, run uncached. Twenty-one gates, including gate-dodging and silenced-gates.

Friction that was not a bug

Things that made the run slower without being defects in either codebase. They are listed because they are what the process actually cost, and none of them appear in a grade.

The repo had no environment, so gates ran against whatever was on PATH

With no virtualenv present, slop-mop resolved a Homebrew mypy rather than a project one. It reported missing yaml stubs that a project environment would have had, so the first type results did not reflect the project's own toolchain.

Building a virtualenv fixed it, and installing pip-audit and pytest-testmon into it was required before two more gates would run at all. That is three manual steps before a full board was even possible.
Creating that environment then destabilised gates that had been passing

The new virtualenv brought ruff 0.16, where the machine had 0.15. Ruff 0.16 formats python blocks inside Markdown by default, so sloppy-formatting started failing on 9 rule-pack documents that had passed minutes earlier. Nothing in the repository had changed.

Resolved by excluding the distributed packs, but the general problem stands: which tool versions a gate resolves can change the board without any code changing.
Cached gate results hid the fixes

Gate results are cached against the files they scan, not against slop-mop's own code. After fixing a false positive in slop-mop, the gate kept returning the cached failure, which reads exactly like the fix not working.

Every verification after a tool change needed --no-cache. Correct behaviour for a released tool, actively misleading while developing against one.
The refit plan went stale on every config change

Excluding a directory edits .sb_config.json, which the plan was generated from, so each subsequent sm refit --iterate opened with a warning that the plan might be stale. Since the standard remedy for several gates is a config change, the warning fired for most of the run.

Not filed as a barnacle: the warning is accurate. It is still noise that trains you to ignore it.

What went wrong

Four of the items below are defects in slop-mop. Two are mistakes we made in their code.

Our regression · a security fix that deleted a feature

A bandit finding flagged urlopen on a URL from an environment variable. The obvious fix — restrict the scheme to http/https — quietly removed a supported workflow: pointing RULEBOOK_AI_INDEX_URL at a local file for offline use, which is also how their entire community-index integration suite builds its fixtures. Three of their tests went red because of us.

Fix: file: was added back to the allowlist. The variable is operator-supplied configuration rather than attacker-controlled input, and the allowlist still excludes other schemes.
Worse · the check that proved it was “pre-existing” was wrong

We reported those three failures as pre-existing, on the strength of stashing our changes and re-running. But git stash only removes uncommitted work — the breaking change was already committed, so it survived the stash and every “baseline” run reproduced the failure. It looked convincingly like someone else’s problem.

Fix: check out the merge-base in a throwaway worktree and run there. All 56 of their tests pass at the merge-base, so both the regression and the incorrect diagnosis were ours.
Barnacle · a scanner that never ran, reported as a finding

On a checkout whose environment lacked pip-audit, dependency-risk failed with “1 security scanner(s) found issues” and a single finding reading “pip-audit found issues” — naming a vulnerability that did not exist, while concealing that nothing had been audited at all. The guard for exactly this already existed in our codebase, with a comment describing exactly this failure. It was never called from anywhere.

Fix: a scanner that fails to start is a broken environment, not a security finding. It now warns and names the install command. The first attempt at that fix was worse than the bug — it made a missing scanner render as “All security checks passed” — and review caught it before it shipped.
Barnacle · the gate that could not tell code from prose

Python subscript-then-call matches the inline-link pattern exactly, so a documented handlers[cmd](args) inside a fenced code block was reported as a broken link to a file named after the argument. After 12 genuinely broken links were fixed, this false positive was the remaining finding.

Fix: fenced blocks are skipped and inline code spans blanked. Review then found two more holes in that fix — multi-backtick spans and spans wrapping a line — both now covered by tests.
Barnacle · refit blamed the wrong gate

A targeted scour runs the requested gate and its dependencies. When a dependency failed, refit still announced the gate it was iterating and pointed at that gate’s log — which had not been rewritten, so it showed the previous run’s output. We chased a stale 13-error type log while the real failure was a one-line formatting complaint.

Fix: the failing gate and its log now come from the artifact’s own first_to_fix, which had been right the entire time.