Skip to content

Add failing_scanner for deterministic scan-error testing - #1

Merged
pipmc merged 1 commit into
mainfrom
pip/add-failing-scanner
Jul 20, 2026
Merged

pipmc merged 1 commit into
mainfrom
pip/add-failing-scanner

Conversation

@pipmc

@pipmc pipmc commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

What

Adds failing_scanner — a scanner whose execute always raises — and registers it in _registry.py alongside the existing word_counter. Both scanners are now resolvable from the inspect_test_utils package.

@scanner(messages="all")
def failing_scanner() -> Scanner[Transcript]:
    async def execute(transcript: Transcript) -> Result:
        raise RuntimeError("failing_scanner: deliberate failure for testing")
    return execute

Why

Hawk's online-scanning smoke test drives a scan whose model-full fixture references word_counter (succeeds) and failing_scanner (fails), asserting exactly one non-fatal scanner_result with scan_error. word_counter already exists on main, but failing_scanner did not exist in any branch/tag/commit.

The nearest existing scanner, random_failure (on the unmerged add-random-failure-scanner branch), is probabilistic (random.random() < failure_rate), so it can't back a deterministic "exactly one scan_error" assertion. failing_scanner always fails, which is what a smoke/error-path test needs.

Notes

  • Draft: exercised end-to-end by Hawk's online-scan smoke scenario (test_online_scan) rather than a unit test here (the repo has no test_scanners.py yet). Happy to add a direct unit test if preferred.
  • Once merged, Hawk's online_scan.yaml fixture can pin the scanner package to a recent main commit that carries both word_counter and failing_scanner (it currently pins an older off-main commit that has neither the failing scanner nor the latest main state).
  • ruff check / ruff format --check clean.

🤖 Generated with Claude Code

A scanner that always raises, so downstream error handling (e.g. a scan
runner recording a per-sample scan_error) can be exercised deterministically.
The existing random_failure scanner (on the add-random-failure-scanner
branch) is probabilistic, so it can't back an exact "one scan_error"
assertion; failing_scanner always fails.

Registered in _registry.py alongside the existing word_counter, so both are
resolvable via the inspect_test_utils package.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@pipmc
pipmc requested a review from QuantumLove July 20, 2026 13:26
@pipmc pipmc self-assigned this Jul 20, 2026
@pipmc
pipmc marked this pull request as ready for review July 20, 2026 13:28

@QuantumLove QuantumLove left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Poor scanner

@pipmc
pipmc merged commit f15f30f into main Jul 20, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants