fix: allow valid heredoc commands in sandbox audit - #3786
Conversation
|
@fancyboi999 辛苦review一下 |
fancyboi999
left a comment
There was a problem hiding this comment.
Thanks for taking this — I traced #3774, and the production change is exactly right. Falling through on ValueError instead of returning block is safe: high-risk patterns are scanned against the whole normalized command before this branch ever matters — once in _classify_command Pass 1 (sandbox_audit_middleware.py:175-178) and again at the top of _classify_single_command (:139-142) — so dropping the shlex-failure block can't let a high-risk command slip through. I confirmed it end-to-end: a heredoc that genuinely breaks shlex (python3 << 'EOF' / echo it's fine / EOF, which raises ValueError: No closing quotation) classifies as block on main and pass on this branch. The fix works.
One thing to fix before merge — the two new regression tests don't actually exercise the branch you changed.
Both test_benign_heredoc_classified_as_pass and test_heredoc_with_high_risk_pattern_still_blocks use heredocs whose quotes are balanced, so shlex.split() parses them without raising — the except ValueError block is never entered. I verified by reverting only sandbox_audit_middleware.py to main and re-running: both still pass (2 passed). They'd stay green even with the fix reverted, so they don't anchor it.
test_benign_heredoc_classified_as_pass: the apostrophe-free body parses cleanly viashlex; it classifies aspasswith or without this change.test_heredoc_with_high_risk_pattern_still_blocks: this blocks via the Pass-1 scan oncat /etc/shadow, beforeshlexis ever reached — it would block even without a heredoc, so it doesn't prove the high-risk guard survives the new fall-through path either.
To make the benign test a real regression guard, use a command that actually trips shlex — a heredoc body with a lone apostrophe:
def test_unparseable_heredoc_classified_as_pass(self):
cmd = "python3 << 'EOF'\necho it's fine\nEOF"
assert _classify_command(cmd) == "pass"That's block on main and pass here, so it fails if the fix ever regresses. For the high-risk counterpart, pair an unparseable heredoc with a high-risk token (e.g. a trailing cat /etc/shadow) so it proves the guard still fires through the new except path rather than only via Pass 1.
Logic ships; just want the tests to lock it in.
|
@fancyboi999 Thanks for your review! I have updated it, could you please take a review again when you have time? Thanks! |
fancyboi999
left a comment
There was a problem hiding this comment.
Re-reviewed — the updated tests now anchor the fix. Thanks for the quick turnaround.
test_unparseable_heredoc_classified_as_pass uses python3 << 'EOF' / echo it's fine / EOF, whose lone apostrophe makes shlex.split() raise ValueError: No closing quotation — so it genuinely drives the changed except branch. I verified by reverting only sandbox_audit_middleware.py to main: that test now fails there (block != pass) and passes on this branch, so it's a real regression guard rather than one that's green either way.
test_unparseable_heredoc_with_high_risk_pattern_still_blocks still blocks (via the Pass-1 high-risk scan on cat /etc/shadow), confirming the fall-through doesn't weaken the high-risk guard. Full file: 149 passed on this branch.
Production change is correct and the tests now lock it in. No further issues from me.
* fix: allow valid heredoc sandbox audit commands * test: cover unparseable heredoc sandbox audit path
* fix: allow valid heredoc sandbox audit commands * test: cover unparseable heredoc sandbox audit path
* fix: allow valid heredoc sandbox audit commands * test: cover unparseable heredoc sandbox audit path
Fixes #3774
Why
SandboxAuditMiddlewarewas blocking valid multiline heredoc commands such as Python image/file analysis snippets.These commands are valid bash, but
shlex.split()can raiseValueErroron heredoc or multiline input. The previous handler treated anyValueErroras suspicious and returnedblock, which caused legitimate image analysis and file inspection commands to fail.The issue and maintainer review confirmed this was over-conservative: high-risk patterns are already scanned on the normalized raw command before the
shlex.split()path, so falling through onValueErrorkeeps the primary high-risk guard in place.What changed
shlexcannot parse are no longer automatically blocked.shlex.split().Surface area
frontend/backend/applanggraph.json, or prompt changedocker/or sandboxed executionskills/backend/pyproject.tomlorfrontend/package.jsonScreenshots / Recording
N/A. This is a backend middleware classification fix with regression tests.
Bug fix verification
Test path that reproduces the bug:
backend/tests/test_sandbox_audit_middleware.py::TestClassifyCommand::test_benign_heredoc_classified_as_passbackend/tests/test_sandbox_audit_middleware.py::TestClassifyCommand::test_heredoc_with_high_risk_pattern_still_blocksValidation
From
backend/: