Skip to content

fix: allow valid heredoc commands in sandbox audit - #3786

Merged
WillemJiang merged 2 commits into
bytedance:mainfrom
Huixin615:fix/sandbox-audit-heredoc-shlex
Jun 25, 2026
Merged

fix: allow valid heredoc commands in sandbox audit#3786
WillemJiang merged 2 commits into
bytedance:mainfrom
Huixin615:fix/sandbox-audit-heredoc-shlex

Conversation

@Huixin615

Copy link
Copy Markdown
Collaborator

Fixes #3774

Why

SandboxAuditMiddleware was blocking valid multiline heredoc commands such as Python image/file analysis snippets.

These commands are valid bash, but shlex.split() can raise ValueError on heredoc or multiline input. The previous handler treated any ValueError as suspicious and returned block, which caused legitimate image analysis and file inspection commands to fail.

The issue and maintainer review confirmed this was over-conservative: high-risk patterns are already scanned on the normalized raw command before the shlex.split() path, so falling through on ValueError keeps the primary high-risk guard in place.

What changed

  • Valid heredoc / multiline commands that shlex cannot parse are no longer automatically blocked.
  • The middleware still scans the raw normalized command for high-risk patterns before attempting shlex.split().
  • Added regression tests for:
    • a benign Python heredoc command, which now passes classification
    • a heredoc command whose shell portion contains a high-risk pattern, which still blocks

Surface area

  • Frontend UI — page / component / setting / interaction under frontend/
  • Backend API — endpoint / SSE event / request-response shape under backend/app
  • Agents / LangGraph — agent node, graph wiring, langgraph.json, or prompt change
  • Sandboxdocker/ or sandboxed execution
  • Skills — change under skills/
  • Dependencies — new/upgraded entry in backend/pyproject.toml or frontend/package.json
  • Default behavior change — changes existing behavior without the user opting in
  • Docs / tests / CI only — no runtime behavior change

Screenshots / Recording

N/A. This is a backend middleware classification fix with regression tests.

Bug fix verification

Test path that reproduces the bug:

  • backend/tests/test_sandbox_audit_middleware.py::TestClassifyCommand::test_benign_heredoc_classified_as_pass
  • backend/tests/test_sandbox_audit_middleware.py::TestClassifyCommand::test_heredoc_with_high_risk_pattern_still_blocks

Validation

From backend/:

uv run pytest tests/test_sandbox_audit_middleware.py
uv run ruff check packages/harness/deerflow/agents/middlewares/sandbox_audit_middleware.py tests/test_sandbox_audit_middleware.py
uv run ruff format --check packages/harness/deerflow/agents/middlewares/sandbox_audit_middleware.py tests/test_sandbox_audit_middleware.py
Additional git validation from repo root:
git diff --check
Results:
149 passed
ruff check: all checks passed
ruff format --check: 2 files already formatted
git diff --check: no whitespace errors

@github-actions github-actions Bot added area:agents Agents, subagents, graph wiring, prompts, langgraph.json needs-validation Touches front/back contract surface; needs real-path validation risk:high High risk: backend API, agents, sandbox, auth, deps, CI size/S PR changes 20-100 lines labels Jun 25, 2026
@Huixin615

Copy link
Copy Markdown
Collaborator Author

@fancyboi999 辛苦review一下

@fancyboi999 fancyboi999 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this — I traced #3774, and the production change is exactly right. Falling through on ValueError instead of returning block is safe: high-risk patterns are scanned against the whole normalized command before this branch ever matters — once in _classify_command Pass 1 (sandbox_audit_middleware.py:175-178) and again at the top of _classify_single_command (:139-142) — so dropping the shlex-failure block can't let a high-risk command slip through. I confirmed it end-to-end: a heredoc that genuinely breaks shlex (python3 << 'EOF' / echo it's fine / EOF, which raises ValueError: No closing quotation) classifies as block on main and pass on this branch. The fix works.

One thing to fix before merge — the two new regression tests don't actually exercise the branch you changed.

Both test_benign_heredoc_classified_as_pass and test_heredoc_with_high_risk_pattern_still_blocks use heredocs whose quotes are balanced, so shlex.split() parses them without raising — the except ValueError block is never entered. I verified by reverting only sandbox_audit_middleware.py to main and re-running: both still pass (2 passed). They'd stay green even with the fix reverted, so they don't anchor it.

  • test_benign_heredoc_classified_as_pass: the apostrophe-free body parses cleanly via shlex; it classifies as pass with or without this change.
  • test_heredoc_with_high_risk_pattern_still_blocks: this blocks via the Pass-1 scan on cat /etc/shadow, before shlex is ever reached — it would block even without a heredoc, so it doesn't prove the high-risk guard survives the new fall-through path either.

To make the benign test a real regression guard, use a command that actually trips shlex — a heredoc body with a lone apostrophe:

def test_unparseable_heredoc_classified_as_pass(self):
    cmd = "python3 << 'EOF'\necho it's fine\nEOF"
    assert _classify_command(cmd) == "pass"

That's block on main and pass here, so it fails if the fix ever regresses. For the high-risk counterpart, pair an unparseable heredoc with a high-risk token (e.g. a trailing cat /etc/shadow) so it proves the guard still fires through the new except path rather than only via Pass 1.

Logic ships; just want the tests to lock it in.

@github-actions github-actions Bot added size/XS PR changes < 20 lines and removed size/S PR changes 20-100 lines labels Jun 25, 2026
@Huixin615

Copy link
Copy Markdown
Collaborator Author

@fancyboi999 Thanks for your review! I have updated it, could you please take a review again when you have time? Thanks!

@fancyboi999 fancyboi999 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed — the updated tests now anchor the fix. Thanks for the quick turnaround.

test_unparseable_heredoc_classified_as_pass uses python3 << 'EOF' / echo it's fine / EOF, whose lone apostrophe makes shlex.split() raise ValueError: No closing quotation — so it genuinely drives the changed except branch. I verified by reverting only sandbox_audit_middleware.py to main: that test now fails there (block != pass) and passes on this branch, so it's a real regression guard rather than one that's green either way.

test_unparseable_heredoc_with_high_risk_pattern_still_blocks still blocks (via the Pass-1 high-risk scan on cat /etc/shadow), confirming the fall-through doesn't weaken the high-risk guard. Full file: 149 passed on this branch.

Production change is correct and the tests now lock it in. No further issues from me.

@WillemJiang WillemJiang added this to the 2.1.0 milestone Jun 25, 2026
@WillemJiang
WillemJiang merged commit b0ce4b5 into bytedance:main Jun 25, 2026
17 checks passed
yydspanda pushed a commit to yydspanda/deer-flow that referenced this pull request Jun 28, 2026
* fix: allow valid heredoc sandbox audit commands

* test: cover unparseable heredoc sandbox audit path
marvin9551 pushed a commit to marvin9551/deer-flow that referenced this pull request Aug 21, 2026
* fix: allow valid heredoc sandbox audit commands

* test: cover unparseable heredoc sandbox audit path
jihtsan pushed a commit to jihtsan/dnx-deer-flow that referenced this pull request Aug 29, 2026
* fix: allow valid heredoc sandbox audit commands

* test: cover unparseable heredoc sandbox audit path
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agents Agents, subagents, graph wiring, prompts, langgraph.json needs-validation Touches front/back contract surface; needs real-path validation risk:high High risk: backend API, agents, sandbox, auth, deps, CI size/XS PR changes < 20 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] SandboxAuditMiddleware incorrectly blocks valid heredoc commands due to shlex.ValueError

3 participants