fix(reliability): persist longer under sustained agent work - #9538
Open
Sphinx-8 wants to merge 1 commit into
Open
fix(reliability): persist longer under sustained agent work#9538Sphinx-8 wants to merge 1 commit into
Sphinx-8 wants to merge 1 commit into
Conversation
Three failure modes surfaced after ~18h of continuous use: 1. 410 window too short — tabNotFoundResponse only answered 410 browser_restarted for 5 minutes after a browser restart. An agent that pauses longer than that (or waits out a slow relaunch) hits a plain 404 on its dead tab and treats the tab as permanently gone. Widen the window to 30 minutes and name the constant. 2. Memory-pressure restart was idle-only — the 30s interval bailed out when any session was open, so a browser tree leaking past BROWSER_RSS_RESTART_THRESHOLD_MB was never restarted until the session went idle, risking OOM on long work. Now the browser-tree RSS check runs in both phases; in the active phase sessions are closed first so clients get a clean 503 session_expired and recreate. Native-growth restart stays idle-only (its baseline is only meaningful within one launch cycle). 3. Crashed tabs waited out the full idle timeout — a page whose content process crashed held its slot for the full TAB_INACTIVITY_MS. Reap crashed tabs after a short quiet period (min(60s, TAB_INACTIVITY_MS)), long enough for the in-flight call that surfaced the crash to finish. Tests: browserRestartWindow.test.js (window + crashed-reaper logic), browserRestart410.test.js (real server: stop browser, dead tab -> 410 browser_restarted, garbage ID -> 404), memoryPressure.test.js updated for the active-phase semantics.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three failure modes observed after ~18h of continuous single-user agent use (DGX Spark, systemd user service, one persistent tab):
1. 410 "browser_restarted" window too short
tabNotFoundResponseonly answered 410browser_restartedfor 5 minutes after a browser restart. Any client that pauses longer than that (agents routinely do — or waits out a slow relaunch) hits a plain 404 on its dead tab ID and treats the tab as permanently gone instead of recreating it.Widened the window to 30 minutes and named it
BROWSER_RESTARTED_WINDOW_MS.2. Memory-pressure restart was idle-only
The 30s interval bailed out when any session was open, so a Camoufox process tree leaking past
BROWSER_RSS_RESTART_THRESHOLD_MB(default 1500MB) was never restarted while a session lived — the exact "long work" scenario — and could OOM the host first.Now:
session_expired(retryable: true) and the client recreates its session;closeBrowserFully.3. Crashed tabs waited out the full idle timeout
A page whose content process crashed (
page.crash→tabState.crashed) kept holding its registered slot for the fullTAB_INACTIVITY_MS(5 min). The reaper now reaps crashed tabs after a short quiet period —min(60s, TAB_INACTIVITY_MS)— long enough for the in-flight call that surfaced the crash to finish and retry on a new tab.Tests
tests/unit/browserRestartWindow.test.js(new) — 410 window boundaries + crashed-reaper decision logic (pure, no browser).tests/unit/browserRestart410.test.js(new) — real server + Camoufox: create tab →POST /stop→ dead tab ID answers 410browser_restarted, garbage ID answers 404.tests/unit/memoryPressure.test.js— mirrored decision function updated for active-phase semantics; 3 expectation changes..gitignore— ignore.worktrees/(local delegation worktrees were being double-discovered by jest).Full local run: unit 1621 passed, e2e 74 passed, plugins 43 passed.
No route changes, so
openapi.jsonis untouched (openapi coverage test passes).