Skip to content

feat(gateway): add /health/ready readiness probe backed by the database - #5166

Merged
WillemJiang merged 9 commits into
bytedance:mainfrom
Michael-Yu2978:feat/gateway-readiness
Sep 4, 2026
Merged

feat(gateway): add /health/ready readiness probe backed by the database#5166
WillemJiang merged 9 commits into
bytedance:mainfrom
Michael-Yu2978:feat/gateway-readiness

Conversation

@Michael-Yu2978

Copy link
Copy Markdown
Contributor

Why

GET /health only proves the process is up: it returns 200 even when the persistence engine cannot reach the database. Orchestrators already treat it as a readiness gate (docker-compose.yaml marks the gateway service healthy and nginx depends_on service_healthy), so a DB outage or a still-migrating Postgres leaves the stack 'healthy' while every request fails.

What changed

  • New GET /health/ready endpoint: bounded SELECT 1 against the existing persistence engine (deerflow.persistence.engine.get_engine) with a 2s timeout.
  • Response is 200 {'status': 'ready', 'database': 'ok'} when reachable, 503 {'status': 'degraded', 'database': 'unreachable'} when the probe fails, and 200 ready with database=not_configured for backend=memory (nothing to probe).
  • GET /health is unchanged (pure liveness), and /health/ready is public through the existing /health auth whitelist.
  • docker-compose.yaml gateway healthcheck now polls /health/ready so service_healthy reflects database reachability.
  • Documented both endpoints in backend/app/gateway/AGENTS.md.

Surface area

  • Backend API - new GET /health/ready endpoint under backend/app/gateway
  • Sandbox / Docker - gateway healthcheck in docker/docker-compose.yaml now gates on readiness
  • Frontend UI / Agents / Skills / Dependencies
  • Default behavior change - existing /health unchanged; the prod compose healthcheck is stricter (503 while the database is unreachable)

Validation

  • New unit tests in backend/tests/test_gateway_health.py cover ok / unreachable / not_configured probe results and the 200/503 payload mapping (6 passed).
  • app.gateway.app imports cleanly and registers both /health and /health/ready.
  • ruff check + ruff format clean.

AI assistance

Tool(s) used: Codex (coding agent)

How you used it: design, implementation, and unit tests produced with AI assistance; reviewed before commit.

  • I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.

## Why

GET /health only proves the process is up: it returns 200 even when the persistence engine cannot reach the database. Orchestrators already treat it as a readiness gate (docker-compose.yaml marks the gateway service healthy and nginx depends_on service_healthy), so a DB outage or a still-migrating Postgres leaves the stack 'healthy' while every request fails.

## What changed

- New GET /health/ready endpoint: bounded SELECT 1 against the existing persistence engine (deerflow.persistence.engine.get_engine) with a 2s timeout.

- Response is 200 {'status': 'ready', 'database': 'ok'} when reachable, 503 {'status': 'degraded', 'database': 'unreachable'} when the probe fails, and 200 ready with database=not_configured for backend=memory (nothing to probe).

- GET /health is unchanged (pure liveness), and /health/ready is public through the existing /health auth whitelist.

- docker-compose.yaml gateway healthcheck now polls /health/ready so service_healthy reflects database reachability.

- Documented both endpoints in backend/app/gateway/AGENTS.md.

## Surface area

- [x] Backend API - new GET /health/ready endpoint under backend/app/gateway

- [x] Sandbox / Docker - gateway healthcheck in docker/docker-compose.yaml now gates on readiness

- [ ] Frontend UI / Agents / Skills / Dependencies

- [x] Default behavior change - existing /health unchanged; the prod compose healthcheck is stricter (503 while the database is unreachable)

## Validation

- New unit tests in backend/tests/test_gateway_health.py cover ok / unreachable / not_configured probe results and the 200/503 payload mapping (6 passed).

- app.gateway.app imports cleanly and registers both /health and /health/ready.

- ruff check + ruff format clean.

## AI assistance

**Tool(s) used:** Codex (coding agent)

**How you used it:** design, implementation, and unit tests produced with AI assistance; reviewed before commit.

- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
@Michael-Yu2978
Michael-Yu2978 marked this pull request as ready for review September 3, 2026 14:32
@github-actions github-actions Bot added area:backend Gateway / runtime / core backend under backend/ area:docs Documentation and Markdown only area:sandbox Sandboxed execution and docker/ needs-validation Touches front/back contract surface; needs real-path validation risk:high High risk: backend API, agents, sandbox, auth, deps, CI size/M PR changes 100-300 lines labels Sep 3, 2026

@willem-bd willem-bd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Point the Helm readiness probe at /health/ready

The chart still sets the Gateway readinessProbe.httpGet.path to /health in deploy/helm/deer-flow/templates/gateway-deployment.yaml. As a result, Kubernetes continues marking the pod ready and routing traffic when the database is unreachable, even though the new endpoint is intended to prevent that failure mode. Please change only the readiness probe path to /health/ready; the liveness probe should remain on /health.

## Why

Review on bytedance#5166 (willem-bd, P1): the chart still probed /health for readiness,
so Kubernetes marked the pod ready and routed traffic while the database was
unreachable - exactly the failure mode /health/ready was added to catch.

## What changed

- deploy/helm/deer-flow/templates/gateway-deployment.yaml: readinessProbe
  httpGet.path now hits /health/ready (DB-backed, 503 while the database is
  unreachable). The liveness probe stays on /health.

## Verification

- One-line path change inside the existing readinessProbe block; git diff
  confirms only the readiness path changed (liveness untouched).

## AI assistance

**Tool(s) used:** Codex (coding agent)

**How you used it:** implemented the reviewer-requested probe path change; reviewed before commit.

- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
@Michael-Yu2978

Copy link
Copy Markdown
Contributor Author

Thanks @willem-bd — addressed in 895d6157: deploy/helm/deer-flow/templates/gateway-deployment.yaml now probes /health/ready for readiness (so Kubernetes stops routing when the database is unreachable), and the liveness probe remains on /health.

@WillemJiang
WillemJiang requested a review from zhfeng September 3, 2026 23:43
Comment thread backend/app/gateway/health.py
…Store backend

## Why

Follow-up review on bytedance#5166 (willem-bd, P1): get_engine() only represents the
ORM backend selected by `database:`. The legacy `checkpointer:` section takes
precedence for the LangGraph checkpointer and Store, so a split configuration
(a local SQLite/memory `database:` with `checkpointer.type: postgres`) could
report 200 while the PostgreSQL backend agent runs depend on was down.

## What changed

- GET /health/ready now probes both persistence halves: the ORM engine behind
  `database:` (unchanged) and the effective LangGraph checkpointer/Store
  backend resolved with the runtime's own rule (legacy `checkpointer:` config,
  otherwise derived from `database:`), for memory/sqlite/postgres backends.

- The payload gains a `checkpointer` field with the same
  ok / not_configured / unreachable vocabulary as `database`; 503 degraded is
  returned when either probe is unreachable.

- Probes are bounded by the existing 2s timeout: sqlite via aiosqlite SELECT 1
  on the resolved path, postgres via a bounded psycopg AsyncConnection SELECT 1
  on the DSN with the configured search_path. A missing driver for a configured
  backend degrades readiness (the runtime could not run either).

- Documented the two-probe semantics in the endpoint docstring and
  backend/app/gateway/AGENTS.md.

## Verification

- New tests: healthy ORM engine + unreachable legacy checkpointer backend ->
  503 degraded with database: ok / checkpointer: unreachable; checkpointer
  probe mapping for memory/sqlite(postgres missing-driver) backends; existing
  payload tests now pin the checkpointer field.

- cd backend && python -m pytest tests/test_gateway_health.py: 11 passed.

- app.gateway.app imports cleanly; ruff check + ruff format clean.

## AI assistance

**Tool(s) used:** Codex (coding agent)

**How you used it:** design, implementation, and unit tests produced with AI assistance; reviewed before commit.

- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
@github-actions github-actions Bot added size/L PR changes 300-700 lines and removed size/M PR changes 100-300 lines labels Sep 4, 2026
@Michael-Yu2978

Copy link
Copy Markdown
Contributor Author

Thanks @willem-bd — addressed in 21a70b8c. GET /health/ready now probes both persistence halves:

  • the ORM engine behind database: (unchanged, reported under database), and
  • the effective LangGraph checkpointer/Store backend resolved with the runtime's own rule (legacy checkpointer: section first, otherwise derived from database:), probed per backend: memorynot_configured, sqlite → bounded SELECT 1 on the resolved path via aiosqlite, postgres → bounded psycopg AsyncConnection SELECT 1 on the DSN with the configured search_path.

The payload now includes a checkpointer field (same ok / not_configured / unreachable vocabulary as database), and the endpoint returns 503 degraded when either probe is unreachable — so a healthy ORM engine no longer masks a down legacy checkpointer backend.

Coverage added: healthy ORM engine + unreachable legacy checkpointer backend → 503 with database: ok / checkpointer: unreachable, plus per-backend probe mapping tests. tests/test_gateway_health.py: 11 passed; ruff clean. Thread resolved.

@zhfeng zhfeng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found two readiness correctness issues that are not covered by the existing review threads.

Comment thread backend/app/gateway/health.py Outdated
Comment thread backend/app/gateway/health.py

@willem-bd willem-bd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the latest revision. I found three remaining readiness correctness issues.

Comment thread backend/app/gateway/health.py Outdated
Comment thread backend/app/gateway/health.py Outdated
Comment thread backend/app/gateway/health.py Outdated
…up checkpointer snapshot

## Why

Second round of review on bytedance#5166 (zhfeng P1/P2, willem-bd P1/P1/P2). Three
correctness issues remained in the readiness endpoint:

- The database and checkpointer probes ran sequentially, each allowed 2s, so a
  healthy response could take almost 4s - past Kubernetes' 1s default
  readinessProbe timeout and inside Docker's 3s client timeout. A slow but
  healthy backend could make every replica unready.

- The checkpointer probe re-resolved process-wide, hot-reloaded configuration
  per request, while app.state.checkpointer/store are built once from the
  startup_config snapshot in langgraph_runtime(). After a live config edit the
  endpoint could probe a backend the running gateway does not use, and a
  resolution failure was swallowed into None -> not_configured -> 200.

- The SQLite probe opened the path with aiosqlite.connect(), which creates the
  file when missing: a deleted checkpoint database was silently resurrected as
  an empty file and reported ok instead of surfacing the outage.

## What changed

- backend/app/gateway/health.py: the two probes now run concurrently beneath a
  single endpoint-wide deadline (_READINESS_DEADLINE_SECONDS=3.0) so a healthy
  response completes within one probe window (~2s), never the sum of both.
  A probe that overruns the deadline degrades the endpoint instead of hanging.

- langgraph_runtime() now records the checkpointer/Store config resolved from
  the same startup_config snapshot its checkpointer/store singletons are built
  from (app.state.checkpointer_config); /health/ready probes that snapshot and
  never re-resolves hot-reloaded config. resolve_checkpointer_config() returns
  None on resolution failure and the endpoint fails closed (503, checkpointer:
  unreachable) instead of reporting not_configured.

- The SQLite probe opens disk-backed paths with the non-creating mode=rw URI
  flag, so a missing database file stays missing and yields unreachable;
  in-memory forms (:memory:, file:...mode=memory) have nothing external to
  probe and report not_configured like the memory backend.

- Orchestrator timeouts now sit above the endpoint bound: Helm readinessProbe
  gains timeoutSeconds: 5 (Kubernetes default is 1s) and the docker-compose
  gateway healthcheck client timeout moves from 3s to 5s.

## Verification

- New regression tests: concurrent probes keep total elapsed time within one
  probe window; a probe ignoring its budget trips the endpoint deadline to 503;
  missing SQLite file stays absent and yields unreachable; in-memory SQLite
  forms map to not_configured; missing startup snapshot / config resolution
  failure fail closed to 503; resolve_checkpointer_config() raising is covered.

- cd backend && python -m pytest tests/test_gateway_health.py: 21 passed;
  tests/test_gateway_docs_toggle.py and lifespan/shutdown gateway suites pass.

- ruff check + ruff format clean on all changed files.

## AI assistance

**Tool(s) used:** Codex (coding agent)

**How you used it:** implemented the reviewer-requested concurrency/deadline, startup-snapshot probing, fail-closed resolution, and non-creating SQLite probe; reviewed before commit.

- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.

@willem-bd willem-bd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one availability issue in the current revision.

Comment thread backend/app/gateway/health.py
…strict gate

## Why

Review on bytedance#5166 (willem-bd, P1): every request to /health/ready opened a new
PostgreSQL connection in _probe_postgres_backend, outside both the ORM pool and
the runtime checkpointer pool. The route is public through the /health auth
prefix and nginx proxies /health/*, so concurrent unauthenticated requests
could create an unbounded number of connections (each held for up to two
seconds), exhaust PostgreSQL max_connections, and take down both normal
traffic and the readiness probe itself.

## What changed

- backend/app/gateway/health.py: connection-opening checkpointer probes
  (sqlite connect, postgres AsyncConnection.connect) now run inside a strict
  per-process gate - an asyncio.Lock cached per running event loop - so at
  most one probe connection can be in flight per worker process. Requests
  that queue behind the gate are still shed by the existing endpoint-wide
  deadline, so a flood cannot pile up new connections or open files.

- Memory and unknown-backend decisions stay outside the gate; payload and
  probe semantics are unchanged. The serialization is documented in the
  module docstring and backend/app/gateway/AGENTS.md.

## Verification

- New regression test: 8 concurrent readiness_payload() requests against an
  instrumented sqlite probe assert the maximum number of in-flight probe
  connections is 1 while every request still returns 200.

- cd backend && python -m pytest tests/test_gateway_health.py: 22 passed;
  tests/test_gateway_docs_toggle.py and tests/test_gateway_lifespan_shutdown.py
  also pass on the merged main head.

- ruff check + ruff format clean on all changed files.

## AI assistance

**Tool(s) used:** Codex (coding agent)

**How you used it:** implemented the reviewer-requested strict concurrency bound for the public readiness probe; reviewed before commit.

- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
@WillemJiang WillemJiang added this to the 2.1.0 milestone Sep 4, 2026
@WillemJiang
WillemJiang merged commit 4791e94 into bytedance:main Sep 4, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:backend Gateway / runtime / core backend under backend/ area:docs Documentation and Markdown only area:sandbox Sandboxed execution and docker/ needs-validation Touches front/back contract surface; needs real-path validation risk:high High risk: backend API, agents, sandbox, auth, deps, CI size/L PR changes 300-700 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants