feat(gateway): add /health/ready readiness probe backed by the database - #5166
Conversation
## Why
GET /health only proves the process is up: it returns 200 even when the persistence engine cannot reach the database. Orchestrators already treat it as a readiness gate (docker-compose.yaml marks the gateway service healthy and nginx depends_on service_healthy), so a DB outage or a still-migrating Postgres leaves the stack 'healthy' while every request fails.
## What changed
- New GET /health/ready endpoint: bounded SELECT 1 against the existing persistence engine (deerflow.persistence.engine.get_engine) with a 2s timeout.
- Response is 200 {'status': 'ready', 'database': 'ok'} when reachable, 503 {'status': 'degraded', 'database': 'unreachable'} when the probe fails, and 200 ready with database=not_configured for backend=memory (nothing to probe).
- GET /health is unchanged (pure liveness), and /health/ready is public through the existing /health auth whitelist.
- docker-compose.yaml gateway healthcheck now polls /health/ready so service_healthy reflects database reachability.
- Documented both endpoints in backend/app/gateway/AGENTS.md.
## Surface area
- [x] Backend API - new GET /health/ready endpoint under backend/app/gateway
- [x] Sandbox / Docker - gateway healthcheck in docker/docker-compose.yaml now gates on readiness
- [ ] Frontend UI / Agents / Skills / Dependencies
- [x] Default behavior change - existing /health unchanged; the prod compose healthcheck is stricter (503 while the database is unreachable)
## Validation
- New unit tests in backend/tests/test_gateway_health.py cover ok / unreachable / not_configured probe results and the 200/503 payload mapping (6 passed).
- app.gateway.app imports cleanly and registers both /health and /health/ready.
- ruff check + ruff format clean.
## AI assistance
**Tool(s) used:** Codex (coding agent)
**How you used it:** design, implementation, and unit tests produced with AI assistance; reviewed before commit.
- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
willem-bd
left a comment
There was a problem hiding this comment.
[P1] Point the Helm readiness probe at /health/ready
The chart still sets the Gateway readinessProbe.httpGet.path to /health in deploy/helm/deer-flow/templates/gateway-deployment.yaml. As a result, Kubernetes continues marking the pod ready and routing traffic when the database is unreachable, even though the new endpoint is intended to prevent that failure mode. Please change only the readiness probe path to /health/ready; the liveness probe should remain on /health.
## Why Review on bytedance#5166 (willem-bd, P1): the chart still probed /health for readiness, so Kubernetes marked the pod ready and routed traffic while the database was unreachable - exactly the failure mode /health/ready was added to catch. ## What changed - deploy/helm/deer-flow/templates/gateway-deployment.yaml: readinessProbe httpGet.path now hits /health/ready (DB-backed, 503 while the database is unreachable). The liveness probe stays on /health. ## Verification - One-line path change inside the existing readinessProbe block; git diff confirms only the readiness path changed (liveness untouched). ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** implemented the reviewer-requested probe path change; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
|
Thanks @willem-bd — addressed in |
…Store backend ## Why Follow-up review on bytedance#5166 (willem-bd, P1): get_engine() only represents the ORM backend selected by `database:`. The legacy `checkpointer:` section takes precedence for the LangGraph checkpointer and Store, so a split configuration (a local SQLite/memory `database:` with `checkpointer.type: postgres`) could report 200 while the PostgreSQL backend agent runs depend on was down. ## What changed - GET /health/ready now probes both persistence halves: the ORM engine behind `database:` (unchanged) and the effective LangGraph checkpointer/Store backend resolved with the runtime's own rule (legacy `checkpointer:` config, otherwise derived from `database:`), for memory/sqlite/postgres backends. - The payload gains a `checkpointer` field with the same ok / not_configured / unreachable vocabulary as `database`; 503 degraded is returned when either probe is unreachable. - Probes are bounded by the existing 2s timeout: sqlite via aiosqlite SELECT 1 on the resolved path, postgres via a bounded psycopg AsyncConnection SELECT 1 on the DSN with the configured search_path. A missing driver for a configured backend degrades readiness (the runtime could not run either). - Documented the two-probe semantics in the endpoint docstring and backend/app/gateway/AGENTS.md. ## Verification - New tests: healthy ORM engine + unreachable legacy checkpointer backend -> 503 degraded with database: ok / checkpointer: unreachable; checkpointer probe mapping for memory/sqlite(postgres missing-driver) backends; existing payload tests now pin the checkpointer field. - cd backend && python -m pytest tests/test_gateway_health.py: 11 passed. - app.gateway.app imports cleanly; ruff check + ruff format clean. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** design, implementation, and unit tests produced with AI assistance; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
|
Thanks @willem-bd — addressed in
The payload now includes a Coverage added: healthy ORM engine + unreachable legacy checkpointer backend → 503 with |
zhfeng
left a comment
There was a problem hiding this comment.
I found two readiness correctness issues that are not covered by the existing review threads.
willem-bd
left a comment
There was a problem hiding this comment.
Re-reviewed the latest revision. I found three remaining readiness correctness issues.
…up checkpointer snapshot ## Why Second round of review on bytedance#5166 (zhfeng P1/P2, willem-bd P1/P1/P2). Three correctness issues remained in the readiness endpoint: - The database and checkpointer probes ran sequentially, each allowed 2s, so a healthy response could take almost 4s - past Kubernetes' 1s default readinessProbe timeout and inside Docker's 3s client timeout. A slow but healthy backend could make every replica unready. - The checkpointer probe re-resolved process-wide, hot-reloaded configuration per request, while app.state.checkpointer/store are built once from the startup_config snapshot in langgraph_runtime(). After a live config edit the endpoint could probe a backend the running gateway does not use, and a resolution failure was swallowed into None -> not_configured -> 200. - The SQLite probe opened the path with aiosqlite.connect(), which creates the file when missing: a deleted checkpoint database was silently resurrected as an empty file and reported ok instead of surfacing the outage. ## What changed - backend/app/gateway/health.py: the two probes now run concurrently beneath a single endpoint-wide deadline (_READINESS_DEADLINE_SECONDS=3.0) so a healthy response completes within one probe window (~2s), never the sum of both. A probe that overruns the deadline degrades the endpoint instead of hanging. - langgraph_runtime() now records the checkpointer/Store config resolved from the same startup_config snapshot its checkpointer/store singletons are built from (app.state.checkpointer_config); /health/ready probes that snapshot and never re-resolves hot-reloaded config. resolve_checkpointer_config() returns None on resolution failure and the endpoint fails closed (503, checkpointer: unreachable) instead of reporting not_configured. - The SQLite probe opens disk-backed paths with the non-creating mode=rw URI flag, so a missing database file stays missing and yields unreachable; in-memory forms (:memory:, file:...mode=memory) have nothing external to probe and report not_configured like the memory backend. - Orchestrator timeouts now sit above the endpoint bound: Helm readinessProbe gains timeoutSeconds: 5 (Kubernetes default is 1s) and the docker-compose gateway healthcheck client timeout moves from 3s to 5s. ## Verification - New regression tests: concurrent probes keep total elapsed time within one probe window; a probe ignoring its budget trips the endpoint deadline to 503; missing SQLite file stays absent and yields unreachable; in-memory SQLite forms map to not_configured; missing startup snapshot / config resolution failure fail closed to 503; resolve_checkpointer_config() raising is covered. - cd backend && python -m pytest tests/test_gateway_health.py: 21 passed; tests/test_gateway_docs_toggle.py and lifespan/shutdown gateway suites pass. - ruff check + ruff format clean on all changed files. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** implemented the reviewer-requested concurrency/deadline, startup-snapshot probing, fail-closed resolution, and non-creating SQLite probe; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
willem-bd
left a comment
There was a problem hiding this comment.
I found one availability issue in the current revision.
…strict gate ## Why Review on bytedance#5166 (willem-bd, P1): every request to /health/ready opened a new PostgreSQL connection in _probe_postgres_backend, outside both the ORM pool and the runtime checkpointer pool. The route is public through the /health auth prefix and nginx proxies /health/*, so concurrent unauthenticated requests could create an unbounded number of connections (each held for up to two seconds), exhaust PostgreSQL max_connections, and take down both normal traffic and the readiness probe itself. ## What changed - backend/app/gateway/health.py: connection-opening checkpointer probes (sqlite connect, postgres AsyncConnection.connect) now run inside a strict per-process gate - an asyncio.Lock cached per running event loop - so at most one probe connection can be in flight per worker process. Requests that queue behind the gate are still shed by the existing endpoint-wide deadline, so a flood cannot pile up new connections or open files. - Memory and unknown-backend decisions stay outside the gate; payload and probe semantics are unchanged. The serialization is documented in the module docstring and backend/app/gateway/AGENTS.md. ## Verification - New regression test: 8 concurrent readiness_payload() requests against an instrumented sqlite probe assert the maximum number of in-flight probe connections is 1 while every request still returns 200. - cd backend && python -m pytest tests/test_gateway_health.py: 22 passed; tests/test_gateway_docs_toggle.py and tests/test_gateway_lifespan_shutdown.py also pass on the merged main head. - ruff check + ruff format clean on all changed files. ## AI assistance **Tool(s) used:** Codex (coding agent) **How you used it:** implemented the reviewer-requested strict concurrency bound for the public readiness probe; reviewed before commit. - [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
Why
GET /health only proves the process is up: it returns 200 even when the persistence engine cannot reach the database. Orchestrators already treat it as a readiness gate (docker-compose.yaml marks the gateway service healthy and nginx depends_on service_healthy), so a DB outage or a still-migrating Postgres leaves the stack 'healthy' while every request fails.
What changed
Surface area
Validation
AI assistance
Tool(s) used: Codex (coding agent)
How you used it: design, implementation, and unit tests produced with AI assistance; reviewed before commit.