Skip to content

fix(middleman): stop 500ing on models whose lab has no dispatch class [SEN-237] - #1404

Open
metr-background-agents[bot] wants to merge 1 commit into
mainfrom
fix/sen-237-unknown-lab-500
Open

fix(middleman): stop 500ing on models whose lab has no dispatch class [SEN-237]#1404
metr-background-agents[bot] wants to merge 1 commit into
mainfrom
fix/sen-237-unknown-lab-500

Conversation

@metr-background-agents

@metr-background-agents metr-background-agents Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Overview

A model config whose lab has no entry in middleman's provider-dispatch table (api_to_class) made POST /completions return 500 SafeInternalError — a request that can never succeed, reported to Sentry on every attempt. In prd this fired for gpt-4-assistants (lab openai-assistants, a lab middleman has never implemented).

While confirming the root cause I found the same bad row also KeyErrors the entire /permitted_models_info listing for every user in that model's group, which is the more serious of the two. Both are fixed here.

Linear: https://linear.app/metrevals/issue/SEN-237/safeinternalerror-unknown-lab-openai-assistants-for-model-gpt-4
Sentry: https://metr-sh.sentry.io/issues/HAWK-483

Approach

Nothing validates lab before request time: ModelInfo is a plain dataclass, LabName is a static-only Literal, and the admin read schema deliberately tolerates unknown labs (admin/schemas.py:112), so DB JSONB rows can carry anything. It's reachable through the validated admin API too — LabName is only asserted to be a superset of api_to_class (test_admin_models.py:522), and dummy-chat is in LabName while its api_to_class entry is commented out.

Three small changes, no behaviour change for well-configured models:

  1. models.py _load_all_models — log model_config.unknown_lab (structured, error level) at load time, so the misconfiguration stays visible to operators once the request-time 500 stops paging Sentry. Mirrors the model_config.unknown_fields_dropped tolerate-and-log precedent immediately above it.
  2. apis.py:581 — raise BadReq (400) instead of SafeInternalError (500). validate_completions_req already answers "model you asked for isn't usable" with BadReq on this same path, and passthrough already treats this exact condition as a client-visible 404 model not found (passthrough.py:578/714/787) — the unified path was the sole outlier. A 5xx tells callers to retry something unsatisfiable, and Sentry's Starlette integration reports 5xx HTTPExceptions (hence handled: yes / mechanism: starlette on the event).
  3. models.py to_public — guarded api_to_class.get(...); report features/is_chat as None (both already None-able on PublicModelInfo, and the are_details_secret branch already returns None for both, so consumers handle it) instead of KeyError-ing the whole listing.

Alternative ruled out: dropping undispatchable models from the registry. dead models are deliberately retained "for permission checks on old data", and get_labs_for_public_names() feeds the cross-lab scan safeguard on private transcripts (models.py:383 → server.py:566). Removing rows would silently strip group/lab metadata for historical data and make that safeguard fail open — the exact failure mode PLT-671 is open to close. Keeping the row registered but unusable preserves both invariants.

Side benefit: the old message interpolated model.lab into a client-visible response body (SafeInternalError isn't redacted), disclosing the lab of a secret model — precisely what are_details_secret exists to hide. The lab now only appears in server-side logs.

Not addressed here: the underlying prd data. openai-assistants isn't a real middleman lab, so that gpt-4-assistants row is unusable regardless and should be removed or repointed at openai-chat/openai-responses. Flagged on the Linear ticket; touching prod data is out of scope for this change.

Testing & validation

Three tests, each confirmed failing on main before the fix:

  • test_undispatchable_lab_is_a_bad_request_not_an_internal_error — reproduces the reported event exactly (gpt-4-assistants / openai-assistants); was SafeInternalError: 500, now BadReq.
  • test_to_public_survives_unknown_lab — was KeyError: 'openai-assistants', now degrades to features=None / is_chat=None.
  • test_unknown_lab_is_logged_at_load_time — asserts the structured log fires once, and that the row stays registered (guards the invariant above).
uv run pytest                        # 1106 passed — full middleman suite, no regressions
uvx ruff@0.15.21 check / format      # clean
uv run basedpyright                  # 115 errors before, 115 after — zero introduced
scripts/dump_openapi.py middleman.server:app   # byte-identical to base → api-compat/oasdiff gate unaffected
  • Verified the change works (commands / manual steps described above)
  • Added or updated tests where it makes sense

Code quality

  • pre-commit run --all-files passes (ruff, basedpyright/mypy, eslint/prettier/tsc, shellcheck — what CI's Lint job runs)

Before merging

  • PR title is a Conventional Commit with a lower-case subject — it becomes the squash-merge commit subject and drives the SemVer bump
  • All commits are signed and show as Verified on GitHub — see Commit signing

@metr-background-agents
metr-background-agents Bot requested a review from a team as a code owner August 20, 2026 21:24
@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown

🥥 preview on hawk/prd

1 meaningful change(s) · 🔁 1 replace — 3 rebuild-churn hidden

  • 🔁 middleman-task-def · replace · aws:ecs/taskDefinition:TaskDefinition
Show diffs (1 resource(s))

🔁 middleman-task-def · replace · aws:ecs/taskDefinition:TaskDefinition

       containerDefinitions: (json) [
-          [0]: {
-              cpu             : 128
-              environment     : [
-                  [0]: {
-                      name : "DD_APM_ENABLED"
-                      value: "true"
                     }
-                  [1]: {
-                      name : "DD_APM_NON_LOCAL_TRAFFIC"
-                      value: "true"
                     }
-                  [2]: {
-                      name : "DD_APM_RECEIVER_SOCKET"
-                      value: "/var/run/datadog/apm.socket"
                     }
-                  [3]: {
-                      name : "DD_DOGSTATSD_NON_LOCAL_TRAFFIC"
-                      value: "true"
                     }
-                  [4]: {
-                      name : "DD_ECS_FARGATE"
-                      value: "true"
                     }
-                  [5]: {
-                      name : "DD_ENV"
-                      value: "prd"
                     }
-                  [6]: {
-                      name : "DD_PROCESS_AGENT_ENABLED"
-                      value: "false"
                     }
-                  [7]: {
-                      name : "DD_SITE"
-                      value: "us3.datadoghq.com"
                     }
-                  [8]: {
-                      name : "DD_TAGS"
-                      value: "env:prd service:middleman"
                     }
-                  [9]: {
-                      name : "ECS_FARGATE"
-                      value: "true"
                     }
                 ]
-              essential       : false
-              healthCheck     : {
-                  command    : [
-                      [0]: "CMD"
-                      [1]: "agent"
-                      [2]: "health"
                     ]
-                  interval   : 30
-                  retries    : 3
-                  startPeriod: 15
-                  timeout    : 5
                 }
-              image           : "public.ecr.aws/datadog/agent:7"
-              logConfiguration: {
-                  logDriver: "awslogs"
-                  options  : {
-                      awslogs-group        : "prd/middleman"
-                      awslogs-region       : "us-west-2"
-                      awslogs-stream-prefix: "datadog-agent"
                     }
                 }
-              memory          : 256
-              mountPoints     : [
-                  [0]: {
-                      containerPath: "/var/run/datadog"
-                      readOnly     : false
-                      sourceVolume : "dd-sockets"
                     }
                 ]
-              name            : "datadog-agent"
-              portMappings    : [
-                  [0]: {
-                      containerPort: 8126
-                      hostPort     : 8126
-                      protocol     : "tcp"
                     }
-                  [1]: {
-                      containerPort: 8125
-                      hostPort     : 8125
-                      protocol     : "udp"
                     }
                 ]
-              secrets         : [
-                  [0]: {
-                      name     : "DD_API_KEY"
-                      valueFrom: "arn:aws:secretsmanager:us-west-2:328726945407:secret:prd/platform/datadog-api-key-wSsRpp"
                     }
                 ]
-              systemControls  : []
-              volumesFrom     : []
             }
-          [1]: {
-              cpu              : 8064
-              dependsOn        : [
-                  [0]: {
-                      condition    : "START"
-                      containerName: "datadog-agent"
                     }
                 ]
-              environment      : [
-                  [0]: {
-                      name : "DD_AGENT_HOST"
-                      value: "localhost"
                     }
-                  [1]: {
-                      name : "DD_DOGSTATSD_PORT"
-                      value: "8125"
                     }
-                  [2]: {
-                      name : "DD_DOGSTATSD_TAGS"
-                      value: "service:middleman,env:prd"
                     }
-                  [3]: {
-                      name : "DD_ENV"
-                      value: "prd"
                     }
-                  [4]: {
-                      name : "DD_LOGS_INJECTION"
-                      value: "true"
                     }
-                  [5]: {
-                      name : "DD_SERVICE"
-                      value: "middleman"
                     }
-                  [6]: {
-                      name : "DD_SITE"
-                      value: "us3.datadoghq.com"
                     }
-                  [7]: {
-                      name : "DD_TRACE_AGENT_URL"
-                      value: "unix:///var/run/datadog/apm.socket"
                     }
-                  [8]: {
-                      name : "DD_TRACE_CLIENT_IP_ENABLED"
-                      value: "true"
                     }
-                  [9]: {
-                      name : "DD_TRACE_CLIENT_IP_HEADER"
-                      value: "X-Forwarded-For"
                     }
-                  [10]: {
-                      name : "DD_TRACE_REQUEST_BODY_ENABLED"
-                      value: "false"
                     }
-                  [11]: {
-                      name : "DD_TRACE_RESPONSE_BODY_ENABLED"
-                      value: "false"
                     }
-                  [12]: {
-                      name : "DD_TRACE_SAMPLE_RATE"
-                      value: "1.0"
                     }
-                  [13]: {
-                      name : "DD_TRACE_SAMPLING_RULES"
-                      value: (json) [
-                          [0]: {
-                              resource   : "GET /health"
-                              sample_rate: 0
                             }
-                          [1]: {
-                              resource   : "GET /health/deep"
-                              sample_rate: 0
                             }
                         ]
                     }
-                  [14]: {
-                      name : "GOOGLE_CLOUD_PROJECT_FOR_PUBLIC_MODELS"
-                      value: "metr-pub"
                     }
-                  [15]: {
-                      name : "HAWK_OTEL_TRACING_ENABLED"
-                      value: "true"
                     }
-                  [16]: {
-                      name : "HAWK_SERVICE_VERSION"
-                      value: "328726945407.dkr.ecr.us-west-2.amazonaws.com/prd-middleman@sha256:084111e12c587132a55c6066e826ea132056dbb08df8387daa1a09b2a97cd4f4"
                     }
-                  [17]: {
-                      name : "MIDDLEMAN_ACCEPT_DEV_ADMIN"
-                      value: "false"
                     }
-                  [18]: {
-                      name : "MIDDLEMAN_ANTHROPIC_PROFILES"
-                      value: (json) {
-                          cvp-prd           : {
-                              federation_rule_id    : "fdrl_0188NqFctiurqpK5RQPrm4Be"
-                              mode                  : "wif"
-                              okta_client_id        : "0oa25x4kvdjvlWx2s1d8"
-                              okta_client_secret_key: "OKTA_ANTHROPIC_WIF_CVP_PRD_CLIENT_SECRET"
-                              okta_scope            : "anthropic:federate"
-                              okta_token_url        : "https://metr.okta.com/oauth2/aus25x4jzx3rqTdfu1d8/v1/token"
-                              organization_id       : "d9d245c1-46ae-4a6f-a259-2d45ac7fd9cb"
-                              service_account_id    : "svac_01MvRBRR6dPf9RBV6h1VpKGV"
-                              workspace_id          : "wrkspc_01VYApxyW4zjSHNQJ6GNU9NB"
                             }
-                          prd-data-retention: {
-                              federation_rule_id    : "fdrl_016oqwb68ejiSoxVP3DDF9TT"
-                              mode                  : "wif"
-                              okta_client_id        : "0oa241qaua0DSWZcI1d8"
-                              okta_client_secret_key: "OKTA_ANTHROPIC_WIF_GENERAL_PRD_CLIENT_SECRET"
-                              okta_scope            : "anthropic:federate"
-                              okta_token_url        : "https://metr.okta.com/oauth2/aus241qbpajIjDhLn1d8/v1/token"
-                              organization_id       : "36a533b0-8a62-4bb4-80a5-a774efa6c965"
-                              service_account_id    : "svac_01QMPfLf1Xe2mub71Hc2snXe"
-                              workspace_id          : "wrkspc_016v1JbWGmwFJPxwwYpnbVbF"
                             }
-                          prd-zdr-default   : {
-                              federation_rule_id    : "fdrl_01SBtrsG1gbucsqQw7eSBGnv"
-                              mode                  : "wif"
-                              okta_client_id        : "0oa241qaua0DSWZcI1d8"
-                              okta_client_secret_key: "OKTA_ANTHROPIC_WIF_GENERAL_PRD_CLIENT_SECRET"
-                              okta_scope            : "anthropic:federate"
-                              okta_token_url        : "https://metr.okta.com/oauth2/aus241qbpajIjDhLn1d8/v1/token"
-                              organization_id       : "36a533b0-8a62-4bb4-80a5-a774efa6c965"
-                              service_account_id    : "svac_01QMPfLf1Xe2mub71Hc2snXe"
-                              workspace_id          : "default"
                             }
-                          predeployment-prd : {
-                              federation_rule_id    : "fdrl_01NkUut8MbvPjWPcAm8CEfm6"
-                              mode                  : "wif"
-                              okta_client_id        : "0oa23f7lr6z9fGgyi1d8"
-                              okta_client_secret_key: "OKTA_ANTHROPIC_WIF_PREDEPLOYMENT_PRD_CLIENT_SECRET"
-                              okta_scope            : "anthropic:federate"
-                              okta_token_url        : "https://metr.okta.com/oauth2/aus23f7nq64KtXumU1d8/v1/token"
-                              organization_id       : "739154c2-f1e0-4d0c-8258-e127be1cdb4f"
-                              service_account_id    : "svac_018NCC4jKNDhwkwEGifroxkP"
-                              workspace_id          : "wrkspc_0128f5pDkQJEaA6vs8RgAdDG"
                             }
                         }
                     }
-                  [19]: {
-                      name : "MIDDLEMAN_API_KEYS_SECRET_ARN"
-                      value: "arn:aws:secretsmanager:us-west-2:328726945407:secret:prd/middleman/api-keys-HIMD9G"
                     }
-                  [20]: {
-                      name : "MIDDLEMAN_AUTH_PROVIDERS"
-                      value: (json) [
-                          [0]: {
-                              admin_groups  : []
-                              audiences     : [
-                                  [0]: "https://model-poking-3"
                                 ]
-                              default_groups: []
-                              issuer        : "https://metr.okta.com/oauth2/aus1ww3m0x41jKp3L1d8"
-                              jwks_uri      : "https://metr.okta.com/oauth2/aus1ww3m0x41jKp3L1d8/v1/keys"
                             }
                         ]
                     }
-                  [21]: {
-                      name : "MIDDLEMAN_CONFIG_FILE"
-                      value: "middleman.yaml"
                     }
-                  [22]: {
-                      name : "MIDDLEMAN_DATABASE_URL"
-                      value: "postgresql+psycopg://middleman@prd-inspect-ai-warehouse.cluster-cwnzyrtfzfzn.us-west-2.rds.amazonaws.com:5432/inspect"
                     }
-                  [23]: {
-                      name : "MIDDLEMAN_ENV"
-                      value: "prd"
                     }
-                  [24]: {
-                      name : "MIDDLEMAN_METRICS_LOG_GROUP"
-                      value: "prd/middleman/metrics"
                     }
-                  [25]: {
-                      name : "MIDDLEMAN_TRAFFIC_LOG_CW_GROUP"
-                      value: "prd/middleman/traffic"
                     }
-                  [26]: {
-                      name : "MIDDLEMAN_TRAFFIC_LOG_LEVEL"
-                      value: "full"
                     }
-                  [27]: {
-                      name : "MIDDLEMAN_TRAFFIC_LOG_S3_BUCKET"
-                      value: "metr-prd-middleman-traffic"
                     }
-                  [28]: {
-                      name : "MIDDLEMAN_VALKEY_URL"
-                      value: "rediss://prd-valkey-yrds5r.serverless.usw2.cache.amazonaws.com:6379"
                     }
-                  [29]: {
-                      name : "SENTRY_DSN"
-                      value: "https://e0505c55080bced50b2d248fbed1f80c@o4506945192919040.ingest.us.sentry.io/4510834768478208"
                     }
-                  [30]: {
-                      name : "SENTRY_ENVIRONMENT"
-                      value: "prd"
                     }
-                  [31]: {
-                      name : "SENTRY_TRACES_SAMPLE_RATE"
-                      value: "0"
                     }
-                  [32]: {
-                      name : "WEB_CONCURRENCY"
-                      value: "16"
                     }
                 ]
-              essential        : true
-              healthCheck      : {
-                  command    : [
-                      [0]: "CMD"
-                      [1]: "python"
-                      [2]: "-c"
-                      [3]: "import urllib.request; urllib.request.urlopen('http://localhost:3500/health', timeout=5)"
                     ]
-                  interval   : 30
-                  retries    : 5
-                  startPeriod: 120
-                  timeout    : 10
                 }
-              image            : "328726945407.dkr.ecr.us-west-2.amazonaws.com/prd-middleman@sha256:084111e12c587132a55c6066e826ea132056dbb08df8387daa1a09b2a97cd4f4"
-              logConfiguration : {
-                  logDriver: "awslogs"
-                  options  : {
-                      awslogs-group        : "prd/middleman"
-                      awslogs-region       : "us-west-2"
-                      awslogs-stream-prefix: "middleman"
-                      max-buffer-size      : "25m"
-                      mode                 : "non-blocking"
                     }
                 }
-              memory           : 16128
-              memoryReservation: 100
-              mountPoints      : [
-                  [0]: {
-                      containerPath: "/var/run/datadog"
-                      readOnly     : false
-                      sourceVolume : "dd-sockets"
                     }
                 ]
-              name             : "middleman"
-              portMappings     : [
-                  [0]: {
-                      containerPort: 3500
-                      hostPort     : 3500
-                      name         : "middleman"
-                      protocol     : "tcp"
                     }
                 ]
-              systemControls   : []
-              volumesFrom      : []
             }
         ]
  => [unknown]
Full preview (including hidden churn) 
Previewing update (prd):
@ previewing update....
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:prd::hawk::pulumi:pulumi:Stack::hawk-prd]
    +-command:local:Command: (replace)
        [id=rds-db-users34171fad]
        [urn=urn:pulumi:prd::hawk::metr:core:CoreStack$metr:core:Rds$command:local:Command::rds-db-users]
        [provider=urn:pulumi:prd::hawk::pulumi:providers:command::default_1_2_1::e73ecd16-257c-4e9d-a9e9-fce4e61957f5]
      ~ triggers: [
          ~ [0]: "1787255757.1500971" => "1787261186.1016502"
        ]
@ previewing update....
    ~ docker-build:index:Image: (update)
        [id=sha256:285bca24c72966b2184a84a8799cc5c285cd8d16a399f96f37176385936a3e58]
        [urn=urn:pulumi:prd::hawk::metr:core:Middleman$docker-build:index:Image::middleman-image]
        [provider=urn:pulumi:prd::hawk::pulumi:providers:docker-build::default_0_0_21::77d5cd9e-f70c-48ba-8c7e-fcf00490e25f]
      - contextHash: "a636571aa4750c6bd4bca95c0d2ef10077d94111244694b67c29d42fe0f368b4"
    +-aws:ecs/taskDefinition:TaskDefinition: (replace)
        [id=prd-middleman]
        [urn=urn:pulumi:prd::hawk::metr:core:Middleman$aws:ecs/taskDefinition:TaskDefinition::middleman-task-def]
        [provider=urn:pulumi:prd::hawk::pulumi:providers:aws::default_7_39_0::0b3d96f1-af08-4dfb-be7d-473b1d47f247]
      ~ containerDefinitions: (json) [
      -     [0]: {
              - cpu             : 128
              - environment     : [
              -     [0]: {
                      - name : "DD_APM_ENABLED"
                      - value: "true"
                    }
              -     [1]: {
                      - name : "DD_APM_NON_LOCAL_TRAFFIC"
                      - value: "true"
                    }
              -     [2]: {
                      - name : "DD_APM_RECEIVER_SOCKET"
                      - value: "/var/run/datadog/apm.socket"
                    }
              -     [3]: {
                      - name : "DD_DOGSTATSD_NON_LOCAL_TRAFFIC"
                      - value: "true"
                    }
              -     [4]: {
                      - name : "DD_ECS_FARGATE"
                      - value: "true"
                    }
              -     [5]: {
                      - name : "DD_ENV"
                      - value: "prd"
                    }
              -     [6]: {
                      - name : "DD_PROCESS_AGENT_ENABLED"
                      - value: "false"
                    }
              -     [7]: {
                      - name : "DD_SITE"
                      - value: "us3.datadoghq.com"
                    }
              -     [8]: {
                      - name : "DD_TAGS"
                      - value: "env:prd service:middleman"
                    }
              -     [9]: {
                      - name : "ECS_FARGATE"
                      - value: "true"
                    }
                ]
              - essential       : false
              - healthCheck     : {
                  - command    : [
                  -     [0]: "CMD"
                  -     [1]: "agent"
                  -     [2]: "health"
                    ]
                  - interval   : 30
                  - retries    : 3
                  - startPeriod: 15
                  - timeout    : 5
                }
              - image           : "public.ecr.aws/datadog/agent:7"
              - logConfiguration: {
                  - logDriver: "awslogs"
                  - options  : {
                      - awslogs-group        : "prd/middleman"
                      - awslogs-region       : "us-west-2"
                      - awslogs-stream-prefix: "datadog-agent"
                    }
                }
              - memory          : 256
              - mountPoints     : [
              -     [0]: {
                      - containerPath: "/var/run/datadog"
                      - readOnly     : false
                      - sourceVolume : "dd-sockets"
                    }
                ]
              - name            : "datadog-agent"
              - portMappings    : [
              -     [0]: {
                      - containerPort: 8126
                      - hostPort     : 8126
                      - protocol     : "tcp"
                    }
              -     [1]: {
                      - containerPort: 8125
                      - hostPort     : 8125
                      - protocol     : "udp"
                    }
                ]
              - secrets         : [
              -     [0]: {
                      - name     : "DD_API_KEY"
                      - valueFrom: "arn:aws:secretsmanager:us-west-2:328726945407:secret:prd/platform/datadog-api-key-wSsRpp"
                    }
                ]
              - systemControls  : []
              - volumesFrom     : []
            }
      -     [1]: {
              - cpu              : 8064
              - dependsOn        : [
              -     [0]: {
                      - condition    : "START"
                      - containerName: "datadog-agent"
                    }
                ]
              - environment      : [
              -     [0]: {
                      - name : "DD_AGENT_HOST"
                      - value: "localhost"
                    }
              -     [1]: {
                      - name : "DD_DOGSTATSD_PORT"
                      - value: "8125"
                    }
              -     [2]: {
                      - name : "DD_DOGSTATSD_TAGS"
                      - value: "service:middleman,env:prd"
                    }
              -     [3]: {
                      - name : "DD_ENV"
                      - value: "prd"
                    }
              -     [4]: {
                      - name : "DD_LOGS_INJECTION"
                      - value: "true"
                    }
              -     [5]: {
                      - name : "DD_SERVICE"
                      - value: "middleman"
                    }
              -     [6]: {
                      - name : "DD_SITE"
                      - value: "us3.datadoghq.com"
                    }
              -     [7]: {
                      - name : "DD_TRACE_AGENT_URL"
                      - value: "unix:///var/run/datadog/apm.socket"
                    }
              -     [8]: {
                      - name : "DD_TRACE_CLIENT_IP_ENABLED"
                      - value: "true"
                    }
              -     [9]: {
                      - name : "DD_TRACE_CLIENT_IP_HEADER"
                      - value: "X-Forwarded-For"
                    }
              -     [10]: {
                      - name : "DD_TRACE_REQUEST_BODY_ENABLED"
                      - value: "false"
                    }
              -     [11]: {
                      - name : "DD_TRACE_RESPONSE_BODY_ENABLED"
                      - value: "false"
                    }
              -     [12]: {
                      - name : "DD_TRACE_SAMPLE_RATE"
                      - value: "1.0"
                    }
              -     [13]: {
                      - name : "DD_TRACE_SAMPLING_RULES"
                      - value: (json) [
                      -     [0]: {
                              - resource   : "GET /health"
                              - sample_rate: 0
                            }
                      -     [1]: {
                              - resource   : "GET /health/deep"
                              - sample_rate: 0
                            }
                        ]

                    }
              -     [14]: {
                      - name : "GOOGLE_CLOUD_PROJECT_FOR_PUBLIC_MODELS"
                      - value: "metr-pub"
                    }
              -     [15]: {
                      - name : "HAWK_OTEL_TRACING_ENABLED"
                      - value: "true"
                    }
              -     [16]: {
                      - name : "HAWK_SERVICE_VERSION"
                      - value: "328726945407.dkr.ecr.us-west-2.amazonaws.com/prd-middleman@sha256:084111e12c587132a55c6066e826ea132056dbb08df8387daa1a09b2a97cd4f4"
                    }
              -     [17]: {
                      - name : "MIDDLEMAN_ACCEPT_DEV_ADMIN"
                      - value: "false"
                    }
              -     [18]: {
                      - name : "MIDDLEMAN_ANTHROPIC_PROFILES"
                      - value: (json) {
                          - cvp-prd           : {
                              - federation_rule_id    : "fdrl_0188NqFctiurqpK5RQPrm4Be"
                              - mode                  : "wif"
                              - okta_client_id        : "0oa25x4kvdjvlWx2s1d8"
                              - okta_client_secret_key: "OKTA_ANTHROPIC_WIF_CVP_PRD_CLIENT_SECRET"
                              - okta_scope            : "anthropic:federate"
                              - okta_token_url        : "https://metr.okta.com/oauth2/aus25x4jzx3rqTdfu1d8/v1/token"
                              - organization_id       : "d9d245c1-46ae-4a6f-a259-2d45ac7fd9cb"
                              - service_account_id    : "svac_01MvRBRR6dPf9RBV6h1VpKGV"
                              - workspace_id          : "wrkspc_01VYApxyW4zjSHNQJ6GNU9NB"
                            }
                          - prd-data-retention: {
                              - federation_rule_id    : "fdrl_016oqwb68ejiSoxVP3DDF9TT"
                              - mode                  : "wif"
                              - okta_client_id        : "0oa241qaua0DSWZcI1d8"
                              - okta_client_secret_key: "OKTA_ANTHROPIC_WIF_GENERAL_PRD_CLIENT_SECRET"
                              - okta_scope            : "anthropic:federate"
                              - okta_token_url        : "https://metr.okta.com/oauth2/aus241qbpajIjDhLn1d8/v1/token"
                              - organization_id       : "36a533b0-8a62-4bb4-80a5-a774efa6c965"
                              - service_account_id    : "svac_01QMPfLf1Xe2mub71Hc2snXe"
                              - workspace_id          : "wrkspc_016v1JbWGmwFJPxwwYpnbVbF"
                            }
                          - prd-zdr-default   : {
                              - federation_rule_id    : "fdrl_01SBtrsG1gbucsqQw7eSBGnv"
                              - mode                  : "wif"
                              - okta_client_id        : "0oa241qaua0DSWZcI1d8"
                              - okta_client_secret_key: "OKTA_ANTHROPIC_WIF_GENERAL_PRD_CLIENT_SECRET"
                              - okta_scope            : "anthropic:federate"
                              - okta_token_url        : "https://metr.okta.com/oauth2/aus241qbpajIjDhLn1d8/v1/token"
                              - organization_id       : "36a533b0-8a62-4bb4-80a5-a774efa6c965"
                              - service_account_id    : "svac_01QMPfLf1Xe2mub71Hc2snXe"
                              - workspace_id          : "default"
                            }
                          - predeployment-prd : {
                              - federation_rule_id    : "fdrl_01NkUut8MbvPjWPcAm8CEfm6"
                              - mode                  : "wif"
                              - okta_client_id        : "0oa23f7lr6z9fGgyi1d8"
                              - okta_client_secret_key: "OKTA_ANTHROPIC_WIF_PREDEPLOYMENT_PRD_CLIENT_SECRET"
                              - okta_scope            : "anthropic:federate"
                              - okta_token_url        : "https://metr.okta.com/oauth2/aus23f7nq64KtXumU1d8/v1/token"
                              - organization_id       : "739154c2-f1e0-4d0c-8258-e127be1cdb4f"
                              - service_account_id    : "svac_018NCC4jKNDhwkwEGifroxkP"
                              - workspace_id          : "wrkspc_0128f5pDkQJEaA6vs8RgAdDG"
                            }
                        }

                    }
              -     [19]: {
                      - name : "MIDDLEMAN_API_KEYS_SECRET_ARN"
                      - value: "arn:aws:secretsmanager:us-west-2:328726945407:secret:prd/middleman/api-keys-HIMD9G"
                    }
              -     [20]: {
                      - name : "MIDDLEMAN_AUTH_PROVIDERS"
                      - value: (json) [
                      -     [0]: {
                              - admin_groups  : []
                              - audiences     : [
                              -     [0]: "https://model-poking-3"
                                ]
                              - default_groups: []
                              - issuer        : "https://metr.okta.com/oauth2/aus1ww3m0x41jKp3L1d8"
                              - jwks_uri      : "https://metr.okta.com/oauth2/aus1ww3m0x41jKp3L1d8/v1/keys"
                            }
                        ]

                    }
              -     [21]: {
                      - name : "MIDDLEMAN_CONFIG_FILE"
                      - value: "middleman.yaml"
                    }
              -     [22]: {
                      - name : "MIDDLEMAN_DATABASE_URL"
                      - value: "postgresql+psycopg://middleman@prd-inspect-ai-warehouse.cluster-cwnzyrtfzfzn.us-west-2.rds.amazonaws.com:5432/inspect"
                    }
              -     [23]: {
                      - name : "MIDDLEMAN_ENV"
                      - value: "prd"
                    }
              -     [24]: {
                      - name : "MIDDLEMAN_METRICS_LOG_GROUP"
                      - value: "prd/middleman/metrics"
                    }
              -     [25]: {
                      - name : "MIDDLEMAN_TRAFFIC_LOG_CW_GROUP"
                      - value: "prd/middleman/traffic"
                    }
              -     [26]: {
                      - name : "MIDDLEMAN_TRAFFIC_LOG_LEVEL"
                      - value: "full"
                    }
              -     [27]: {
                      - name : "MIDDLEMAN_TRAFFIC_LOG_S3_BUCKET"
                      - value: "metr-prd-middleman-traffic"
                    }
              -     [28]: {
                      - name : "MIDDLEMAN_VALKEY_URL"
                      - value: "rediss://prd-valkey-yrds5r.serverless.usw2.cache.amazonaws.com:6379"
                    }
              -     [29]: {
                      - name : "SENTRY_DSN"
                      - value: "https://e0505c55080bced50b2d248fbed1f80c@o4506945192919040.ingest.us.sentry.io/4510834768478208"
                    }
              -     [30]: {
                      - name : "SENTRY_ENVIRONMENT"
                      - value: "prd"
                    }
              -     [31]: {
                      - name : "SENTRY_TRACES_SAMPLE_RATE"
                      - value: "0"
                    }
              -     [32]: {
                      - name : "WEB_CONCURRENCY"
                      - value: "16"
                    }
                ]
              - essential        : true
              - healthCheck      : {
                  - command    : [
                  -     [0]: "CMD"
                  -     [1]: "python"
                  -     [2]: "-c"
                  -     [3]: "import urllib.request; urllib.request.urlopen('http://localhost:3500/health', timeout=5)"
                    ]
                  - interval   : 30
                  - retries    : 5
                  - startPeriod: 120
                  - timeout    : 10
                }
              - image            : "328726945407.dkr.ecr.us-west-2.amazonaws.com/prd-middleman@sha256:084111e12c587132a55c6066e826ea132056dbb08df8387daa1a09b2a97cd4f4"
              - logConfiguration : {
                  - logDriver: "awslogs"
                  - options  : {
                      - awslogs-group        : "prd/middleman"
                      - awslogs-region       : "us-west-2"
                      - awslogs-stream-prefix: "middleman"
                      - max-buffer-size      : "25m"
                      - mode                 : "non-blocking"
                    }
                }
              - memory           : 16128
              - memoryReservation: 100
              - mountPoints      : [
              -     [0]: {
                      - containerPath: "/var/run/datadog"
                      - readOnly     : false
                      - sourceVolume : "dd-sockets"
                    }
                ]
              - name             : "middleman"
              - portMappings     : [
              -     [0]: {
                      - containerPort: 3500
                      - hostPort     : 3500
                      - name         : "middleman"
                      - protocol     : "tcp"
                    }
                ]
              - systemControls   : []
              - volumesFrom      : []
            }
        ]
 => [unknown]
    ~ aws:ecs/service:Service: (update)
        [id=arn:aws:ecs:us-west-2:328726945407:service/prd-platform/prd-middleman]
        [urn=urn:pulumi:prd::hawk::metr:core:Middleman$aws:ecs/service:Service::middleman-service]
        [provider=urn:pulumi:prd::hawk::pulumi:providers:aws::default_7_39_0::0b3d96f1-af08-4dfb-be7d-473b1d47f247]
      ~ taskDefinition: "arn:aws:ecs:us-west-2:328726945407:task-definition/prd-middleman:625" => [unknown]
@ previewing update....
Resources:
    ~ 2 to update
    +-2 to replace
    4 changes. 684 unchanged

@revmischa
revmischa requested review from tbroadley and removed request for revmischa August 24, 2026 18:43
tbroadley
tbroadley reviewed Aug 24, 2026
View reviewed changes

@tbroadley tbroadley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add an alert for this metric?

Can we have some check that fails when someone tries to create a model with an unknown lab?

Comment thread middleman/src/middleman/models.py
Comment on lines +176 to +179
# A model config can name a lab we have no dispatch class for. Don't let one
# such row KeyError the whole listing — /permitted_models_info maps this over
# every permitted model, so it would 500 for everyone in that model's group.
# Report the capability fields as unknown rather than guessing.

@tbroadley tbroadley Aug 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should 400 in this case instead of 500ing, but I don't think we should silently report incomplete data.

This branch was previously deployed

1 inactive deployment
prd-pulumi-preview — c28960c7 Deployed Aug 20, 2026 by metr-background-agents[bot] via Pulumi Preview (prd) #5651
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tbroadley tbroadley tbroadley left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tbroadley @claude