fix(providers): preserve image references and fail open on unknown vision support - #11548
fix(providers): preserve image references and fail open on unknown vision support#11548kudai wants to merge 2 commits into
Conversation
16953b5 to
35335cd
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 35335cdffc
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
35335cd to
3717fda
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3717fda02a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
3717fda to
176a7a4
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 176a7a426f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
176a7a4 to
b24b3e3
Compare
|
@codex review |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
|
🤖 Reviewed at Blocking:
|
b24b3e3 to
5dcca42
Compare
|
All three points addressed in 5dcca42: Blocking — databricks clobber: Non-blocking — provider_name None: Note — wider than the name: covered by the rename above + the |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5dcca42ad7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Serving endpoints may use arbitrary aliases (e.g. team-chat) whose metadata resolves to an image-capable upstream model. Session config canonicalized against the alias leaves supports_vision unset, so request formatting treated the endpoint as text-only and silently omitted attached and tool-result images. Derive supports_vision from the resolved upstream model (canonical registry) in DatabricksProvider::stream before formatting the request, for both the responses and chat-completions paths.
5dcca42 to
05000a1
Compare
…sion Follow-up to aaif-goose#11496 (avoid sending images to non-vision models). Asymmetry fix (michaelneale review): the detect_image_path text branch keeps the path in the text so a downstream vision model can still act on it, while an explicit Image block produced a dead placeholder. ImageContent carries only base64 data + mime type (no path/URI survives), so the placeholders in formats/openai.rs, formats/databricks.rs and formats/openai_responses.rs now surface the mime type. Design question (None semantics): unknown (None) vision capability is now fail-open - images are sent, preserving pre-PR behavior for models absent from the canonical catalog (ollama/llava, openai_compatible, vllm, llamacpp, custom aliases). Only an explicit Some(false) strips images. Known non-vision catalog models still resolve to Some(false) and are gated. Codex review follow-ups: - Rehydrate canonical capabilities for legacy saved configs in the shared Agent::model_config_for_session path (covers CLI turns + ACP/Desktop loads, not just the CLI builder), preserving the Azure Foundry skip. - Databricks: when an endpoint alias resolves to a different upstream model, the resolved model's vision capability takes precedence over alias-based canonicalization (with_resolved_vision_support). Databricks alias resolution (cherry-pick 685b8098e): model_supports_vision now returns Option<bool> and leaves unknown resolved models as None so the gate fail-opens, instead of converting them to Some(false).
05000a1 to
c32a6f3
Compare
|
@codex review |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
Follow-up to #11496 (merged as 98dd530). Addresses review feedback from michaelneale and the Codex thread on formats/databricks.rs.
1. Databricks alias vision capability (cherry-picked from kudai/detect-vision-support)
Addresses the Codex review comment: serving endpoints may use arbitrary aliases (e.g. team-chat) whose metadata resolves to an image-capable upstream model. Session config canonicalized against the alias leaves supports_vision unset, so request formatting treated the endpoint as text-only and silently omitted images.
DatabricksProvider::stream now derives supports_vision from the resolved upstream model (canonical registry) before formatting, for both the responses and chat-completions paths. Unknown resolved models stay None so the formatter gate fail-opens (see #2).
2. Design question: None vision capability is now fail-open
The gate at all five sites was supports_vision.unwrap_or_default() → None = strip. Since supports_vision is only populated for models in the canonical catalog, this silently stripped images from every non-catalog model — ollama, openai_compatible, vllm and llamacpp have no catalog entries, so a genuinely vision-capable llava:13b or qwen2.5-vl on Ollama stopped receiving images, with no escape hatch (base_model_config_from_user_config hardcodes None).
Per michaelneale's review, the gates now fail open: unknown (None) → send images (pre-PR behavior); only an explicit Some(false) strips. Known non-vision catalog models still resolve to Some(false) and are gated, so the original #10311 fix is unchanged.
3. Asymmetry fix: omitted-image placeholder surfaces the mime type
The detect_image_path text branch keeps the path in the text (a downstream vision subagent can still act on it), while an explicit MessageContentBlock::Image produced a dead placeholder. ImageContent carries only base64 data + mime type — no path/URI survives — so the placeholders in formats/openai.rs, formats/databricks.rs and formats/openai_responses.rs now include the mime type, and the intentional split is documented at each site.
Testing