Skip to content

fix(providers): preserve image references and fail open on unknown vision support - #11548

Open
kudai wants to merge 2 commits into
aaif-goose:mainfrom
kudai:kudai/databricks-vision-followup
Open

fix(providers): preserve image references and fail open on unknown vision support#11548
kudai wants to merge 2 commits into
aaif-goose:mainfrom
kudai:kudai/databricks-vision-followup

Conversation

@kudai

@kudai kudai commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #11496 (merged as 98dd530). Addresses review feedback from michaelneale and the Codex thread on formats/databricks.rs.

1. Databricks alias vision capability (cherry-picked from kudai/detect-vision-support)

Addresses the Codex review comment: serving endpoints may use arbitrary aliases (e.g. team-chat) whose metadata resolves to an image-capable upstream model. Session config canonicalized against the alias leaves supports_vision unset, so request formatting treated the endpoint as text-only and silently omitted images.

DatabricksProvider::stream now derives supports_vision from the resolved upstream model (canonical registry) before formatting, for both the responses and chat-completions paths. Unknown resolved models stay None so the formatter gate fail-opens (see #2).

2. Design question: None vision capability is now fail-open

The gate at all five sites was supports_vision.unwrap_or_default() → None = strip. Since supports_vision is only populated for models in the canonical catalog, this silently stripped images from every non-catalog model — ollama, openai_compatible, vllm and llamacpp have no catalog entries, so a genuinely vision-capable llava:13b or qwen2.5-vl on Ollama stopped receiving images, with no escape hatch (base_model_config_from_user_config hardcodes None).

Per michaelneale's review, the gates now fail open: unknown (None) → send images (pre-PR behavior); only an explicit Some(false) strips. Known non-vision catalog models still resolve to Some(false) and are gated, so the original #10311 fix is unchanged.

3. Asymmetry fix: omitted-image placeholder surfaces the mime type

The detect_image_path text branch keeps the path in the text (a downstream vision subagent can still act on it), while an explicit MessageContentBlock::Image produced a dead placeholder. ImageContent carries only base64 data + mime type — no path/URI survives — so the placeholders in formats/openai.rs, formats/databricks.rs and formats/openai_responses.rs now include the mime type, and the intentional split is documented at each site.

Testing

  • cargo test -p goose-provider-types: 565 passed
  • cargo test -p goose-providers: 159 passed
  • cargo check -p goose: clean
  • Updated tests that asserted strip-on-None to use explicit with_vision_support(false); added coverage for fail-open (None → image sent) and for the databricks helper returning None for unknown aliases.

@kudai
kudai force-pushed the kudai/databricks-vision-followup branch 2 times, most recently from 16953b5 to 35335cd Compare August 25, 2026 06:31

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 35335cdffc

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread crates/goose-provider-types/src/formats/openai_responses.rs
Comment thread crates/goose-providers/src/databricks.rs Outdated
@kudai
kudai force-pushed the kudai/databricks-vision-followup branch from 35335cd to 3717fda Compare August 25, 2026 06:49
@kudai

kudai commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3717fda02a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread crates/goose-provider-types/src/formats/openai_responses.rs
@kudai
kudai force-pushed the kudai/databricks-vision-followup branch from 3717fda to 176a7a4 Compare August 25, 2026 07:05
@kudai

kudai commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 176a7a426f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread crates/goose/src/agents/agent.rs Outdated
@kudai
kudai force-pushed the kudai/databricks-vision-followup branch from 176a7a4 to b24b3e3 Compare August 25, 2026 07:14
@kudai

kudai commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: b24b3e3b9d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@michaelneale

Copy link
Copy Markdown
Collaborator

🤖 Reviewed at b24b3e3b9. CI fully green (all 17 required checks pass). The core of this PR is warranted and the fail-open flip is correct — but there's one clobbering bug in the new Databricks helper that I'd fix before merge.

Blocking: with_resolved_vision_support destroys an explicit capability when the alias resolves to an unknown upstream

crates/goose-providers/src/databricks.rs:190-198:

if config.supports_vision.is_none() || effective_model_name != model_config.model_name {
    config.supports_vision = Self::model_supports_vision(effective_model_name);
}

model_supports_vision returns None for anything absent from the canonical registry. So when the endpoint is an alias (effective_model_name != model_config.model_name) and the resolved upstream is not in the catalog, an explicit Some(false) is overwritten with None — the user's deliberate setting is destroyed rather than refined by better data.

Combined with the fail-open in this same PR, that Some(false) case is a live regression: an endpoint explicitly marked text-only now yields Noneunwrap_or(true) → images get sent → provider 400. Pre-PR that config correctly stripped the image.

I proved it by compiling this against your head:

let alias = ModelConfig::new("team-chat").with_vision_support(false);
assert_eq!(alias.supports_vision, Some(false));
let resolved =
    DatabricksProvider::with_resolved_vision_support(&alias, "some-unlisted-endpoint");
assert_eq!(resolved.supports_vision, None); // passes — explicit Some(false) destroyed

Only overwrite when the lookup actually yields something. This keeps resolved-upstream precedence (both existing assertions in resolved_upstream_capability_takes_precedence_over_alias_canonicalization still hold) but never downgrades a known value to unknown:

if let Some(v) = Self::model_supports_vision(effective_model_name) {
    config.supports_vision = Some(v);
}

The new test at databricks.rs:840-848 only covers alias→known upstream, so this path is uncovered — worth adding the alias→unknown case with an explicit setting.

Non-blocking: rehydrate_canonical_capabilities no-ops when provider_name is None

crates/goose/src/model_config.rs:51-59 returns the config untouched when provider_name is None. The three agent-side callers pass session.provider_name.as_deref() (agents/agent.rs:1013, permission/permission_judge.rs:20, security/adversary_inspector.rs:23) — None for exactly the older/partially-initialised sessions this was added to repair. Such a session on openai/o3-mini keeps supports_vision: None, the gate now yields true, and the image is sent to a model that can't take it — an error that didn't happen pre-PR.

Note session/session_manager.rs:609 passes the live provider.get_name() instead. Using the live provider name (or falling back to Config::global().get_goose_provider()) at all four sites would close the hole and make the four call sites consistent.

Also worth a note

rehydrate_canonical_capabilities delegates straight to apply_canonical_limits, which also back-fills context_limit, max_tokens and reasoning, not just capabilities. A session that deliberately left context_limit unset for dynamic resolution now gets a pre-baked canonical value before provider.get_context_limit() at agent.rs:1834. Probably fine, but it's wider than the name suggests and isn't covered by the new test.

Confirmed good: the mime-type placeholders are consistent across all three formatters; all five gate sites were flipped (no stragglers); azure_foundry is correctly skipped; with_canonical_limits only writes into None fields so it can't clobber explicit settings; and the agent.rs refactor to model_config_for_session is behaviour-preserving with character-identical error strings.

@kudai
kudai force-pushed the kudai/databricks-vision-followup branch from b24b3e3 to 5dcca42 Compare August 26, 2026 03:10
@kudai

kudai commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

All three points addressed in 5dcca42:

Blocking — databricks clobber: with_resolved_vision_support now only overwrites when the canonical lookup yields a value (if let Some(v) = model_supports_vision(...)), so an explicit Some(false) on an alias that resolves to an unlisted upstream survives instead of being downgraded to None → fail-open. Added the uncovered case to the test: team-chat with Some(false) resolving to some-unlisted-endpoint stays Some(false). Both precedence assertions still hold.

Non-blocking — provider_name None: rehydrate_canonical_defaults (renamed from ...capabilities since it delegates to apply_canonical_limits and back-fills context_limit/max_tokens/reasoning too) now falls back to Config::global().get_goose_provider() when the caller passes None, closing the hole for older/partially-initialised sessions at all four call sites.

Note — wider than the name: covered by the rename above + the model_config_for_session_rehydrates_legacy_capabilities test now asserts context_limit() == 128_000 (gpt-4o), documenting that canonical defaults beyond capabilities are back-filled.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5dcca42ad7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread crates/goose-providers/src/databricks.rs
Serving endpoints may use arbitrary aliases (e.g. team-chat) whose metadata
resolves to an image-capable upstream model. Session config canonicalized
against the alias leaves supports_vision unset, so request formatting treated
the endpoint as text-only and silently omitted attached and tool-result images.

Derive supports_vision from the resolved upstream model (canonical registry)
in DatabricksProvider::stream before formatting the request, for both the
responses and chat-completions paths.
@kudai
kudai force-pushed the kudai/databricks-vision-followup branch from 5dcca42 to 05000a1 Compare August 28, 2026 04:20
…sion

Follow-up to aaif-goose#11496 (avoid sending images to non-vision models).

Asymmetry fix (michaelneale review): the detect_image_path text branch keeps
the path in the text so a downstream vision model can still act on it, while
an explicit Image block produced a dead placeholder. ImageContent carries only
base64 data + mime type (no path/URI survives), so the placeholders in
formats/openai.rs, formats/databricks.rs and formats/openai_responses.rs now
surface the mime type.

Design question (None semantics): unknown (None) vision capability is now
fail-open - images are sent, preserving pre-PR behavior for models absent from
the canonical catalog (ollama/llava, openai_compatible, vllm, llamacpp, custom
aliases). Only an explicit Some(false) strips images. Known non-vision catalog
models still resolve to Some(false) and are gated.

Codex review follow-ups:
- Rehydrate canonical capabilities for legacy saved configs in the shared
  Agent::model_config_for_session path (covers CLI turns + ACP/Desktop loads,
  not just the CLI builder), preserving the Azure Foundry skip.
- Databricks: when an endpoint alias resolves to a different upstream model,
  the resolved model's vision capability takes precedence over alias-based
  canonicalization (with_resolved_vision_support).

Databricks alias resolution (cherry-pick 685b8098e): model_supports_vision now
returns Option<bool> and leaves unknown resolved models as None so the gate
fail-opens, instead of converting them to Some(false).
@kudai
kudai force-pushed the kudai/databricks-vision-followup branch from 05000a1 to c32a6f3 Compare August 28, 2026 05:00
@kudai

kudai commented Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: c32a6f3d22

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants