BinaryTrajectoryComparisonConfig.comparison_temperature is silently discarded: the comparison query runs at the model's own temperature #1556

Describe the bug

BinaryTrajectoryComparisonConfig.comparison_temperature is documented as an override, but the value never reaches the API. Setting it has no effect: the pairwise-comparison (chooser) query runs at the model's own temperature instead.

The field and its contract, sweagent/agent/action_sampler.py:102-103:

comparison_temperature: float | None = None
"""Override the model's temperature. If None, take the temperature configured for the model."""

The call site that is supposed to apply it, sweagent/agent/action_sampler.py:288:

response = self._model.query(messages, temperature=self.config.comparison_temperature)["message"]  # type: ignore

Root cause. The call above passes temperature but not n. LiteLLMModel.query's signature is query(self, history: History, n: int = 1, temperature: float | None = None) (sweagent/agent/models.py:794), so n defaults to 1 and that 1 is forwarded to _query. _query only honours its temperature argument in the n is None branch (sweagent/agent/models.py:783-791):

def _query(
    self, messages: list[dict[str, str]], n: int | None = None, temperature: float | None = None
) -> list[dict]:
    if n is None:
        return self._single_query(messages, temperature=temperature)   # temperature honoured
    outputs = []
    # not needed for openai, but oh well.
    for _ in range(n):
        outputs.extend(self._single_query(messages))                   # temperature dropped
    return outputs

With n=1 the loop branch is taken and _single_query(messages) is called with no temperature, so _single_query falls back to its own default — temperature=self.config.temperature if temperature is None else temperature (sweagent/agent/models.py:725) — and litellm.completion receives the model temperature. The caller's intent is expressed at line 288 but never reached.

This is reachable from a plain config: agent.action_sampler: {type: binary_trajectory_comparison, comparison_temperature: 0.9} in a run YAML → DefaultAgentConfig.action_sampler (sweagent/agent/agents.py:162) → DefaultAgent._init (sweagent/agent/agents.py:493-494) → DefaultAgent.step calls self._action_sampler.get_action(...) (sweagent/agent/agents.py:1031-1034) → line 288 whenever >= 2 distinct parseable completions are sampled.

Steps/commands/code to Reproduce

FULL COMMAND (setup + run):

git clone https://github.com/SWE-agent/SWE-agent.git
cd SWE-agent
git checkout 3ea751c087f32b16e039a2233dd6eefecef325d5
python -m venv .venv-probe && .venv-probe/bin/pip install -e .
.venv-probe/bin/python repro_comparison_temperature_sampler.py

repro_comparison_temperature_sampler.py — drives the real BinaryTrajectoryComparison with a real LiteLLMModel, stubbing only the API boundary (litellm.completion) so no API key is needed, and records the temperature that actually reaches it. Call #1 and #2 are the two sampled candidate completions, call #3 is the pairwise-comparison query:

import sweagent.agent.models as M
from sweagent.agent.action_sampler import BinaryTrajectoryComparison, BinaryTrajectoryComparisonConfig
from sweagent.agent.models import GenericAPIModelConfig, LiteLLMModel
from sweagent.agent.problem_statement import EmptyProblemStatement
from sweagent.tools.parsing import Identity
from sweagent.tools.tools import ToolConfig, ToolHandler

TOOL_CONFIG = ToolConfig(parse_function=Identity())
TOOL_HANDLER = ToolHandler(TOOL_CONFIG)
MODEL_TEMPERATURE = 0.0
COMPARISON_TEMPERATURE = 0.9

calls: list[tuple[str, float | None]] = []


class _FakeMessage:
    def __init__(self, content: str):
        self.content = content
        self.tool_calls = None


class _FakeChoice:
    def __init__(self, content: str):
        self.message = _FakeMessage(content)


class _FakeResponse:
    def __init__(self, content: str):
        self.choices = [_FakeChoice(content)]
        self.usage = {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2}


_CONTENTS = ["action-a", "action-b", "judgement"]


def fake_completion(**kwargs):
    # calls 0,1 = the two sampled candidate completions; call 2 = the pairwise comparison query
    content = _CONTENTS[len(calls)] if len(calls) < len(_CONTENTS) else "judgement"
    calls.append((content, kwargs.get("temperature")))
    return _FakeResponse(content)


M.litellm.completion = fake_completion
M.litellm.cost_calculator.completion_cost = lambda *a, **k: 0.0
M.litellm.utils.token_counter = lambda *a, **k: 1

config = GenericAPIModelConfig(
    name="gpt-4o",
    temperature=MODEL_TEMPERATURE,
    per_instance_cost_limit=0,
    total_cost_limit=0,
)
model = LiteLLMModel(config, TOOL_CONFIG)

sampler_config = BinaryTrajectoryComparisonConfig(
    min_n_samples=2,
    max_n_samples=2,
    comparison_temperature=COMPARISON_TEMPERATURE,
    instance_template="{{problem_statement}}",
    comparison_template="{{thought1}}{{action1}}{{thought2}}{{action2}}",
)
sampler = BinaryTrajectoryComparison(sampler_config, model, TOOL_HANDLER)

out = sampler.get_action(
    problem_statement=EmptyProblemStatement(),
    trajectory=[],
    history=[{"role": "user", "content": "solve it"}],
)

print(f"configured model temperature      : {MODEL_TEMPERATURE}")
print(f"config.comparison_temperature     : {COMPARISON_TEMPERATURE}")
print(f"temperatures reaching litellm     : {[t for _, t in calls]}")
print(f"n candidate samples, then comparison calls: {[c for c, _ in calls]}")
print(f"sampler returned completion       : {out.completion['message']!r}")
assert len(calls) == 3, f"expected 2 samples + 1 comparison call, got {calls}"
assert all(t != COMPARISON_TEMPERATURE for _, t in calls[2:]), "comparison temperature was applied"
print("=> the comparison query (call #3) ran at the model temperature, not the configured override")

The same effect is visible one level lower without the sampler, by calling LiteLLMModel.query directly with the stub above:

model.query(hist, temperature=0.9)        # -> temperature reaching litellm: [0.0]
model.query(hist, n=1, temperature=0.9)   # -> temperature reaching litellm: [0.0]
model.query(hist, n=None, temperature=0.9)  # -> temperature reaching litellm: [0.9]  (documented branch)

Error message/results

Observed output (verbatim):

=== A) direct LiteLLMModel.query calls, temperature=0.9 ===
query(hist, temperature=0.9)             -> temperature reaching litellm: [0.0]
query(hist, n=1, temperature=0.9)        -> temperature reaching litellm: [0.0]
query(hist, n=None, temperature=0.9)     -> temperature reaching litellm: [0.9]
=== B) real sampler: BinaryTrajectoryComparison.get_action ===
temperatures reaching litellm     : [0.0, 0.0, 0.0]
VERDICT: DROPPED

A second, independent run recorded the (n, temperature) pair at the litellm.completion boundary, before and after a one-line fix:

B) query(n=None, temp=0.9)     -> litellm (n, temp) calls: [(None, 0.9)]
A) query(temp=0.9) default n   -> litellm (n, temp) calls: [(None, 0.0)]
C) sampler comparison (temp=0.9) -> litellm (n, temp) calls: [(None, 0.0), (None, 0.0), (None, 0.0)]
D) _single_query(temp=0.9)     -> litellm (n, temp) calls: [(None, 0.9)]
# after 1-line fix (n=None at action_sampler.py:288):
C) sampler comparison (temp=0.9) -> litellm (n, temp) calls: [(None, 0.0), (None, 0.0), (None, 0.9)]

All three calls report 0.0 — the model temperature — while comparison_temperature is 0.9.

Expected results. The pairwise-comparison query (call #3) should reach litellm.completion with temperature=0.9, per the field's own docstring at action_sampler.py:102-103 ("Override the model's temperature. If None, take the temperature configured for the model."), and per the working sibling branch inside the same class: LiteLLMModel._query's if n is None: return self._single_query(messages, temperature=temperature) (models.py:786-787), where _single_query does honour an explicit argument (temperature=self.config.temperature if temperature is None else temperature, models.py:725). The n=None, temperature=0.9 line above confirms that branch applies it correctly; the sampler just never takes it.

System Information

• OS: macOS 15 (darwin 25.6.0), arm64
• Python: 3.12 (.venv-probe), SWE-agent 1.1.0
• Commit: 3ea751c087f32b16e039a2233dd6eefecef325d5
• Install: pip install -e . from the repo (development version)
• Location: sweagent/agent/action_sampler.py:288, sweagent/agent/models.py:783-794

Checklist

• I'm running with the latest docker container/on the latest development version (i.e., I ran git pull))
• I have copied the full command/code that I ran (as text, not as screenshot!)
• If applicable: I have copied the full log file/error message that was the result (as text, not as screenshot!)
• I have enclosed code/log messages in triple backticks (docs) and clicked "Preview" to make sure it's displayed correctly.

Related: #1484 touches the same _query loop (it reports that n is not forwarded to the provider, so n samples re-send the full history). It is a distinct defect — this report is about temperature being dropped on the sampler path — but a fix to either one has to touch that same _query branch, so the two are worth considering together.

Happy to open a PR: either the one-line caller fix (self._model.query(messages, n=None, temperature=self.config.comparison_temperature), since query returns result[0] when n is None), or fixing LiteLLMModel._query to pass temperature=temperature into self._single_query(messages) inside the for _ in range(n) loop so the documented override holds for every sampled call. Say which you'd prefer and I'll send it.