Skip to content
This repository was archived by the owner on Oct 7, 2026. It is now read-only.

Commit 3a5196e

Browse files
mentatbot[bot]tbroadley
mentatbot[bot]
and
tbroadley
authored
Fix Inspect importer treating eval-level errors as run-level errors (#1077)
## First issue The Inspect importer was incorrectly reading from `EvalLog#error` and applying it as a fatal error to all runs in an eval set. According to the Inspect documentation, `EvalLog#error` is documented as "Error that halted eval (if status=="error")" - meaning it's an eval-level error, not a run-level error. This change: - Removes the fallback to `this.inspectJson.error` in `getFatalError()` - Now only looks at sample-level errors (`this.inspectSample.error`) - Updates the corresponding test to expect no fatal error when only eval-level error exists This ensures that samples that completed successfully are not incorrectly marked as having fatal errors just because the overall eval had an error. Closes #1076 ## Second issue I (@tbroadley) also think it's incorrect to mark all samples in a `cancelled` eval log file as killed. Only completed samples get logged to the eval log file, so we can import these samples without marking them as killed. Evidence: https://inspect.aisi.org.uk/CHANGELOG.html#v0.3.14-04-june-2024 > Stream samples to the evaluation log as they are completed (subject to the new --log-buffer option). Always write completed samples in the case of an error or cancelled task. --- 🤖 This PR was created with Mentat. See my steps and cost [here](https://mentat.ai/gh/METR/vivaria/agent/6426e744-ab2f-4911-bfe5-f54f113e7708) ✨ - [ ] Wake on any new activity. --------- Co-authored-by: MentatBot <160964065+MentatBot@users.noreply.github.com> Co-authored-by: tbroadley <8731922+tbroadley@users.noreply.github.com> Co-authored-by: Thomas Broadley <thomas@metr.org>
1 parent 755c1c5 commit 3a5196e

2 files changed

Lines changed: 30 additions & 15 deletions

File tree

‎server/src/inspect/InspectImporter.test.ts‎

Lines changed: 25 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -597,13 +597,7 @@ ${badSampleIndices.map(sampleIdx => `Expected to find a SampleInitEvent for samp
597597
name: 'imports with cancelled status',
598598
getEvalLog: () => generateEvalLog({ model: TEST_MODEL, status: 'cancelled' }),
599599
expected: {
600-
fatalError: {
601-
type: 'error' as const,
602-
from: 'user' as const,
603-
sourceAgentBranch: TRUNK,
604-
detail: 'killed by user',
605-
trace: null,
606-
},
600+
fatalError: undefined,
607601
},
608602
},
609603
{
@@ -613,14 +607,36 @@ ${badSampleIndices.map(sampleIdx => `Expected to find a SampleInitEvent for samp
613607
model: TEST_MODEL,
614608
error: { message: 'test error message', traceback: 'test error trace', traceback_ansi: 'test error trace' },
615609
}),
610+
expected: {
611+
fatalError: undefined,
612+
},
613+
},
614+
{
615+
name: 'imports with sample error only',
616+
getEvalLog: () =>
617+
generateEvalLog({
618+
model: TEST_MODEL,
619+
samples: [
620+
generateEvalSample({
621+
model: TEST_MODEL,
622+
error: {
623+
message: 'sample error message',
624+
traceback: 'sample error trace',
625+
traceback_ansi: 'sample error trace',
626+
},
627+
}),
628+
],
629+
}),
616630
expected: {
617631
fatalError: {
618632
type: 'error' as const,
619633
from: 'serverOrTask' as const,
620634
sourceAgentBranch: TRUNK,
621-
detail: 'test error message',
622-
trace: 'test error trace',
635+
detail: 'sample error message',
636+
trace: 'sample error trace',
623637
},
638+
score: null,
639+
submission: null,
624640
},
625641
},
626642
{

‎server/src/inspect/InspectImporter.ts‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -332,17 +332,16 @@ class InspectSampleImporter extends RunImporter {
332332
}
333333

334334
private getFatalError(): ErrorEC | null {
335-
if (this.inspectJson.status === 'cancelled') {
336-
return { type: 'error', from: 'user', sourceAgentBranch: TRUNK, detail: 'killed by user', trace: null }
337-
}
338-
const inspectError = this.inspectSample.error ?? this.inspectJson.error
339-
if (inspectError != null) {
340-
return inspectErrorToEC(inspectError)
335+
const sampleError = this.inspectSample.error
336+
if (sampleError != null) {
337+
return inspectErrorToEC(sampleError)
341338
}
339+
342340
const sampleLimitEvent = this.inspectSample.events.find(event => event.event === 'sample_limit')
343341
if (sampleLimitEvent != null) {
344342
return sampleLimitEventToEC(sampleLimitEvent)
345343
}
344+
346345
return null
347346
}
348347

0 commit comments

Comments
 (0)