---

title: "Setting Limits"

llms-description: Setting time, message, token, and cost limits on evaluation tasks, samples, and agent execution.

---

## Overview

In open-ended model conversations (for example, an agent evaluation with tool usage) it's possible that a model will get "stuck" attempting to perform a task with no realistic prospect of completing it. Further, sometimes models will call commands in a sandbox that take an extremely long time (or worst case, hang indefinitely).

For this type of evaluation it's normally a good idea to set limits on some combination of total time, total messages, turns, tokens used, and/or cost. This article covers:

1. [Sample Limits](#sample-limits) --- limits applied to individual samples within a task.

2. [Scoped Limits](#scoped-limits) --- limits applied to arbitrary blocks of code.

3. [Agent Limits](#agent-limits) --- limits applied to agent execution.

## Sample Limits {#sample-limits}

Sample limits don't result in errors, but rather an early exit from execution (samples that encounter limits are still scored, albeit nearly always as "incorrect").

### Time Limit

Here we set a `time_limit` of 15 minutes (15 x 60 seconds) for each sample within a task:

``` python

@task

def intercode_ctf():

return Task(

dataset=read_dataset(),

solver=[

system_message("system.txt"),

use_tools([bash(timeout=3 * 60)]),

generate(),

],

time_limit=15 * 60,

scorer=includes(),

sandbox="docker",

)

```

Note that we also set a timeout of 3 minutes for the `bash()` command. This isn't required but is often a good idea so that a single wayward bash command doesn't consume the entire `time_limit`.

We can also specify a time limit at the CLI or when calling `eval()`:

``` bash

inspect eval ctf.py --time-limit 900

```

Appropriate timeouts will vary depending on the nature of your task so please view the above as examples only rather than recommend values.

### Working Limit

{{< include _working_limits.md >}}

Here we set an `working_limit` of 10 minutes (10 x 60 seconds) for each sample within a task:

``` python

@task

def intercode_ctf():

return Task(

dataset=read_dataset(),

solver=[

system_message("system.txt"),

use_tools([bash(timeout=3 * 60)]),

generate(),

],

working_limit=10 * 60,

scorer=includes(),

sandbox="docker",

)

```

### Message Limit

{{< include _message_limits.md >}}

Here we set a `message_limit` of 30 for each sample within a task:

``` python

@task

def intercode_ctf():

return Task(

dataset=read_dataset(),

solver=[

system_message("system.txt"),

use_tools([bash(timeout=120)]),

generate(),

],

message_limit=30,

scorer=includes(),

sandbox="docker",

)

```

This sets a limit of 30 total messages in a conversation before the model is forced to give up. At that point, whatever `output` happens to be in the `TaskState` will be scored (presumably leading to a score of incorrect).

### Token Limit

{{< include _token_limits.md >}}

Here we set a `token_limit` of 500K for each sample within a task:

``` python

@task

def intercode_ctf():

return Task(

dataset=read_dataset(),

solver=[

system_message("system.txt"),

use_tools([bash(timeout=120)]),

generate(),

],

token_limit=(1024*500),

scorer=includes(),

sandbox="docker",

)

```

::: callout-important

It's important to note that the `token_limit` is for all tokens used within the execution of a sample. If you want to limit the number of tokens that can be yielded from a single call to the model you should use the `max_tokens` generation option.

:::

### Turn Limit

{{< include _turn_limits.md >}}

Here we set a `turn_limit` of 300 for each sample within a task:

``` python

@task

def intercode_ctf():

return Task(

dataset=read_dataset(),

solver=[

system_message("system.txt"),

use_tools([bash(timeout=120)]),

generate(),

],

turn_limit=300,

scorer=includes(),

sandbox="docker",

)

```

This limits the agent to 300 model generations before it is forced to give up. As with other sample limits, whatever `output` happens to be in the `TaskState` at that point will be scored.

### Cost Limit

{{< include _cost_limits.md >}}

Here we set a `cost_limit` of $2.00 for each sample within a task:

``` python

@task

def intercode_ctf():

return Task(

dataset=read_dataset(),

solver=[

system_message("system.txt"),

use_tools([bash(timeout=120)]),

generate(),

],

cost_limit=2.00,

scorer=includes(),

sandbox="docker",

)

```

::: callout-important

The `cost_limit` requires model cost data to be configured via `set_model_cost()` or `--model-cost-config`. An error will be raised if a cost limit is set without cost data for all models used in the evaluation.

:::

#### Model Cost {#model-cost}

Cost tracking requires cost data for each model present in the eval or eval set. There are two ways to set cost data:

**Python API:**

``` python

from inspect_ai.model import set_model_cost, ModelCost

set_model_cost("openai/gpt-4o", ModelCost(

input=2.50, output=10.00,

input_cache_write=0, input_cache_read=1.25,

))

```

**CLI (YAML or JSON file):**

Each model needs a price set for `input`, `output`, `input_cache_write`, and `input_cache_read`. Prices should be given in dollars per million tokens. Set unused fields to `0`.

Below is an example cost config file given in YAML:

``` yaml

openai/gpt-4o:

input: 2.50

output: 10.00

input_cache_write: 0

input_cache_read: 1.25

anthropic/claude-sonnet-4-5-20250514:

input: 3.00

output: 15.00

input_cache_write: 3.75

input_cache_read: 0.30

```

(As of Feb 9 2026, all major model providers count reasoning tokens as output tokens, so no separate price needs to be provided for reasoning tokens. If your use case requires separate calculation of reasoning token prices, contact us.)

When model cost data is configured, costs will be tracked for the sample as a whole, as well as any events within the sample that have a ModelUsage field.

Additionally, configuring model cost data allows setting sample cost limits:

``` bash

inspect eval ctf.py --model-cost-config pricing.yaml --cost-limit 2.00

```

### Custom Limit

When limits are exceeded, a `LimitExceededError` is raised and caught by the main Inspect sample execution logic. If you want to create custom limit types, you can enforce them by raising a `LimitExceededError` as follows:

``` python

from inspect_ai.util import LimitExceededError

raise LimitExceededError(

"custom",

value=value,

limit=limit,

message=f"A custom limit was exceeded: {value}"

)

```

### Query Usage

We can determine how much of a sample limit has been used, what the limit is, and how much of the resource is remaining:

``` python

sample_time_limit = sample_limits().time

print(f"{sample_time_limit.remaining:.0f} seconds remaining")

```

Note that `sample_limits()` only retrieves the sample-level limits, not [scoped limits](#scoped-limits) or [agent limits](#agent-limits).

## Scoped Limits {#scoped-limits}

You can also apply limits at arbitrary scopes, independent of the sample or agent-scoped limits. For instance, applied to a specific block of code. For example:

``` python

with token_limit(1024*500):

...

```

A `LimitExceededError` will be raised if the limit is exceeded. The `source` field on `LimitExceededError` will be set to the `Limit` instance that was exceeded.

When catching `LimitExceededError`, ensure that your `try` block encompasses the usage of the limit context manager as some `LimitExceededError` exceptions are raised at the scope of closing the context manager:

``` python

try:

with token_limit(1024*500):

...

except LimitExceededError:

...

```

The `apply_limits()` function accepts a list of `Limit` instances. If any of the limits passed in are exceeded, the `limit_error` property on the `LimitScope` yielded when opening the context manager will be set to the exception. By default, all `LimitExceededError` exceptions are propagated. However, if `catch_errors` is true, errors which are as a direct result of exceeding one of the limits passed to it will be caught. It will always allow `LimitExceededError` exceptions triggered by other limits (e.g. Sample scoped limits) to propagate up the call stack.

``` python

with apply_limits(

[token_limit(1000), message_limit(10)], catch_errors=True

) as limit_scope:

...

if limit_scope.limit_error:

print(f"One of our limits was hit: {limit_scope.limit_error}")

```

### Checking Usage

You can query how much of a limited resource has been used so far via the `usage` property of a scoped limit. For example:

``` python

with token_limit(10_000) as limit:

await generate()

print(f"Used {limit.usage:,} of 10,000 tokens")

```

If you're passing the limit instance to `apply_limits()` or an agent and want to query the usage, you should keep a reference to it:

``` python

limit = token_limit(10_000)

with apply_limits([limit]):

await generate()

print(f"Used {limit.usage:,} of 10,000 tokens")

```

### Time Limit

To limit the wall clock time to 15 minutes within a block of code:

``` python

with time_limit(15 * 60):

...

```

Internally, this uses [`anyio`'s cancellation scopes](https://anyio.readthedocs.io/en/stable/cancellation.html). The block will be cancelled at the first yield point (e.g. `await` statement).

### Working Limit

{{< include _working_limits.md >}}

To limit the working time to 10 minutes:

``` python

with working_limit(10 * 60):

...

```

Unlike time limits, this is not driven by `anyio`. It is checked periodically such as from `generate()` and after each `Solver` runs.

### Message Limit

{{< include _message_limits.md >}}

Scoped message limits behave differently to scoped token limits in that only the innermost active `message_limit()` is checked.

To limit the conversation length within a block of code:

``` python

@agent

def myagent() -> Agent:

async def execute(state: AgentState):

with message_limit(50):

# A LimitExceededError will be raised when the limit is exceeded

...

with message_limit(None):

# The limit of 50 is temporarily removed in this block of code

...

```

::: callout-important

It's important to note that `message_limit()` limits the total number of messages in the conversation, not just "new" messages appended by an agent.

:::

### Token Limit

{{< include _token_limits.md >}}

To limit the total number of tokens which can be used in a block of code:

``` python

@agent

def myagent(tokens: int = (1024*500)) -> Agent:

async def execute(state: AgentState):

with token_limit(tokens):

# a LimitExceededError will be raised if the limit is exceeded

...

```

The limits can be stacked. Tokens used while a context manager is open count towards all open token limits.

``` python

@agent

def myagent() -> Solver:

async def execute(state: AgentState):

with token_limit(1024*500):

...

with token_limit(1024*200):

# Tokens used here count towards both active limits

...

```

::: callout-important

It's important to note that `token_limit()` is for all tokens used *while the context manager is open*. If you want to limit the number of tokens that can be yielded from a single call to the model you should use the `max_tokens` generation option.

:::

#### Suspending Token Limits

To run a block of code that should not count against any active token limits, use `suspend_token_limit()`:

``` python

with token_limit(10_000):

await generate() # counts against the 10k budget

with suspend_token_limit():

# tokens used here are not metered against the 10k limit,

# and any inner `token_limit()` is also suspended

await expensive_summary()

await generate() # counts again

```

Unlike `with token_limit(None):`, which only suppresses the innermost limit's check, `suspend_token_limit()` fully disables both recording and checking across all active token limits for the duration of the block.

### Turn Limit

{{< include _turn_limits.md >}}

To limit the total number of turns (model generations) which can be used in a block of code:

``` python

@agent

def myagent(turns: int = 300) -> Agent:

async def execute(state: AgentState):

with turn_limit(turns):

# a LimitExceededError will be raised if the limit is exceeded

...

```

Like token limits, turn limits can be stacked — a generation counts towards all open turn limits. To run generations that should not count against any active turn limit, use `suspend_turn_limit()`:

``` python

with turn_limit(10):

await generate() # counts against the 10 turn budget

with suspend_turn_limit():

# generations here are not metered against the 10 turn limit

await auxiliary_generate()

await generate() # counts again

```

### Cost Limit

{{< include _cost_limits.md >}}

To limit the total cost within a block of code:

``` python

@agent

def myagent(budget: float = 2.00) -> Agent:

async def execute(state: AgentState):

with cost_limit(budget):

# a LimitExceededError will be raised if the limit is exceeded

...

```

Cost limits work similarly to token limits, with stacking and tracking of costs used while the context manager is open.

::: callout-important

Using `cost_limit()` requires model cost data to be configured via `set_model_cost()` or `--model-cost-config`. See [Model Cost](#model-cost) for details.

:::

## Agent Limits {#agent-limits}

{{< include _agent_limits.md >}}