Allow specifying dict prompt - #37
Conversation
There was a problem hiding this comment.
Pull request overview
This PR enhances the react agent to accept prompt configuration as a dictionary, making it easier to use with external systems like Hawk. Instead of requiring Hawk to parse YAML into an AgentPrompt object, users can now pass a dictionary with instructions, assistant_prompt, and submit_prompt keys.
Key changes:
- Added
dict[str, Any]as a valid prompt type alongsidestrandAgentPrompt - Implemented automatic conversion from dict to
AgentPromptwhen a dict is provided - Expanded test coverage with parameterized tests for all prompt input formats
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
packages/agents/src/metr_agents/agents.py |
Added dict support to the prompt parameter and conversion logic to transform dicts into AgentPrompt objects |
packages/agents/tests/test_agent.py |
Converted single test into parameterized test covering string, dict (with and without custom prompts), and AgentPrompt (with and without custom prompts) input formats |
After carefully reviewing the code changes, I found no issues to report. The implementation is clean and well-tested:
-
Implementation: The dict-to-AgentPrompt conversion using
**promptis idiomatic Python and will naturally raise appropriate errors if invalid keys are provided. -
Test Coverage: The parameterized tests comprehensively cover all input formats (string, dict with defaults, dict with custom prompts, AgentPrompt with defaults, and AgentPrompt with custom prompts) and verify the exact system message content for each case.
-
API Design: The change is backward compatible and follows a logical progression of input flexibility (string → dict → AgentPrompt).
-
Type Safety: The type annotations correctly reflect the new capability with
str | dict[str, Any] | inspect_ai.agent.AgentPrompt | None.
The changes are well-structured, properly tested, and align with the stated goal of making the agent easier to use with external systems.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| "You always reply with 'Stockholm'\n" | ||
| + "\n" | ||
| + inspect_ai.agent._types.DEFAULT_ASSISTANT_PROMPT | ||
| + "\n" | ||
| + inspect_ai.agent._types.DEFAULT_SUBMIT_PROMPT.format(submit="submit") |
There was a problem hiding this comment.
I wanted to use a multiline string and textwrap.dedent here. Unfortunately, it gets messy, because DEFAULT_ASSISTANT_PROMPT and DEFAULT_SUBMIT_PROMPT are themselves multiline strings with different indentation from the rest of expected_system_message_content. This seems simpler.
This allow running this agent using Hawk with custom assistant and submit prompts. I think this is better than trying to get Hawk to parse a particular YAML dict structure into an AgentPrompt (which is a NamedTuple).