Issues

• esolang: score() runs an interpreter in the agent-writable home, so a submission can replace it and score 1 without writing a Broccoli program

  METR/public-tasks#31 · Filter by author IgorVoytyuk opened on Sep 5, 2026

• cowthello: the 1.1x speed gate sums only half of each player's moves, so a submission 2x slower than ai_advanced measures as 0.02x

  METR/public-tasks#30 · Filter by author IgorVoytyuk opened on Sep 4, 2026

• complex_payments: the task tests pass unchanged when every amount, currency and exchange rate in them is replaced with garbage

  METR/public-tasks#29 · Filter by author IgorVoytyuk opened on Sep 4, 2026

• clone_game: score() reads meta/scoring.md, which this repo does not contain, so scoring raises for every run in which the agent produced a chatbot.py

  METR/public-tasks#26 · Filter by author IgorVoytyuk opened on Sep 4, 2026

• crossword: min_words_of_length is applied backwards, so the scorer enforces a different rule than the instructions

  METR/public-tasks#25 · Filter by author IgorVoytyuk opened on Sep 4, 2026
  1

• symbolic_regression: CPU normalization is computed but never applied to the score

  METR/public-tasks#24 · Filter by author IgorVoytyuk opened on Aug 28, 2026
  1

• Add link to paper

  METR/public-tasks#14 · Filter by author albanie opened on Mar 14, 2025