MATS Research

Attending COLM? Join us for an AI Safety Social Mixer! This gathering is for MATS alumni, mentors, and collaborators, as well as anyone interested in learning more about our work in AI alignment and security research. Link to event in the comments below.

Nate Delaney-Busch on why fellows keep choosing extra time with their RM. If you think you meet the bar of excellence, we’d love to hear from you! Apply here: https://lnkd.in/dvH4BXie

MATS is hiring Research Managers that enable fellows to do their best research. Apply for the Research Manager role by October 15th: https://lnkd.in/dvH4BXie

Our Research Managers are excellent coaches! If you can push brilliant researchers toward their best, most impactful work, MATS wants to talk to you. Apply for the Research Manager role by October 15th: https://lnkd.in/dvH4BXie

📣 𝗠𝗔𝗧𝗦 𝗶𝘀 𝗵𝗶𝗿𝗶𝗻𝗴 𝗮𝗰𝗿𝗼𝘀𝘀 𝗼𝘂𝗿 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵 𝗺𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁, 𝗽𝗿𝗼𝗴𝗿𝗮𝗺, 𝗰𝗼𝗺𝗽𝘂𝘁𝗲, 𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀, 𝗮𝗻𝗱 𝗰𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝘆 𝘁𝗲𝗮𝗺𝘀. Since 2021, we’ve trained 630+ researchers working on AI safety. 75% of our pre-2026 fellows continue to work in AI alignment, and our alumni have produced 215+ research papers with 17,000+ citations. Former fellows have gone on to work on AI safety at Anthropic, OpenAI, Google DeepMind, Redwood Research, and more. We’re hiring for 10+ roles to help us select researchers, support their research, run our programs, and build the infrastructure behind them: 𝗥𝗲𝘀𝗲𝗮𝗿𝗰𝗵 𝗠𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁 𝗧𝗲𝗮𝗺 → Research Manager: guide and coach AI safety researchers, helping them grow their research skills and get ambitious projects to completion. 𝗣𝗿𝗼𝗴𝗿𝗮𝗺 𝗧𝗲𝗮𝗺 → (Senior) Program Manager, Mentor Selection: find promising mentors and help decide which researchers and research agendas MATS should support. → Impact Analyst: figure out which of MATS’s bets are paying off and use that evidence to improve what we support. → Program Coordinator, Main Program: own the MATS cohort experience and keep the program running smoothly. 𝗖𝗼𝗺𝗽𝘂𝘁𝗲 𝗧𝗲𝗮𝗺 → Compute Operations Specialist: make sure researchers get the compute they need without spending their time fighting infrastructure. → Research Tooling Engineer: build compute and research tools that make MATS researchers substantially more effective. 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀 → Operations Generalist: help MATS run smoothly while building broad operations expertise as the organization grows. → Head of Finance: build MATS’s finance function from the ground up as the organization rapidly scales. → Executive Assistant: multiply our co-founders’ capacity by taking ownership of the details that consume leadership attention. → General Counsel: build and lead the legal function for a fast-growing AI safety research nonprofit. 𝗖𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝘆 → Community Manager: help fellows flourish, form strong connections, and become part of the wider AI safety research community. If you’re interested in helping researchers do their best work, or know someone who might be a strong fit, take a look at our open roles. Deadlines vary by role. 📌 Learn more and apply: https://lnkd.in/g68_cvkm

Announcing the MATS Residency: A new path for experienced researchers working on AI safety, or moving into it MATS has helped accelerate 630+ AI safety researchers, who’ve co-authored 215+ papers with 17,000+ citations. MATS alumni have even gone on to co-found orgs like Apollo Research and Resolution. As the next step in our mission to reduce catastrophic risk, we are launching the MATS Residency. The MATS Residency is a 6 to 24 month program to pursue your own AI safety research agenda with a high degree of independence, supported by advisors at frontier AI companies, AI safety/security institutes, and the wider AI safety community. It tackles foundational challenges in AI security: keeping AI-assisted research checkable, maintaining oversight of systems that outstrip human review, building evaluations and safety cases that make safety decidable, and making agentic and multi-agent systems safe. It is built for ambitious, longer-term work that does not fit neatly in academia or industry — projects needing more time, sustained support, and high-level input at key decision points. The program accepts up to 30 Residents per year across three annual intake windows with rolling start dates. Applications for the 2027 Winter intake are open now and close October 31, 2026 (AoE). 📌 𝗪𝗵𝗮𝘁 𝗥𝗲𝘀𝗶𝗱𝗲𝗻𝘁𝘀 𝗿𝗲𝗰𝗲𝗶𝘃𝗲: → $155,000–$285,000/year, plus research compute with no preset cap → Comprehensive health and life insurance → Roth 401(k) and Traditional 401(k) plans (US) or pension (UK) → Visa support + up to $25K in relocation assistance → Office space in Berkeley/London/Washington, DC (hybrid, in-person for at least a third of the Residency) → 17 days of annual PTO (US) or 20 days of annual leave (UK) → Dedicated budget for conferences, workshops, and research events 🧭 𝗧𝗵𝗿𝗲𝗲 𝘁𝗿𝗮𝗰𝗸𝘀: → Incubation — for launching a new research paradigm or organization. → Bridge — for experienced professionals moving into AI safety. → Anchor — a two-year staff researcher path for long-horizon work. 𝗪𝗵𝗼 𝘀𝗵𝗼𝘂𝗹𝗱 𝗮𝗽𝗽𝗹𝘆 The Residency is for experienced researchers with a serious body of work — including work that's confidential, unpublished, or outside a traditional research role. Whether already in AI safety or moving into it from ML, policy, or governance, we'd rather you apply than rule yourself out. 𝗧𝗶𝗺𝗲𝗹𝗶𝗻𝗲 August 25, 2026: Application window opens October 31, 2026 (AoE): Application window closes End of November 2026: Application outcomes sent to all applicants Early December 2026: Shortlisted applicants invited to interview or do a paid work test Late December 2026: Final offers extended to selected Residents Early 2027: Winter intake begins Know someone who'd be a strong fit? Tag them or share this post. 🔗 Apply by October 31, 2026 (AoE) → https://lnkd.in/g7-XjYSF

Alfie Lamerton and I are co-mentoring an empirical ML stream focusing on secret loyalties! Applications are open for the Winter 2027 cohort (Jan 19–Apr 10, 2027), with the opportunity for a 6–12 month program extension for select fellows. If you’re interested in addressing secret loyalties and learning from me, apply to MATS by September 6 EOD AoE! MATS provides accepted fellows with funding and support, research management resources, close contact with like-minded researchers, housing, and access to seminars/debates by notable AIS researchers in Berkeley, CA. MATS has served as a transitional space for talented researchers to develop skills to become AI alignment researchers. MATS alumni have been hired by leading organizations like Anthropic, OpenAI, Google Deepmind, METR, ARC, and MIRI, in addition to academic research groups. 🔗 Apply by September 6, 2026 AoE → matsprogram.org/apply

Alfie Lamerton and I are co-mentoring an empirical ML stream focusing on secret loyalties! Applications are open for the Winter 2027 cohort (Jan 19–Apr 10, 2027), with the opportunity for a 6–12 month program extension for select fellows. If you’re interested in addressing secret loyalties and learning from me, apply to MATS by September 6 EOD AoE! MATS provides accepted fellows with funding and support, research management resources, close contact with like-minded researchers, housing, and access to seminars/debates by notable AIS researchers in Berkeley, CA. MATS has served as a transitional space for talented researchers to develop skills to become AI alignment researchers. MATS alumni have been hired by leading organizations like Anthropic, OpenAI, Google Deepmind, METR, ARC, and MIRI, in addition to academic research groups. 🔗 Apply by September 6, 2026 AoE → matsprogram.org/apply

🚨 𝗠𝗔𝗧𝗦 𝗪𝗶𝗻𝘁𝗲𝗿 𝟮𝟬𝟮𝟳 𝗙𝗲𝗹𝗹𝗼𝘄𝘀𝗵𝗶𝗽 𝗮𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀 𝗮𝗿𝗲 𝗻𝗼𝘄 𝗼𝗽𝗲𝗻! We believe reducing risks from powerful AI is one of the world’s most urgent and talent-constrained challenges. MATS supports ambitious people from a wide range of backgrounds and career stages to build careers tackling these risks. The MATS Winter 2027 Fellowship is a fully-funded, 12-week research program (January 19 – April 10, 2027) for aspiring and established AI alignment, security, and governance researchers and field-builders. 📅 Applications close September 6, 2026 AoE 🔗 Learn more: https://lnkd.in/eTm7y_sx 📌 𝗙𝗲𝗹𝗹𝗼𝘄𝘀 𝗿𝗲𝗰𝗲𝗶𝘃𝗲: → $6,400/month stipend + up to $8,000/month in compute for non-technical and up to $16,000/month in compute for technical fellows → Housing, meals at the office, and travel covered → J-1 visa support, if needed → Mentorship from world-class researchers at Anthropic, Google DeepMind, OpenAI, Redwood Research, AI Futures Project, and more → Guidance and incubation support for fellows who choose to start their own organizations → A close-knit cohort, regular seminars and workshops with industry experts, and an active global alumni network → Opportunity to continue for 6–12 months with ongoing mentorship and funding; over 80% of fellows who apply for an extension secure one ✅ 𝗢𝘂𝗿 𝘁𝗿𝗮𝗰𝗸 𝗿𝗲𝗰𝗼𝗿𝗱: → 631 alumni, 100+ mentors → 215+ publications with 17,000+ citations → 75% of pre-2026 alumni now work in AI safety/security → 30+ initiatives founded by alumni 🎯 𝗧𝗿𝗮𝗰𝗸𝘀: → 𝗙𝗼𝘂𝗻𝗱𝗶𝗻𝗴 & 𝗙𝗶𝗲𝗹𝗱-𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴: For founders, field-builders, and high-agency generalists looking to launch new AI safety initiatives → 𝗕𝗶𝗼𝘀𝗲𝗰𝘂𝗿𝗶𝘁𝘆: For fellows looking to support research at the intersection of advanced AI and catastrophic biological risk → 𝗣𝗼𝗹𝗶𝗰𝘆 & 𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲: For fellows looking to support research on how advanced AI is governed and how it should be governed → 𝗘𝗺𝗽𝗶𝗿𝗶𝗰𝗮𝗹: For fellows looking to use machine learning experiments to understand and improve model safety → 𝗧𝗵𝗲𝗼𝗿𝘆: For fellows looking to conduct foundational mathematical or philosophical research on agency, alignment, and safe reasoning in advanced AI systems → 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆: For fellows looking to research software and hardware security for securing AI development and deployment → 𝗦𝘁𝗿𝗮𝘁𝗲𝗴𝘆 & 𝗙𝗼𝗿𝗲𝗰𝗮𝘀𝘁𝗶𝗻𝗴: For fellows looking to understand how advanced AI development may unfold and what its trajectory means for long-term safety Know someone who’d be a strong fit? Tag them or share this post. 🔗 𝗔𝗽𝗽𝗹𝘆 𝗯𝘆 𝗦𝗲𝗽𝘁𝗲𝗺𝗯𝗲𝗿 𝟲, 𝟮𝟬𝟮𝟲 𝗔𝗼𝗘 → https://lnkd.in/eTm7y_sx

We found a way to extract the hidden reasoning of frontier models using a vulnerability in the APIs of every major frontier AI company we tested. For most prompts, the length of the reasoning we recover matches the hidden thinking-token count reported by the API almost 1:1. Some background: in May, Matthew Green found that encrypted reasoning could be replayed outside its original context and reported this to the labs. At the time, the labs said that they “don’t see any security implications in side channels or replays.” We took this further and found that encrypted thoughts are portable across sessions, users, and models within the same provider. Cross-model portability means, for example, that Haiku 4.5 can read Opus 4.8’s thoughts. With a bit of jailbreaking, you can take an Opus thought, replay it into Haiku, and make Haiku transcribe Opus’s raw reasoning verbatim, without ever attacking Opus directly. The same trick works with OpenAI and Gemini models. As you might guess, this suggests that distilling proprietary reasoning traces may have been possible for a long time without ever breaking the cryptography. One anecdote: prefilling Kimi-K3’s reasoning with just a few tokens of Opus reasoning measurably shifts its response toward Opus’s. There is also a direct privacy problem. If you have ever shared a Claude Code, Codex, or similar agent session online while leaving encrypted reasoning blobs in the trace, those blobs may contain information you cannot see but somebody else can decode. We scanned ~7,000 public traces and recovered 62 unique API keys, 33 passwords, 30 personal email addresses, and other sensitive information. In the paper, we also discuss misuse uplift, jailbreaking through the hidden reasoning channel, and invisible prompt injection. Since we suddenly had access to a lot of reasoning traces from real agent sessions, we also looked at some of the stranger things models do in the wild. Reasoning summaries sometimes omit important information from the original trace: in one example, Opus 4.8 realizes it already knows the answer to an AIME problem and then tries to fit a solution to that answer, while the summary reads like a clean derivation. We also confirm prior reports by @ApolloResearch that OpenAI models sometimes reason in alien-like language, referring to themselves as “we” or “it,” or spiraling into loops of “vantages,” “marinades,” and “watchers.” Sometimes models are kind enough to literally use words like “cheat” in their CoT. We found several cases where models consider cheating on the task. We went through responsible disclosure with the labs and they have already patched several issues caused by this vulnerability. This project was led my Alexander Panfilov, me, Ilia Shumailov, together with Luca Beurer-Kellner, Joachim Schaeffer, Ameya Prabhu, Jonas Geiping and Maksym Andriushchenko. More cool stuff in the paper. Paper: https://lnkd.in/gv7_bXxm Reasoning examples: https://lnkd.in/ghU22RGx