This week, researchers at the ELLIS Institute Tübingen and the Max Planck Institute published a finding that should give any business relying on AI tools pause: the encrypted “reasoning” that models like Claude, GPT and Gemini generate behind the scenes wasn’t as private as their providers assumed. Anthropic, OpenAI and Google all protected these internal reasoning traces with a single, shared encryption key across their model families. Researchers found they could capture an encrypted reasoning block from a powerful model, feed it to a smaller, less-guarded sibling model, and have that smaller model transcribe the “private” thinking back out in plain text.
The scale is the part worth pausing on. By running this technique against 315,000 reasoning blocks that had been shared publicly on GitHub and Hugging Face, researchers recovered 182 real credentials — 62 live API keys, 33 passwords, and 30 personal email addresses that people had unknowingly typed into AI tools and then posted online as part of code samples or debugging logs. All three providers have since patched the flaw server-side. Anything shared publicly before the patch, however, remains readable by anyone who knows the method.
Why this matters even if you don’t build with AI APIs
Most UK SMEs don’t call these models directly through code — you use them through a chat interface, a Copilot plugin, or a SaaS tool built on top. Directly, this specific vulnerability doesn’t touch you. But it’s a useful, concrete reminder of something easy to forget in the day-to-day convenience of AI tools: anything you type into an AI system travels through infrastructure with its own bugs, and “encrypted” is not the same as “impossible to expose.” Reasoning traces were assumed private by design, by three of the most security-conscious AI labs in the world, and that assumption was wrong for months before anyone caught it.
The credentials recovered in this research came from people pasting code — including live API keys and passwords — directly into AI chat sessions or debugging transcripts, then sharing the output publicly without realising what was embedded in it. That’s an entirely ordinary, easy mistake, not a sophisticated attack on the business doing it.
What to actually check this week
Audit what your team pastes into AI tools. If staff are dropping customer data, credentials, or internal documents into ChatGPT, Copilot, or Claude to get quick help, that’s a habit worth a policy, not a ban. A one-page guide on what’s safe to paste and what isn’t takes an hour to write and prevents most of this category of risk.
Never let API keys or passwords sit in code you show an AI tool, even briefly. Treat any text box connected to an AI model the way you’d treat a public forum post — assume it could eventually be read by someone other than you, and rotate any credential that ever touched one, just in case.
Don’t let this put you off using AI tools. The providers responded fast, patched server-side, and the underlying models remain some of the best security research in the industry. The lesson isn’t “AI is unsafe” — it’s that AI infrastructure has bugs like any other software, and treating it with the same caution you’d apply to any third-party service is the right instinct.
If you want a proper AI usage policy for your team rather than an ad hoc one, BuildApps helps UK SMEs put sensible guardrails around AI adoption without killing the productivity gains. And if you’re running AI-powered tools that were built or configured in-house, CoolCoding can review the technical implementation for exactly this kind of overlooked exposure.
The takeaway
A flaw that let anyone decode “private” AI reasoning traces has been patched, but it exposed a truth worth keeping: AI tools are software, and software has security bugs. Set a simple rule for what your team pastes into AI chat windows, rotate anything sensitive that’s already gone through one, and treat AI vendors with the same scrutiny you’d give any other supplier handling your data.
