Determinism isn't the main hurdle. It would be a moderate effort to engineer a deterministic claude that would be no more useful than the current one.
The key characteristic of a compiler is that it takes a source language with a well-defined meaning to a target language with a well-defined meaning.
The reason claude fails to be a compiler is that its input language (english) can't be precisely defined. (And even if that were possible, it would very often fail to preserve the meaning when "compiling")
This is a case of lethal trifecta. This particular one can be fixed by either not giving the AI private data, or by removing the exfiltration opportunity. Why does the comment-summary bot need access to your private video ids? Why does it need to be able to output links?
Most cases of prompt injection are harder to fix, and the success of the products they occur in relies on engineers who should know better sticking their heads in the sand about security risks.
Sadly, I suspect this will be the best piece of marketing they could ever hope for. "It's so advanced the government made us add extra security* to stop hackers!"
Surprisingly, I've found this works shockingly well (along with any plausible-sounding reason why it was wrong of the model to refuse) to "jailbreak" many models I've played with thus far. They're all just so eager to please...
Here's the threat model I (a luddite) use to evaluate these. The claude code harness can be mostly trusted, the model cannot be trusted because it is exposed to untrusted data from the internet, and there is no separation of data/code in an llm [0][1].
I want to avoid running untrusted code on my local machine, because it could steal secrets, install malware, etc.
Since the model is allowed to write without restriction (I think) to the project directory, anything in the project directory is also untrusted. Running standard commands from the system is fine, as long as you know what those commands are going to do. Running anything from the local directory should be avoided because the code is untrusted.
This is just one security model, there are many others! If a person is running claude in a stronger sandbox, that changes the model considerably. What threat model do you use to evaluate whether an agent's actions are safe?
The security layer needs to parse the full agent activity with the context. It watches everything, but only interrupts the human when it matters.
Commands that can run arbitrary code need to be treated differently and can't get escalated in this opaque way.
A large part of the solution should be to drastically reduce the amount of permission approval prompts a user gets. This ensures the ones he does get are evaluated with the same concentration a manager gives a new hire's most consequential decisions.
Most importantly, because we ask him rarely, when we do he feels the accountability. The yes is his.
that's a great point, and also the problem with relying on a human-in-the-loop to catch these kind of issues when it can be circumvented even if they were perfect
Don’t rely on your non deterministic agent and its creators to secure your software. Design defense in depth and trust guardrails that don’t expect Anthropic to vibe good security into existence.
If you start by treating any autonomous actor in your system as an actor with the potential to go rogue the design starts to create itself
Agents should make better use of OS sandboxing facilities with finer-grained ACLs.
Less: Do you want to run "npm run build"?
More: "npm run build" tried to read your Chrome cookie database, do you want to allow that?
Some agents like Codex use sandboxing on Linux/MacOS but the permissions are far too coarse - they'll run the command in a relatively strict sandbox and when it fails they'll ask you to allowlist the command as a whole, forever. There should be a new permission prompt every time a command tries to do something new.
Claude suggests (or used to suggest - it's been a while) to allowlist "bash" which completely defeats the point. If you do that the agent can run `bash -c "echo literally anything"`
Not using agents at all. It could edit your code to do something malicious when you run it. Not even once. Not even if the agent has a gun to your head.
assume no deep learning, of any kind is involved: you write a program, you are the author, right? you compile the code, are you still the author? do you have to attribute co-authorship to gcc/llvm/oracle?
i think not, you are still the author, same as when anyone else uses an llm to write code.
I think that's selling the theorems a little short. A math system with arithmetic is equal to, or more powerful than, a computer. For an example, even classical logic comes with the law of excluded middle that can say (internally) if a program halts or not. Incompleteness applies to all the stronger systems as well.
There is no logic that is more expressive than a Turing machine. In fact, just about every logic you know can only expressive necessarily terminating programs. There is a bit of an issue on what exactly someone means by expressive, but if we're talking programs that compute outputs from inputs (without caring about the invariants imposed on said programs) then this holds.
> Beginning January 2026, all ACM publications and related artifacts in the ACM Digital Library will be made open access.
reply