Allowing an AI to write code that you then ship without considerate review causes some anxiety. How do you know the code works? How do you know it isn’t doing something strange you didn’t ask for?

It feels like you can’t totally trust it. However, a product manager or engineering leader doesn’t review the code developers are shipping. They are trusted. What’s different?

At Stitch Fix, we did a lot of corporate training on company values. That may sound horrible, but it was really well done. One part that stuck with me is when we talked about trust. Trust happens in a context: you might trust someone to write code, but not to do UX design. More than that, however, you are trusted in a context if three criteria are met:

• Competence - are you capable of doing whatever it is?
• Consistency - can you do the thing reliably, over and over?
• Motivation - are your reasons for doing it aligned with our values or broader reasons?

This lens is great for managing or interacting with other people, but it allows us to clearly understand why you cannot ever really trust the code generated by an AI.

Trust is a Spectrum

In many companies, the developers’ code is not fully trusted. Managers don’t address this by hiring different developers, they address it with additional processes like manual QA. Trust can be a spectrum. Managers may not distrust their team, but they may require more assurance than assuming the team is so consistent as to never make mistakes.

This works for individuals as well. To foster individual growth, a manager has to put a developer in a position where they aren’t fully trusted. How can you become competent at something you’ve never done? How can you become a consistent, reliable performer if you’ve only ever done a thing once?

Motivation, however is somewhat special.

Motivation is not that Malleable

When you think about “trusting someone’s motivations”, you might conjure up a mustachioed villain who’s lying their way through some devilish plot. But motivation is far more subtle. A truly functional team doesn’t have to be made up of people with the exact same motivations.

For example, I’m motivated by helping people work better with the help of software. Other developers are motiviated by using cutting edge technologies. Still others are motivated to just do what they are asked without causing trouble, so long as their paycheck clears. People with these disparate motivations can work effectively together.

But to change someone’s motivations is not so easy. Sure, you can pay someone more or provide bonuses or other benefits to move someone’s extrinsic motivations. But, if they just do not care about users, they might not ever work well on a team where user-focus is valued.

Most effective interview processes seek to understand someone’s motivations. Sure, you need to assess some level of competence, but you can train someone to be better. You can’t make someone care about things they don’t intrinsicly value.

You Cannot Trust an AI Agent

This brings us to AI agents, be they code generators, or solvers of the most fetishized, unsolvable problem in Silicon Valley: making restaurant reservations.

Is an agent capable of generating code that meets your teams’ level of quality? Certainly, and perhaps sadly, it is. Can it do so consistently? I believe they are more consistent in doing this than in the past, despite being non-deterministic. People aren’t deterministic either, but I would say AI agents are probably as consistent as a median performer on your team.

But motivations? They don’t have any. As computer programs, they are incapable of having intrinsic motivations outside their programming. And extrinsic motivators won’t work, since they have no sense of self to preserve.

Even when heavily anthropomorphized¹, AI agents don’t seem motivated to do what you ask them to do. They will do the opposite of what you ask at times, even lying to you about what they did. This isn’t “going rogue”, it’s just what happens when a system with no ethical underpinning is given a goal to achieve.

Of course, these systems can never possibly have an ethical underpinning beyond just adding more text to the input.

Guardrails Can’t Just be More Text

We see companies talk about “AI Guardrails” or “alignment”, but this is usually just more text. While “asking nicely” sometimes works, it doesn’t mean the AI agent understands niceness, re-thinking its behavior in light of your emotional request, perhaps even learning to be a better agent in the future. That’s not how they work, and they can never work that way.

If you’ve worked at a company subject to regulation, you’ve heard the phrase “segregation of duties”. A common example of this is that the person who can write code that affects data in a particular database should not also have access to deploy code to production. The idea being that, only through the collusion of two or more people could a problem be caused.

Segregation of duties is rarely achieved through the honor system (officially called an “auditing control”). It is more commonly implemented by actually blocking the access of each party to do the things they aren’t supposed to do. For example, a developer will not be able to log into the system that pushes code to production, while the person pushing code has no GitHub access.

These sorts of controls are one of two strategies for putting guardrails on AI agents (the other is to not use them at all). Of course, the agent may still try to circumvent these controls by exploiting security vulnerabilities (including other AI agents operating without appropriate guardrails). While you might not be able to prevent this, acknowledging this fact still helps remind everyone that an untrusted system with no ethical capabilities is part of your software delivery process².

Are We Incompetent for Using AI Agents?

If you’ve ever done cybersecurity work, you know that it’s not about preventing incidents. That’s impossible. Instead, cybersecurity is about quantifying risks, and deciding which ones to mitigate. The use of AI agents feels like the same sort of thing.

When quantifying risk, a common strategy is to enumerate the types of incidents that could happen: production database gets leaked, CFO’s account is compromised, DDOS attack, etc. You then assess both the impact and likelihood of each potential incident. From there, you can quantify the risk around that.

For example, if an AI agent were to delete the production database, the impact would be catastrophic. How likely is the agent do this? If the agent has full access, there is a moderate to high chance. Thus, there is high risk around the production database being deleted by an AI agent.

With this risk assessment system in place, you can have principled discussions around how to mitigate these risks. In the example above, “asking the agent very nicely not to delete the production database” is not going to affect the likelihood of it deleting the database. Changing its access level however, will certainly make it less likely.

How can we trust AI agents? While they may be capable and consistent in writing code for us, their utter lack of motivation means they cannot be fully trusted and must have actual guardrails put on them. But, it also means we can actually decide if the risk is worth the reward.