naildrivin5.com

Website of David Bryant Copeland

The Answer to Slop is Accountability

Get my weekly newsletter

“Slop” is low-quality output from an LLM. When slop is part of a user-facing product, it pulls down the quality of the product. The way to fix this is by consequences to product owner.

I don’t believe all LLM output is slop, though I understand that achieving high-quality output is largely the function of the LLM vendor’s models and not anything under my control. That being said, an LLM’s output as part of an overall product offering is under the control of people: product managers (primarily) and programmers (less so).

Terrible Features are Made by People

We’ve all experienced AI-powered product features that make the product worse. From macOS’s constant nagging you to “write with Siri1” to full-blown support chatbots that give poor advice, AI-powered product features are hit or miss at best.

Before AI, shitty product features fall to the product managers (PMs). Assuming the programmers built what was asked, then what was asked of them is the problem. That’s on the PM. I don’t see any difference now that AI is available as a tool for building features.

Imagine if a PM proposed a solution to reduce customer support cost. The only catch? It gave wrong answers sometimes and and there was no way to measure how wrong it was. That PM would have a tough time selling that solution to their superiors. Programmers probably wouldn’t want to build it. Who likes building software that doesn’t work by design?

That said, you could imagine some sort of cost/benefit analysis that showed some percentage of wrong answers were acceptable if it reduced support cost by some other amount. How many PMs did that analysis before AI? How many are doing it now?

Software and technology famously has no system of ethics or accountability. We are not a real profession. We don’t even hold companies accountable for quantifiably harmful security incidents, so we are a long way from holding PMs accountable for shitty user experiences brought on by AI slop. Say what you will about lawyers, but they can be held accountable.

Evaluate Your Feature Without Using the Term “AI”

A way I’d try to manage this would be to design features so that a user could understand and use them but without saying the feature was “AI-powered”. I’d want features that weren’t just a thin layer on top of a chatbot.

Consider the New York Times Cooking app’s proposal to add AI-powered recipe modification. Given the public sentiment in general, and the existing terrible outcomes regarding recipes, why would the product manager (or product designer) design this feature this way?

Is it useful to have a recipe app that can help with substitutions? Of course! We’ve all had to search the Internet for “what if I don’t have any cornstarch?” or whatever. Being able to do that in our recipe app and get structured results instead of Reddit posts seems good. So how do we explain the product idea without using the words “AI” or “LLM”?

The user types text describing the change the want to make. The system then responds with its own text describing your ingredients in detail and describes how the recipe will change. The user then sees a recipe updated with their change.

So far so mediocre. Nothing offensive, but my spidey sense is already tingling about a UI designed around a wall of text the user must read. But still, seems better than just searching the Internet. There’s more:

The proposed implementation will provide incorrect information some unknown percent of the time. We can’t measure this directly, however we can set expectations with a “beta” tag, ask for user feedback, and place text on the feature that says ‘our recommendations may contain mistakes’.

We’ve gone from a magical experience to something more questionable. Including the text “our recommendations may contain mistakes” is ridiculous2. Why would we ship a tool that we know makes mistakes? Why does someone cooking a meal want to follow a recipe that may contains mistakes?

Without using the terms “AI” or “LLM”, it becomes much more clear who is responsible for the behavior of the software: the New York Times. And that makes the potential mistakes worse. It’s not Sam Altman’s fault, it’s the NYT’s fault!

Only People Can Be Held Accountable

I would personally be embarrassed to ship the NYT Cooking app’s recipe substitutions feature without at least some way of measuring those mistakes so they could be managed and reduced. I would not pawn those mistakes off on someone else. I would own the decision to include a sometimes-wrong-technology in the feature. A PM bringing me this idea as-is would need some serious correction about what their job is.

Imagine a system of feedback and incentives where the outcomes of product features were tied to the performance evaluation of the PMs (and, where applicable, the programmers). Most companies don’t have anything like this, even pre-AI. This is the only way to curtail this nonsense.

Instead of rewarding PMs for just changing shit to get promoted, the efficacy of features they propose should be monitored and used as feedback for them to improve at their jobs. And if they are on the hook for low-quality AI output—slop—you can be sure there will start to be a lot less of it.

After all, if the New York Times tells me that I can make buttermilk biscuits with oil instead of butter, that’s on the New York Times, not some AI vendor whose product they voluntarily chose to integrate.

  1. 1I almost always select text on iOS to fix spelling mistakes brought on by autocorrect. Why on earth is “write with Siri” an option in that case? How many ways does Apple envision I can write two consecutive words that I need an LLM to help me?↩
  2. 2Slapping a “beta” tag on something is not a software beta test. It's as if an entire generation of product managers equates a “beta” tag with a free pass for the software not working. An actual beta test includes a subset of users who are expected to find and report problems. Just A/B testing them and seeing where they click is not that. ↩