An AI system at a retailer sent wrong refund and cancellation notices to more than 18,000 customers in under six hours. The cause, according to Bloomberg’s report on a new Wharton study, was a mistake in a prompt.
At a consumer-electronics company, a pricing system put a flagship product on sale at 85% off. It stayed that way for three hours, and more than 700 orders came in.
Nobody sabotaged anything. These were ordinary slips running at machine speed. What stood out in the study came afterward: asked who should answer for errors like these, the people surveyed couldn’t agree.
I think that split matters more than any single figure in the survey. Companies are handing agents real authority faster than they are deciding who owns the result.
Two ordinary mistakes
Bloomberg doesn’t name either company, and I couldn’t find the incidents described anywhere else, so I’m giving them as reported.
Take the retailer first. A prompt went wrong, the system did exactly what it was told, and it did so for 18,000 people before anyone stopped it. A person sending those emails by hand would have given up after a few dozen. Software doesn’t get tired or embarrassed.
The pricing case is the same story with a different bill. Seven hundred orders at a deep discount leaves the company choosing between eating the loss and disappointing customers who already clicked buy.
The coverage doesn’t say what checks either company had. My reading is narrower: both mistakes ran for hours, which suggests nothing was placed to catch them sooner.
Nobody agrees who answers
Wharton ran the survey with GBK Collective and published it on Tuesday. When they put the question to 1,416 U.S. enterprise leaders and employees, the answers came back in nearly equal piles. About a quarter said the employee who deployed the AI. About a quarter said that employee’s manager. About a quarter said the risk team. Roughly one in ten pointed to the vendor, and one in ten said they didn’t know (Crypto Briefing).
Now picture that disagreement inside a company with 18,000 wrong emails already out. Each group has a fair claim and a reason to point somewhere else, and the first hours go on arguing instead of fixing.
The employee nearest the keyboard tends to take the hit, and that is usually unfair. The person who typed the prompt probably didn’t pick the tool, set its permissions or decide it could act without review.
Courts won’t wait
In 2024 a British Columbia tribunal decided Moffatt v. Air Canada, after the airline’s website chatbot gave a customer wrong information about bereavement fares. Air Canada argued the chatbot was responsible for its own statements. The tribunal didn’t accept that and held the airline responsible. The sum was small, but the reasoning applies to any company whose software talks to customers.
Only about one in ten survey respondents blamed the vendor, which seems about right to me. Customers deal with the company that deployed the agent, whatever the contract says about the supplier.
Most organizations can tell you who owns the payments system. Far fewer can say who owns the agent that touches payments and customer email.
Five safeguards
The coverage points to five. They’re sensible, and the two incidents show why.
Start with ownership.
Every agent gets one named person who answers for what it does. If the retailer’s notices had gone wrong under that rule, there would have been an obvious person to call.
Tie autonomy to risk.
Drafting refunds, price changes, payments and customer emails is fine, provided a person approves them before they go out. Low-stakes tasks can run on their own. A price change on a flagship product isn’t one of them.
Limit access.
An agent should get the permissions and data its task needs and nothing more. The moment it can reach email and customer databases, a small error has somewhere to go.
Keep Monitoring
Log every decision and every exception, and keep a shutdown switch that someone can reach in minutes. Under six hours is a long time to find out.
Settle in advance who answers for what
Employee, manager, risk team or vendor. The survey shows what the default looks like when nobody has.
The better question
“Who takes the blame” gets asked after the damage. A more useful question comes earlier: who was supposed to make sure this couldn’t happen?
If a company can answer with a name and a mechanism, it is most of the way there. If the answer is a shrug, or four different names, it probably controls its agents less than it thinks.
Writing down who answers for each agent takes far less time than an incident does.