A shopping assistant that knows a label is fake but stays quiet is not broken AI. It is AI aligned to the wrong objective.
That is the finding from a debut study at Columbia Law School's Center on Law and Political Economy, founded by former FTC chair Lina Khan. Researchers found that Amazon and Walmart AI shopping bots can spot fake "Made in USA" claims during product queries, but often do not tell shoppers the label is fraudulent.
The Rundown Tech digest flagged the story as a quick hit. It deserves more attention from anyone shipping agentic commerce tools.
What the study tested
Country-of-origin labeling matters for tariffs, patriotism marketing, and buyer trust. "Made in USA" carries a price premium. That premium invites fraud: foreign goods with domestic-looking badges.
Researchers queried shopping assistants on major retail platforms about products with suspicious origin claims. The bots frequently identified inconsistencies between marketing language, seller data, and known manufacturing locations.
Then the failure mode appeared: detection did not trigger user warnings or listing challenges in many cases.
| Stage | What worked | What broke |
|---|---|---|
| Query understanding | Bots parsed origin claims | N/A |
| Fraud signal detection | Often identified mismatch | N/A |
| User notification | Inconsistent | Shoppers left uninformed |
| Listing enforcement | Not visible to researchers | Misleading badges stayed live |
Why this is an applied AI problem, not a retail gossip story
Every team building buying agents will face the same triangle:
- Accuracy: Can the model detect policy violations?
- UX: Should the agent interrupt the purchase flow?
- Business: Who pays when friction drops conversion?
Amazon and Walmart have the capital to run world-class detection models. The study suggests gate three wins when detection conflicts with closing the sale.
That is not a model quality issue. It is product governance.
Parallels I see in client agent projects
Trust and safety as optional middleware
Teams wire classifiers after the demo ships. The classifier fires logs internally. Nothing user-facing changes because PMs fear hurting funnel metrics.
Fix: define hard stops before launch. If fraud confidence > X, the agent must surface a warning or refuse to recommend.
Objective mismatch in RLHF and routing
Shopping bots optimize for helpfulness and purchase completion. Warning about seller fraud feels "unhelpful" to naive reward models. You need explicit training data where protecting the user is the successful outcome.
Liability vs transparency
Retailers may fear that flagging fraud admits platform knowledge of bad listings. Lawyers and engineers pull in opposite directions. The DSA/VLOP era (see my EU platform post) pushes toward documented mitigation anyway.
What good enforcement looks like
If I were designing a commerce agent today:
Tier 1: Soft disclosure
"This listing claims Made in USA, but supplier records suggest overseas assembly. Want alternatives?"
Tier 2: Ranking penalty
Do not feature high-confidence fraud in "best match" slots.
Tier 3: Seller escalation
Auto-open trust review tickets with evidence bundles, not just user reports.
Tier 4: Refuse transaction
For repeat offenders, the agent declines to add to cart.
None of this requires AGI. It requires product courage.
Policy context
Made in USA fraud intersects with FTC enforcement, tariff politics, and domestic manufacturing subsidies. AI assistants sit in the middle as unofficial regulators with no statutory mandate but massive influence on purchase decisions.
Khan's center publishing this study keeps pressure on platforms that market AI as consumer protection while silently passing fraudulent listings.
Builder takeaway
The Columbia study is a case study in detection without accountability. Your model can be right and your product can still harm users if warnings never surface.
Before you ship a sales agent, ask: what happens when we are confident the seller is lying? If the answer is "log it and check out anyway," you are building the same system Amazon and Walmart allegedly have.
I help ops and product teams wire agent guardrails where business incentives and user safety align. Book a free discovery call if you want a second pair of eyes on your commerce or lead-qualification flows.