I gave an AI tens of thousands of products to fix and one rule. It never touches the shop.
The shop sells home textiles from about a dozen suppliers. Their data arrives with missing colours, sizes and materials, and with titles that nobody would ever search for. Missing attributes mean the shop’s filters miss products, and bad titles mean search does too. Fixing it by hand is not a plan.
Early on, the model suggested 200x220 cm for a duvet cover whose description had no size at all. It was marked high confidence, and the evidence quote looked convincing, except the number wasn’t in it. It was the most common size in the category, a sensible guess dressed up as a fact. That was the moment I decided the AI would never write to the shop directly.
Everything else in this post grew out of that rule. It comes in three parts: making the answers trustworthy, making it practical across the whole catalogue, and running it safely from day to day.
Making the answers trustworthy
The AI suggests, a person decides
The model is a great intern. It’s fast, it never gets tired, and it isn’t bored by product number 14,327. It’s also just as confident when it’s wrong.
The tempting design is to let it write straight into the shop and fix mistakes later. I didn’t, because “later” never comes. A wrong value in a live shop spreads into filters, feeds and search results before anyone notices it.
So every answer becomes a suggestion. A person accepts it, edits it or throws it away. A person’s value always wins, and nothing automatic can overwrite a field someone has edited. A wrong suggestion costs one click.
Every value needs a source
For every value, the model also returns where it found it and how sure it is. Here’s one, with the quote translated.
"dimensions": {
"value": "180x250",
"evidence": "Duvet cover set, size 180x250, soft-washed cotton (translated)",
"confidence": "high"
}The evidence is for the reviewer. They don’t reread a whole supplier description. They read the quote and decide.
But a quote can be made up too, as the duvet cover showed. So every number in a suggestion must appear in the supplier’s text or in a person’s edit. If it doesn’t, the suggestion is rejected.
// Simplified.
function numbersAreInSource(string $suggestion, string $source): bool
{
preg_match_all('/\d+(?:[.,]\d+)?/', $suggestion, $matches);
foreach ($matches[0] as $number) {
// Whole numbers only, so "25" doesn't pass because the source says "250".
$whole = '/(?<![\d.,])' . preg_quote($number, '/') . '(?![.,]?\d)/';
if (!preg_match($whole, $source)) {
return false;
}
}
return true;
}The real version also treats “1,5” and “1.5” as the same number. It’s still only a few lines, and it catches the most dangerous mistake an AI makes with product data: a believable number that nobody wrote. A wrong colour looks wrong. A wrong size looks fine until the duvet cover doesn’t fit.
It has a clear limit, though. It proves that a number exists in the source, not that it belongs to this field. A set whose description lists 200x220 for the duvet cover and 50x70 for the pillowcases gives the model two real numbers to mix up, and the check passes both. That’s one reason a person still reviews every suggestion.
Let code say no, let people say yes
The facts check is one of a few hard checks. A suggestion is rejected when a size or composition has the wrong format, when a category doesn’t exist or sits under the wrong parent, or when a new title drops the original name’s model words, piece count or dimensions.
Everything else is a note, not a rejection. A meta description that’s too long, low confidence, a title that doesn’t follow the usual pattern, or a claim that isn’t in the supplier text all get flagged for the reviewer. Code is good at “never”. People are better at “probably fine”.
Automation is earned
Autonomy gets added the way you’d give it to a new colleague.
There is a daily run that makes suggestions on its own. For each supplier, it switches on only after a person has run one full pass by hand and every answer from it is in. It also leaves out titles and categories.
Auto-apply exists too. It’s set per field and off by default. Even when it’s on, it won’t overwrite a person’s edit or a clean size, dimensions, composition or colour from the supplier, and it skips any answer flagged for adding facts that aren’t in the source.
Making it practical at scale
Ask for less
Every question you ask the model is another chance for it to be wrong.
If a person has already set a field, the model isn’t asked about it. If the model’s answer matches what the supplier already says, the answer is dropped. I’d like to go one step further and not ask about fields the supplier already fills cleanly, but today the model still gets asked and the duplicate is thrown away afterwards.
We learned the same lesson from the other side. The models think before they answer by default, and that thinking counts against the output limit. On our cheapest model, it ate the answer. In one pass, about 4 in 10 answers were cut off, and that model used about three times as much output per answer as the most expensive one. Now each model’s output limit is sized from what it actually uses in production, and thinking can be switched off per model.
One queue message per job
The obvious design is one queue message per product. Ten thousand products, ten thousand messages, and the queue does the rest. I didn’t do that.
Every AI job is one queue message that runs one command. A job is a run someone starts from the admin, the batch collect or a supplier’s daily run. A single worker takes them one at a time.
The reason is coordination. With a message per product, every message has to check the budget, find its run, update progress and decide what to do when its siblings fail. With one message per job, the job owns its locks, its budget and its progress in one place, and the parallel part becomes a detail inside it.
This is a choice for this system, not a rule. If every product were independent and nothing shared a budget, a message per product would be fine, and the queue would give you retries and scaling for free. Here, every call spends from the same daily limit and belongs to a run someone is watching, so I wanted one owner.
Warm the cache, then go parallel
When someone wants results now, the job asks the first product alone. That request writes the prompt cache, which holds the rules and the category tree, the same for every product.
If you skip that, the first eight parallel requests all miss the cache and all write it. On our most expensive model, those eight cache writes were 18% of a run’s cost. That’s one measurement on one model, but it was enough to keep the warm-up.
Then the job starts up to eight child processes. Each one loops.
- Claim a product with
SKIP LOCKED. - Under a lock, check today’s spending plus this call’s worst case against the daily limit, and reserve the worst case as a hold.
- Release the lock.
- Call the API.
- Record the answer with its actual cost, and drop the hold.
Claiming work with SKIP LOCKED
Every child needs the next pending product, and no two children may take the same one. SKIP LOCKED does that in one statement, without children waiting on each other.
-- Simplified.
UPDATE job_item
SET status = 'claimed'
WHERE id = (
SELECT id FROM job_item
WHERE job_id = :job AND status = 'pending'
ORDER BY id
LIMIT 1
FOR UPDATE SKIP LOCKED
)
RETURNING id;A row another child is claiming is locked, so this query skips it and takes the next one.
A budget that holds under concurrency
“Check, then call” is a race. All eight children check, all eight see room, and all eight spend. So the check and the reservation happen together under a lock, and the lock is released before the call, so no child waits on another one’s HTTP request.
// Simplified.
$lock->acquire();
try {
// Recorded costs today + open holds + batch items not yet collected.
$spent = $budget->spentToday();
$worstCase = $prices->worstCase($request);
if ($spent + $worstCase > $budget->dailyLimit()) {
throw new BudgetReached();
}
$hold = $holds->create($request, $worstCase); // counts as spent from now on
} finally {
$lock->release();
}
try {
$answer = $client->send($request); // outside the lock
$answers->record($answer, $answer->cost()); // the real cost now counts
} finally {
$holds->delete($hold); // and the hold stops counting
}It works like a hotel holding your card. While the call runs, the hold counts as spent. When the answer arrives, the real cost is recorded and the hold is dropped. For a moment the budget sees both, which errs on the safe side, but it never sees neither. So the children can’t overshoot the limit together.
When it can wait, batch it
Work that can wait goes through the provider’s batch lane instead. The pass is stored before anything is sent, then submitted in chunks, so if the process dies halfway, the database knows what went out.
A collect job is scheduled every 15 minutes but runs only if a batch is open. Each result is recorded and marked processed in one transaction, so collecting can stop, resume and never record an answer twice. Batch answers go through the same checks as instant ones. There is one way in.
Knowing what it costs
Every request stores its tokens, cache reads and writes included, and what it cost at list prices. It’s our own number, not the invoice, but it’s per product and per prompt version, which the invoice never is.
Before each pass is submitted, its worst case is checked against what’s left of the daily limit, using token counts from a sample of its requests. The batch lane costs half the price, so anything that isn’t urgent goes there. And by default, passes only look at products that are on sale. A product nobody can buy doesn’t need a better title.
Running it safely
Version the prompt, fingerprint the inputs
Prompts change. You spot a mistake and you add a rule. Three weeks later, nobody knows which answers came from which rules.
So the prompt is versioned like code. Every rule change bumps the version and gets one line about what changed. Next to it, every answer stores a fingerprint of the supplier text it was made from. A person’s edits are left out of it on purpose, so an edit doesn’t count as new supplier text.
That gives us two things. First, a pass, the daily run included, asks about a product again only when the prompt got better or the supplier text changed, so it never pays twice for the same answer. Second, when the supplier text changes under an open suggestion, the review page marks it as stale. The reviewer knows the ground moved under it.
Show me what the model saw
When an answer looks wrong, the first question is always “what did the model actually see?”. So there’s a command that prints the exact request for one product, URL, headers and body, without sending it. It needs no API key and spends nothing, and it answers that question in seconds.
Failures stay small
There are two kinds of retry, and they’re deliberately different.
Jobs are never retried by the queue. If a run fails, it’s parked as failed, and the next scheduled run is the retry.
Single API calls are retried, because a busy API is normal. When it answers 429, 503 or 529 during a run, the child asks about the same product again, up to five tries, waiting 2, 4, 8 and then 16 seconds. That sits on top of the client library’s own retries. A busy API should slow a run down, not fail it.
The daily run is also kept apart from the supplier sync. It’s a separate job, so an AI failure can’t change the result of the sync.
Watch what happens downstream
Even good changes need watching. Titles are part of the product URL, so a better title moves the page. The shop keeps every old URL and 301-redirects it to the new one. And the review page leaves titles unticked by default, because accepting one changes the URL. If your titles feed your URLs, have the redirects in place before you improve the titles.
Did it work?
Yes. Products that used to go live with half their details missing now have colours, sizes and materials, so the shop’s filters actually filter. Titles read like something a customer would type into a search box.
The review changed the most. A reviewer checks a suggestion against its quote instead of rereading the supplier’s text. Most suggestions get accepted with one click or a small edit. Reviewing a supplier’s catalogue went from a job nobody wanted to something you can do over a coffee.
The safeguards earn their keep quietly. The facts check still catches the occasional invented number, which is exactly what it’s there for. Stale suggestions get flagged instead of applied over newer supplier text. And the budget is checked before money is spent, not added up at the end of the month.
It isn’t finished. The facts check proves a number exists, not that it’s the right one, so review stays. We still ask about fields the supplier already fills cleanly. And I haven’t run a proper evaluation against a hand-checked set yet, so “most suggestions” is what reviewers see, not a measured rate.
What I believe now
I thought a good enough prompt would eventually make review unnecessary. It didn’t. A better prompt made review faster, never optional.
The model turned out to be the smallest part of the system. Most of the work went into what sits around it: checks that say no, a review that says yes, records that show what it saw, and a budget it can’t outrun. The AI suggests, code checks and a person decides.
Next we’re trying a local open model behind the same versioned prompt, the same checks and the same human review. If it’s worse, the review will show it, the same way it shows every other mistake. That experiment is only cheap to try because of everything above.