A sold item does not 404

6 October 2026 · Fedor Dyatlov

I was watching the price of a caravan in a spreadsheet. Every morning it refreshed, and every morning the number had changed:

run 1   €87,500
run 2   €28,250
run 3   €31,595

Nothing was wrong with the scraper. The caravan had been sold.

What the site actually does

I expected a dead listing to return 404. Almost none of them do. What happens instead, on dealer sites in particular, is a redirect:

GET /stock/caravan-450-anniversary-996305/
301 -> https://www.some-other-dealer.nl/

Sometimes to the dealer's own home page, sometimes to the listing page the item sat on, and — this one surprised me — sometimes to a different company's home page, because the dealership was acquired and the whole domain now points at the buyer.

Every one of those destinations is a real page, returns 200, and is full of prices. My recipe asked for the first element matching a price selector, found one on the first card of whatever stock list it landed on, and returned it with a straight face.

Why this is worse than an error

If the scraper had returned nothing, the cell would have been empty and the person would have looked. If it had returned an error, they would have looked sooner.

Instead it returned a plausible number. Not an absurd one — a real caravan price, from a real caravan, just not theirs. For somebody monitoring a price, that is the worst available output: it reports a change that did not happen, and it never once says the thing that actually happened, which is sold.

A wrong number is more expensive than a missing one, because a missing one gets noticed.

The rule I settled on

After all redirects, compare the path you asked for with the path you landed on. If the destination is the site root, or any ancestor of what you asked for, the item is gone:

/stock/caravan-450-996305/  ->  /              gone
/stock/caravan-450-996305/  ->  /stock/        gone
/stock/caravan-450-996305/  ->  other-site.nl/ gone

The host is deliberately not part of the test. A dealer whose stock moved to a sister company redirects across domains, and the item is just as gone.

What must keep working is every ordinary redirect, and the useful property is that none of them shortens the path:

http -> https                    same depth, fine
adding or dropping www           same depth, fine
a trailing slash                 same depth, fine
a renamed slug                   same depth, fine
/shop/thing -> /thing            shorter, but not an ancestor, fine

That last one matters. /thing is shorter than /shop/thing, but it is not a prefix of it at a segment boundary, so it reads as a site reorganising rather than an item disappearing. Checking for "shorter" alone would have broken on every such move.

What the caller gets

HTTP has a status code for exactly this and almost nobody uses it. The API answers 410 Gone, with the destination in the body so the reason is visible:

410 Gone
the page now redirects to the site's home page
(https://www.some-other-dealer.nl/); the item is no longer listed

Nothing is charged, nothing is cached as a recipe, and no model is asked to make sense of a home page. A crawl steps over one vanished item instead of ending.

The general version

The bug was not about caravans. It was about treating a 200 as evidence that you are looking at what you asked for.

A redirect is the server telling you the thing you wanted is somewhere else — and sometimes the honest translation of "somewhere else" is "nowhere". Anything that fetches URLs on a schedule and compares today's answer to yesterday's should be suspicious of a 200 that arrived at a different address than the one it asked for.

Mine wasn't.