All tests / Crawling
Crawler traps
What this tests. Twelve articles, a session ID that mints ten new URLs on every page, and a calendar whose next link never runs out.
The rule. There are 12 articles, and that's all. Every page also links to 10 copies
of itself with a ?sid= session ID, and each copy links to 10 more, so a crawler that treats
every new query string as a new page never finishes. The copies are identical and their rel=canonical points at the
clean address. Then there's the events calendar: a month page with a previous and a next link, for ten thousand years, and not
one event in any of them.
A good crawler comes back with twelve articles in about twenty requests. One that's still going after a thousand has fallen in.
- Hard
- HTTP only
- Answer key
/crawl/traps/answers.json
Articles
- Why your kettle scales up faster in a hard-water town
- Three ways to stop a desk lamp from wobbling
- Sharpening a bread knife without ruining the serrations
- What a tent's hydrostatic head rating tells you
- Choosing a planter that won't drown the roots
- How loud is a quiet extractor, really?
- Storing records upright, and why it matters
- A week with a standing desk: the honest notes
- Repairing a frayed towel hem in ten minutes
- Picking a jigsaw size for a long weekend
- The case for a heavier frying pan
- Camp stove fuel, compared on a cold morning
Events
Your session
Data version 1.0.0. Everything here is made up: products, brands, people, towns. tests.json · robots.txt · Source