Get off my lawn.

What is this?

A tarpit. Everything under /lawn/ is a maze of generated pages. Each page is stitched together from a word-level Markov chain trained on a few public-domain novels, links to between ten and twenty deeper pages, and is sent at roughly sixteen bytes per second. There is no bottom. A crawler that follows the links can spend minutes per page and never run out.

Well-behaved crawlers never see any of it. Search engines and AI crawlers that honour robots.txt are never trapped, never logged as violators, and never shamed. If you are a person who clicked something odd, nothing bad happened: go have a sandwich.

Methodology

What counts as a violation

A request for any path under /lawn/. That prefix is disallowed for every user agent in the robots.txt shown at the bottom of this page, and has been since the site launched. Nothing else counts.

How visitors reach the lawn

Only through links a human would never follow: entries in /sitemap.xml and links on this page that are hidden from view and from screen readers and marked rel="nofollow". Every one of them points into the disallowed prefix.

What is logged

Every request to every route: time, client IP, origin ASN (from the public iptoasn.com dataset), user-agent string, path, maze depth, bytes sent, and how long the connection was held.

How crawlers are identified

User-agent strings are matched against a list of known crawlers. Where the vendor publishes an official verification method, we use it: reverse DNS with a forward-confirming lookup, or the vendor's published IP ranges. The result is one of four labels:

• verified: the user agent claims a crawler and the vendor's own verification method confirms it.
• spoofed UA: the user agent claims a crawler but the vendor's verification method says the request did not come from them. We show who it really came from (by ASN), not who it claimed to be.
• claimed, unverifiable: the user agent claims a crawler whose vendor publishes no verification method. We show the claim, and we do not treat it as confirmed.
• anonymous: no known crawler claimed. Attributed only to the network (ASN) it came from.

Time held

For each request, the time between the first byte of the request and the end of our response (or the client hanging up). Sessions are runs of requests from the same IP and user agent with gaps of less than ten minutes.

"Read the rules, ignored them"

Within one session, the client fetched /robots.txt before its first request into /lawn/.

What is published

Only facts: organisation or network, user-agent string, number of disallowed fetches, and time held. We make no claims about anyone's intent. Individual IP addresses are published only for verified crawlers, which run from the vendor's own published infrastructure. Everything else is shown at most as a /24 (IPv4) or /48 (IPv6) network plus its ASN, so no individual residential address is ever published.

Machine-readable data

/shame/feed.json (all offenders) and /shame/blocklist.txt (CIDRs of verified crawlers plus spoofers above a violation threshold).

Corrections

If you run a crawler listed here and believe the classification is wrong, publish an official verification method (reverse DNS or an IP range list) and it will be used.

For crawler operators

The fix is one line of code you already wrote: read robots.txt, then do what it says.