No451 · crawler information

No451Bot

Last updated 12 October 2026

No451 checks what a first-time visitor to an adult website meets before content, and whether images, video or audio can be reached before an age check. Checks run at the request of a No451 user. That may be the site's operator, or someone else, in which case we follow your robots.txt and you can opt out (see below).

How to recognise it

Requests that read your site's public indexes are sent with this User-Agent:

No451Bot/1.0 (+https://no451.com/bot)

The robots.txt name (product token) is No451Bot. We follow RFC 9309: we use the group for No451Bot, or the * group if there is none.

What No451Bot fetches

robots.txt

Once per scan, to read your rules and any Sitemap lines.

Sitemaps

/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and sitemaps listed in robots.txt; up to 6 sitemap files per scan. We read media entries only and never fetch the pages a sitemap lists.

Feeds

RSS/Atom feeds linked from your page, and /feed/ on WordPress sites; up to 3 per scan. Media enclosures only.

WordPress media listing

On WordPress sites, one request to /wp-json/wp/v2/media per scan. No paging, no other routes.

Index contents are read in memory to find media file addresses. We keep the addresses (with any signatures masked) and response headers as evidence, not the index itself.

Media files: headers only

To test whether a media file is handed out before an age check, we send HEAD requests to a small sample (at most 10 per check, 20 per scan). A HEAD request returns headers only. We never download images, video or audio, and we never look at or classify what they show.

Visits that use a normal browser identity

Loading the page you asked us to check, and the HEAD requests on media files, use a normal desktop browser User-Agent, not No451Bot. These requests have to see exactly what a visitor sees: some age-check providers skip their check for known bots, and some servers answer bots differently. They are limited to the submitted page and the media files that page (or an index we were allowed to read) already hands out.

On a simple “I'm 18” or date-of-birth gate, the scanner may make one interaction: click the confirm button, or enter one fixed adult test date of birth and submit it once. That test date is never stored.

What we never do

• Download media bodies (images, video, audio).
• Log in, register, subscribe or pay.
• Solve or work around CAPTCHAs or bot checks.
• Bypass an age check or paywall, or go through an age-check provider's check.
• Fill in any form other than the one test date of birth described above.

robots.txt and site owners

For scans requested by anyone who has not verified ownership of your hostname, No451Bot obeys your robots.txt for every index it reads. A robots.txt that can't be reached because of a server error or timeout is treated as “don't read anything”; a 4xx response (for example 404) means there are no rules.

How to block No451Bot

To stop all index reads by No451Bot:

User-agent: No451Bot
Disallow: /

Or block only some paths:

User-agent: No451Bot
Disallow: /wp-json/
Disallow: /feed/

A blocked index is listed in the requester's report as “skipped: disallowed by robots.txt”.

Opt out of checks requested by other people

robots.txt only covers the indexes No451Bot reads. To stop all requests from No451 for checks requested by anyone other than you (loading your page, HEAD requests on media files, and index reads), put your domain on our do-not-scan list:

1. Email bot@no451.com with your domain and whether the opt-out should cover the whole domain and its subdomains, or one hostname only. A web form is planned; for now, email is the only way.
2. We reply with a one-time token.
3. Add a DNS TXT record at _no451.<your-domain> with the value no451-optout=<token>. DNS shows you control the domain itself.
4. Once we see the record, the opt-out is active and we confirm by email. You can remove the record after that; the opt-out stays.

Once active, a check of your domain is refused before we send any request to your site, and the requester is told that the owner has asked us not to scan it. To withdraw the opt-out, email us.

We don't publish the IP addresses our scanner uses: a list would let a site show the scanner a different page from the one visitors get. The do-not-scan list is the way to opt out instead.

Rate limits

Per hostname

At most one live scan every 15 minutes.

Per scan

One browser session, 90-second limit. Index reads and HEAD requests are capped as listed above.

Per requester

Anonymous requests: 3 scans per IP address per day.

Contact

Questions or problems with No451Bot: bot@no451.com. Security reports: see /.well-known/security.txt.