About the crawler

This site runs a small web crawler that indexes technical documentation, package registries and reference material for a search engine used mostly by software agents. If it visited you, this is what it did and how to make it stop.

How it identifies itself

Mozilla/5.0 (compatible; RicciutiBot/0.2; +https://search.caioricciuti.com/bot)

Every request carries that user agent, with a link back to this page.

How it behaves

• Reads robots.txt first and follows it, including Crawl-delay.
• One connection per host, at least 2 seconds between requests to the same host.
• Conditional requests (ETag, If-Modified-Since): an unchanged page costs you nothing.
• Recrawls slower the longer a page stays the same, up to once in 90 days.
• Never renders JavaScript, never logs in, never fills forms, never fetches images, video or fonts.
• Pages marked noindex or noarchive (meta tag or X-Robots-Tag) are never stored.
• Package registries are read through their public APIs under their usage policies, not by crawling their pages.
• Stores the extracted text and a short description of a page, with its address and when it was fetched. No personal data is extracted or kept.

Opting out

User-agent: RicciutiBot
Disallow: /

Takes effect within a day: the host is not fetched again. A noindex meta tag on a single page keeps that page out of the index at its next fetch.

To have pages removed sooner, or for anything else, write tohello@caioricciuti.com with the addresses concerned.