TooruBot — the crawler behind Tooru

User-Agent

TooruBot/1.0 (+https://tooru.ai/bot)

Tooru’s crawler identifies itself with this string. Match on the leading “TooruBot” in your rules.

Who operates it

Tooru (tooru.ai) is a service that checks how AI search and AI agents read a website. It is operated by unType Inc. in Tokyo, Japan.

When it visits

Only when a Tooru user asks for a URL to be diagnosed, or registers a site and asks Tooru to list its pages. TooruBot does not crawl the web on its own, and apart from the API description and MCP server checks described below, it does not follow links beyond the pages it was asked for. It fetches one page at a time per site — across all Tooru users, never two at once — and waits at least one second between pages (longer if robots.txt sets a Crawl-delay).

What it fetches

For each page that is diagnosed:

• The page’s HTML, as served. It does not run JavaScript and does not load images, stylesheets or scripts.
• Once per site rather than per page: the URL of the first page diagnosed, once more with “Accept: text/markdown”, to see whether a Markdown version is offered (fetched with the site files below and reused the same way).
• These files at the root of the site (fetched together once and reused for up to 10 minutes, not once per page):

For the sitemap, it first tries the sitemaps named on Sitemap: lines in robots.txt (same site only, up to 3), then these locations in this order, one request at a time, and stops at the first one that reads as a sitemap:

Only in the following cases does it send a few more requests (each times out after 5 seconds). Sites without these declarations or links get none of them:

• If one of the MCP files above is an MCP server card: one POST to the MCP server URL written in the card, carrying a JSON-RPC “initialize” request (client name “TooruBot”), to check that the server answers. The server may be on another host. Nothing else is sent to it, redirects are followed only within the same origin, and nothing is sent when the card lists only a legacy HTTP+SSE endpoint.
• Only if that MCP server answers 401 or 403: up to two GET requests for OAuth discovery — the protected-resource metadata (the URL given in the WWW-Authenticate header, or /.well-known/oauth-protected-resource on the MCP server’s host unless that is the file already fetched above) and the authorization-server metadata (/.well-known/oauth-authorization-server of the authorization server named there, or of the MCP server’s host).
• Only for a page that links to an API description (<link rel="service-desc">, or links to paths such as openapi.json, swagger.yaml or /v3/api-docs, or to a Swagger UI / Redoc page): up to two of those links, plus one more when a Swagger UI / Redoc page names its description URL. Responses over 2 MB are cut off.

When a user asks Tooru to list a site’s pages, it also reads sitemap files (the ones referenced from robots.txt, the conventional locations below, and any sitemap the user adds) and the links in llms.txt.

Only for domains whose owner has verified ownership in Tooru, when the top page is diagnosed, it also requests the top page once over http:// (to check that visitors are sent to https://) and /.well-known/security.txt.

To check whether a page answers “not modified”, it requests the same page once more with If-None-Match / If-Modified-Since (only when the first response had an ETag or Last-Modified, and only for the top page and up to 4 other pages per diagnosis). It reads at most 1 KB of that second response.

To check that pages which don’t exist return 404, it requests one URL that shouldn’t exist, /tooru-check- followed by random letters, once per diagnosis. It reads at most 1 KB of the response. You may see it in your access logs as a 404.

Each request times out after 10 seconds for HTML and 5 seconds for other files. Responses larger than 5 MB are cut off. Pages are diagnosed one at a time.

If your server refuses it

Only for domains whose owner has verified them in Tooru: if the page answers 401, 403, 406 or 429, or returns a bot-protection challenge, Tooru fetches the same URL exactly once more with a common browser User-Agent. For any other domain, a refusal is final — TooruBot does not come back under another name. The purpose is to tell the user which of two things is happening: their server refuses TooruBot specifically, or it refuses every non-browser client. If the browser request succeeds, the files and requests listed above are fetched with the browser User-Agent too, so that one refusal is not reported as several. Nothing else is retried.

What is kept

The results of each check, and a short digest of the page: title, description, headings, navigation labels, the first 1,500 characters of the main text, the structured-data types and the site icon URL. Raw HTML and the text of robots.txt, llms.txt and /.well-known/ files are not kept, nor are MCP server responses or API descriptions (only their URLs, the server name and version the MCP server reports, and the result of each check). The results belong to the Tooru user who asked for the diagnosis and are deleted with their account.

Where it comes from

Requests come from Vercel’s infrastructure. The IP addresses are not fixed and are not published, so please identify TooruBot by its User-Agent.

TooruBot-Observer (checking cited sources)

TooruBot-Observer/1.0 (+https://tooru.ai/bot)

It visits in two cases. (1) When a Tooru user asks to check where an AI answer came from, it fetches the pages that an AI (the ChatGPT, Claude, Gemini or Perplexity API) cited as sources. (2) As monthly benchmark observation, it fetches the official sites of companies in major stock indices (Nikkei 225, TOPIX Prime, S&P 500, NASDAQ 100 and others), large private companies, notable startups (J-Startup, unicorns), and universities and colleges of technology: the top page and up to 4 main pages linked from it, such as the menu (5 pages in total), once a month as a rule — twice in a month when we change how pages are scored, so the benchmark is measured the new way. In both cases it does not crawl and follows no other links. The same URL is fetched at most once a month (at most twice in a month when the scoring changes, for benchmark observation only), at most 30 pages per site per month in total (plus the re-fetch of those 5 pages in such a month), one page at a time, with at least 1 second between pages (or your Crawl-delay).

For articles on other sites (round-ups, comparison sites and so on) it fetches only robots.txt and the page HTML. For pages on sites a user registered as a competitor, it fetches the same things TooruBot does for a diagnosis (see “What it fetches”).

robots.txt: it reads both “User-agent: TooruBot-Observer” and “User-agent: TooruBot” (or “*” if there is no such group) and does not fetch if either disallows the page. Disallowing only TooruBot-Observer declines source checks while still letting the site owner diagnose their own site in Tooru (TooruBot). If refused, it never retries with a browser User-Agent.

Tooru keeps only the results (company names found in the article, and the result of each diagnostic check). The article text is kept on the observation machine for up to 31 days to match names, then deleted. It is never stored on Tooru’s servers or database.

Requests come from an observation machine operated by Tooru (on a line in Japan). To stop it, use the same methods as for TooruBot (“How to stop it” below; matching the leading “TooruBot” in the User-Agent stops both).

How to stop it

Block the User-Agent “TooruBot” in your server, CDN or WAF settings. If you also want the single browser-User-Agent retry described above to stop, write to us at the address below.

robots.txt: TooruBot follows robots.txt. It does not fetch pages disallowed for “User-agent: TooruBot” (or for “*” when there is no TooruBot group), and it honours Crawl-delay up to 60 seconds. It always reads robots.txt itself, because that is how it knows the rules.

To have your site excluded from diagnosis, write to support@tooru.ai with the domain. Once it is on our exclusion list, no Tooru user can diagnose it, list its pages or register it — including users who have verified ownership — until you ask us to lift it.