Methodology
Scoring Methodology
Everything Tooru measures, how it measures it, and how many points each check is worth. Scores come from mechanically analysing the HTML, robots.txt, structured data and agent-facing files of each page. No human judgement is involved.
Principles
The diagnostic is designed around three principles.
- Objectivity
- Every check is derived from a published technical standard or an official vendor document. The result is a mechanical reading of the page as served; it does not include subjective judgement.
- Reproducibility
- The same URL measured under the same conditions yields the same score. Tooru does not cache results: every run fetches the page again, so a change on your site shows up in the next run and is kept as history for Before/After comparison.
- Transparency
- All 39 scored checks, their points, judgement rules and referenced standards are published on this page, together with the 16 reference-only items that are recorded without points.
Scoring system
The total score is the sum of the four questions (find 19, read 29, understand 38, use 14) = 100 points. Each page (URL) is scored independently; domain-level resources such as robots.txt, sitemap.xml, llms.txt and /.well-known/ files are fetched once per origin. Checks that do not apply to a page are marked “not applicable” and leave the calculation: each question’s score is its points × (points earned ÷ points of the checks that apply), rounded. A question with no applicable checks takes the ratio achieved on the other questions. Some checks are asked only on certain kinds of page, of which there are four: top (the site root or a language root such as /en/), FAQ (FAQPage markup, an FAQ-style path such as /faq, or an FAQ heading), article (an Article type with a publication date, or a news- or blog-style path with a date or an article body) and other.
1. Can AI find you?19 pts · 11 checks2. Can AI read your pages?29 pts · 10 checks3. Does AI understand your business?38 pts · 12 checks4. Will AI use you in its answers?14 pts · 6 checks
Total100 pts · 39 checks
In addition to the 39 scored checks, 16 reference-only items (policy and reference bots, upcoming agent protocols, the crawler fetch result and checks shown as notes only) are recorded with zero points. They never change a score.
Ranks
Each page receives one of five ranks from its total score.
| Rank | Score | Meaning |
|---|---|---|
| S | 90–100 | Very high conformance to the standards; the major AI agents can read and reach the page without friction. |
| A | 75–89 | The main conditions are in place; fixing one or two of the four questions will lift the score further. |
| B | 60–74 | Basic support exists, but some checks still stop AI agents from retrieving information accurately. |
| C | 40–59 | Several of the four questions score low; AI has difficulty reading or using the information. |
| D | 0–39 | Conformance is insufficient; the major AI agents cannot retrieve accurate information from the page. |
A site’s overall rank is derived from the average of the latest score of every URL diagnosed under that site (rounded to one decimal place), applied to the same thresholds.
Checks by question
1. Can AI find you? (19 Points)
Whether search-type AI crawlers can reach the site and find their way around it: the sitemap, the canonical URL, an MCP server declared under /.well-known/, feeds, llms.txt and the Content-Signal policy. If a page cannot be found, nothing else on it matters.
| # | Check | Points | Method | Judgement |
|---|---|---|---|---|
| 1-1 | MCP server declaration (/.well-known/) | 3 | Fetch and connect (initialize) | An MCP server card is looked for at /.well-known/mcp/server-card.json, /.well-known/mcp.json and /.well-known/mcp, and the first file that reads as an MCP card is used (JSON without any MCP-specific key, such as a JSON error page, is not a card). If a card is found, Tooru actually sends a JSON-RPC initialize request to the server URL the card declares. An MCP response = 3. 401/403 with OAuth discovery that leads to valid authorization-server metadata (RFC 9728 protected-resource metadata → RFC 8414) = 3; 401/403 without it = 0. A card without a server URL, a legacy HTTP+SSE endpoint only, an error status, no response or a response that is not MCP = 0. No card = not applicable (no points lost); setting up an MCP server is then suggested in the report’s “Coming next” box. |
| 1-2 | sitemap.xml | 2 | Fetch and validate | The sitemaps named on Sitemap: lines in robots.txt (same site, subdomains included; the first three) are tried first, then /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and /sitemap-index.xml, one at a time, stopping at the first one that reads as a sitemap. Gzipped sitemaps are read too. A urlset or sitemapindex element with loc entries = 2; the element without any loc = 1; anything else, or none found (an HTML response counts as missing) = 0. |
| 1-3 | Canonical URL | 2 | link[rel=canonical] | A link[rel="canonical"] that points to the page itself (differences in the trailing slash, index.html, query string and # are ignored) = 2. Only http/https or www differs = 1; a page other than the top page points to the top page = 1 (a typical template mistake); another page = 1 (it may be an intended merge); not an http(s) URL = 1. Missing = 0. |
| 1-4 | OAI-SearchBot allowed (Search) | 2 | robots.txt parsing | OpenAI’s search crawler (ChatGPT search) is not disallowed. Blocking it removes the site from ChatGPT search results. Allowed or no rule = 2; disallowed = 0. No robots.txt counts as allowed; not applicable when robots.txt cannot be read because of a server error. |
| 1-5 | PerplexityBot allowed (Search) | 2 | robots.txt parsing | Perplexity’s search crawler is not disallowed. Allowed or no rule = 2; disallowed = 0. No robots.txt counts as allowed; not applicable when robots.txt cannot be read because of a server error. |
| 1-6 | Claude-SearchBot allowed (Search) | 1 | robots.txt parsing | Anthropic’s search crawler is not disallowed. Blocking it removes the site from Claude’s search citations. Allowed or no rule = 1; disallowed = 0. No robots.txt counts as allowed; not applicable when robots.txt cannot be read because of a server error. |
| 1-7 | RSS / Atom feed | 2 | Feed link detection | Asked only on the top page (not applicable elsewhere). A link element with rel="alternate" and type application/rss+xml, application/atom+xml or application/feed+json = 2; none = 0. WordPress oEmbed links (text/xml+oembed and similar) are not feeds. |
| 1-8 | /llms.txt present | 2 | File fetch | /llms.txt is fetched as text (an HTML response counts as missing). Present = 2; present but under 10 characters = 1; missing = 0. |
| 1-9 | llms.txt spec compliance | 1 | Format validation | The structure is compared with llmstxt.org: (1) the first line is a “# ” title and it is the only one; (2) a “> ” summary comes after the title, before the first section; (3) there is at least one “## ” section; (4) a section contains a Markdown link list item (“- [name](URL)”, optionally followed by “: notes”); (5) every list item in the sections has that form with a readable URL. Four or more, including the title = 1; otherwise 0 (two or three is a warning). Not applicable when llms.txt is missing. |
| 1-10 | /llms-full.txt present | 1 | File fetch | Present = 1; present but under 100 characters = 0 (warning); missing = 0. |
| 1-11 | Content-Signal | 1 | robots.txt parsing | robots.txt contains a Content-Signal line with at least one of search, ai-input or ai-train set to yes or no (for example “search=yes, ai-train=no”) = 1. No line = 0; a line without such a pair = 0 (warning); no robots.txt = 0. Not applicable when robots.txt cannot be read because of a server error. |
Reference-only items (not scored)
Recorded so the report can show them, but worth zero points. Reference bots are judged like scored bots (allowed / blocked). Agent-layer declarations are judged in three states: valid, malformed (present but not usable) and absent. Reference bots recorded without points: ChatGPT-User, Googlebot, bingbot, Perplexity-User, Google-Agent, Applebot. User-initiated fetchers (ChatGPT-User, Perplexity-User, Google-Agent) act only on a person’s request and are outside robots.txt control. Googlebot, bingbot and Applebot are the search engines themselves; blocking them removes the site from those search results, so the state is recorded rather than scored.
| # | Check | Points | Method | Judgement |
|---|---|---|---|---|
| — | GPTBot allowed (Training) | Info | robots.txt parsing | OpenAI’s training crawler. Whether it is allowed (or has no rule) or disallowed is recorded without points — letting AI train on the site is a policy choice. |
| — | ClaudeBot allowed (Training) | Info | robots.txt parsing | Anthropic’s training crawler. Allowed or disallowed is recorded without points (a policy choice). |
| — | Google-Extended allowed (Training) | Info | robots.txt parsing | Google’s token for Gemini training. Blocking it does not affect Google Search. Allowed or disallowed is recorded without points (a policy choice). |
| — | Applebot-Extended allowed (Training) | Info | robots.txt parsing | Apple’s token for Apple Intelligence training. Allowed or disallowed is recorded without points (a policy choice). |
| — | Other AI crawlers | Info | robots.txt parsing | None of Meta-ExternalAgent, Amazonbot, Bytespider, CCBot, cohere-ai, Google-CloudVertexBot, DeepSeekBot is disallowed = pass; one or more disallowed = warning. Not scored — judged as a group because blocking any of them is a policy choice rather than a defect. |
| — | Googlebot (Search) | Info | robots.txt parsing | Google Search itself. Blocking it removes the site from Google Search entirely, so the state is recorded for reference rather than scored. |
| — | bingbot (Search) | Info | robots.txt parsing | Bing Search and Microsoft Copilot share this crawler and cannot be separated. Recorded for reference. |
| — | Applebot (Search) | Info | robots.txt parsing | Apple’s search crawler (Siri, Spotlight, Safari). Recorded for reference; training use is controlled separately through Applebot-Extended. |
Standards: RFC 9309: Robots Exclusion Protocol · sitemaps.org · Content Signals · llmstxt.org · Model Context Protocol · RFC 9728: OAuth 2.0 Protected Resource Metadata · RFC 9727: api-catalog · OpenAI: Overview of OpenAI crawlers · Anthropic: Does Anthropic crawl data from the web · Google: Overview of Google crawlers and fetchers
2. Can AI read your pages? (29 Points)
Whether an AI that arrives on a person’s behalf can get in and read the page: robots.txt, the user-initiated fetcher, and whether the HTML the server returns carries the content at all (agents do not run JavaScript). Also the heading structure, landmarks, image alternatives, document language, Markdown negotiation, a published API description and response time.
| # | Check | Points | Method | Judgement |
|---|---|---|---|---|
| 2-1 | JavaScript dependence | 6 | Initial HTML analysis | Whether the main text is in the HTML the server returns, before any script runs. Visible body text (script, style and noscript removed) of at least 200, counting a full-width character as 2 (about 100 Japanese characters), with fewer than 10 external scripts = 6, fewer than 20 = 5, 20 or more = 4. Without that much text: a readable noscript fallback = 2 (tag-manager iframes, images and “please enable JavaScript” notices do not count); some text (over 40 by the same count) = 1; otherwise 0. |
| 2-2 | Heading hierarchy | 5 | HTML parsing | No h1 = 0. Exactly one h1 = 2 (more than one = 1); an h2 present = +1; an h3 present alongside an h2 = +1; no skipped level (e.g. h2 → h4) among the headings of the main content = +1. Headings in the header, footer and navigation are not checked for skips. |
| 2-3 | Landmark elements | 4 | HTML parsing | Each of main, nav, header and footer (or the matching ARIA role) present = 0.8; an article or section (or role="article" / "region") = +0.8. The total is rounded to the nearest whole point (max 4). |
| 2-4 | Image alt coverage | 3 | HTML parsing | Share of images that carry an alt attribute. An empty alt="" counts as correct (it is how decorative images are marked). Images with role="presentation" or "none", aria-hidden="true", and 1×1 tracking pixels are not counted. 90% or more = 3, 70% = 2, 50% = 1, below = 0. Not applicable when there are no images to check. |
| 2-5 | robots.txt | 2 | Fetch and parse (RFC 9309) | Fetched and at least 5 characters long = 2; present but shorter = 1. No robots.txt (4xx, or an HTML page returned instead) = 1 — crawlers may fetch everything, but the site states no policy. The server fails (5xx, 429, timeout, unreachable) = 0 — crawlers then treat the whole site as off-limits (stage ② stops). In that case the per-crawler items and Content-Signal are left out of the score as unknown. |
| 2-6 | Language attribute | 2 | HTML parsing | The html element has a non-empty lang attribute = 2; missing = 0. |
| 2-7 | Markdown content negotiation | 2 | Accept header probe | The page URL is requested with Accept: text/markdown (once per site together with the site-level files, using the first page diagnosed; the result is reused for the site’s other pages). A text/markdown (or text/x-markdown) response = 2; anything else, including HTML, an error or no response = 0. |
| 2-8 | Public API reference | 2 | Fetch and validate | Whether a machine-readable API description can actually be fetched. Candidates are <link rel="service-desc"> and links whose path looks like an API description (openapi.json, swagger.yaml, /v3/api-docs and similar) or a Swagger UI / Redoc page. Up to two are fetched; from a Swagger UI / Redoc page, the description URL it names is followed once. A valid OpenAPI 3.x or Swagger 2.0 document (JSON or YAML, with info), or a /.well-known/api-catalog (RFC 9727) whose linkset has at least one link with an href = 2. A description link that cannot be fetched or is not OpenAPI, or an API catalog without such a link = 0 (warning). Not applicable (no penalty) when neither is found — the word “API” or a link to /api alone does not count. |
| 2-9 | Server response time | 2 | HTTP timing | Time until the server starts responding (time to first byte, including redirects), measured from Tooru’s crawler. Downloading the body is not counted, so distant servers are not penalised for page size. Up to 0.8 seconds = 2; up to 2.5 seconds = 1; longer = 0. |
| 2-10 | Claude-User allowed (Agent) | 1 | robots.txt parsing | Anthropic’s user-initiated fetcher (Claude visiting a page on a user’s request) is not disallowed. Allowed or no rule = 1; disallowed = 0. No robots.txt counts as allowed; not applicable when robots.txt cannot be read because of a server error. |
Reference-only items (not scored)
Recorded so the report can show them, but worth zero points. Reference bots are judged like scored bots (allowed / blocked). Agent-layer declarations are judged in three states: valid, malformed (present but not usable) and absent.
| # | Check | Points | Method | Judgement |
|---|---|---|---|---|
| — | ChatGPT-User (Agent) | Info | robots.txt parsing | OpenAI’s user-initiated fetcher. It acts only on an explicit user request and does not honour robots.txt, so the allow/disallow state is recorded but cannot be acted on. |
| — | Perplexity-User (Agent) | Info | robots.txt parsing | Perplexity’s user-initiated fetcher. Acts only on an explicit user request; robots.txt does not control it. Recorded for reference. |
| — | Google-Agent (Agent) | Info | robots.txt parsing | The user agent Gemini Agent and similar tools use when visiting a site on a user’s instruction (added to Google’s crawler list in March 2026). robots.txt does not control it. Recorded for reference. |
| — | WebMCP Declarative API | Info | HTML parsing | form elements carrying both toolname and tooldescription (W3C WebML Community Group draft; Chrome 149+ origin trial). Both attributes non-empty on at least one form = valid; only one of the two = malformed (the tool is not registered); none = absent. The imperative JavaScript API is not judged because the crawler does not execute scripts. |
| — | A2A Agent Card | Info | Fetch and content validation | /.well-known/agent-card.json (A2A Protocol 1.0). A JSON object with a non-empty name and either url or a non-empty interfaces array = valid; fetched but not meeting that = malformed; not published = absent. |
| — | UCP profile | Info | Fetch and content validation | /.well-known/ucp (Universal Commerce Protocol, 2026-08-25 release). A ucp object with version in YYYY-MM-DD form, services and payment_handlers = valid; fetched but not meeting that = malformed; not published = absent. The keys array is not required at this stage. |
| — | Served to the crawler | Info | HTTP fetch (two User-Agents) | Whether the page is served to Tooru’s crawler (User-Agent “TooruBot”). If the server answers 401/403/406/429 or a bot-protection challenge on a domain whose ownership has been verified, the same URL is fetched once more with a browser User-Agent. Served to the browser only = the crawler is refused by User-Agent (stage ② stops). Refused both times, or refused on an unverified domain = the page cannot be diagnosed and the run reports it. Not scored. If the server’s TLS setup is flawed (no intermediate certificate, no secure renegotiation, or a weak key exchange) and the page could be read only by compensating as a browser does, this is shown as an unscored notice, because some AI crawlers may not be able to read it. |
Standards: HTML Living Standard (WHATWG) · RFC 9309: Robots Exclusion Protocol · Anthropic: Does Anthropic crawl data from the web · WebMCP (W3C WebML CG Draft) · A2A Protocol · Universal Commerce Protocol
3. Does AI understand your business? (38 Points)
Whether an AI can work out what the business is: what you offer, who it is for and where you operate, stated in the opening of the page, plus the structured data that identifies the organisation — address, contact, an external reference — and the Schema.org types, property values and Open Graph metadata.
| # | Check | Points | Method | Judgement |
|---|---|---|---|---|
| 3-1 | JSON-LD present | 5 | HTML parsing | At least one script[type="application/ld+json"] block that parses = 5; parseable blocks alongside broken ones = 3 (warning — the broken ones reach no AI); blocks present but none parse = 0 (reported as broken); only FAQPage / QAPage markup = 0 (warning — a FAQ block alone does not say whose page it is or what it is about); none = 0. Top-level arrays, @graph and nested entities are read, and comment or CDATA wrappers are tolerated. When no JSON-LD can be parsed, Schema.org types, entity connections, required properties and property value format are not applicable, so missing JSON-LD costs points only once. |
| 3-2 | Schema.org types | 5 | JSON-LD parsing | Types are recognised through the Schema.org hierarchy (Corporation and Dentist count as Organization, NewsArticle as Article), and the score depends on the kind of page. Top page: an Organization type = 3, WebSite = 2 (only other types = 1). Article page: an Article type = 3, plus 1 each when its author and its publisher are Person or Organization entities (@id references resolved); without an Article type, at most 2 on the rule for other pages. FAQ page: FAQPage / QAPage whose questions all have a name and an answer text, at least one of them appearing in the visible text = 5; otherwise the rule for other pages. Other pages: a type that describes the page’s content (Product, Service, Event, AboutPage, Article and so on) = 5; only site-wide types (Organization, WebSite) = 3; only generic types (WebPage, BreadcrumbList, ImageObject and so on) = 1. FAQPage / QAPage never counts as a content type. Not applicable without parseable JSON-LD, or with FAQ markup only. |
| 3-3 | Property value format | 3 | JSON-LD value validation | Format of the values present on top-level entities (and on the publisher’s logo): datePublished, dateModified, startDate, endDate and uploadDate must be ISO 8601 dates that exist (seconds and the colon in the time-zone offset may be omitted); url must be an absolute URL; image and logo must be absolute URLs or ImageObjects with an absolute url or contentUrl; inLanguage must be a BCP 47 tag (any letter case, e.g. en-us, zh-Hant-TW, es-419); headline must not be empty; author must be a non-empty name, or a Person or Organization with a name. @id references are resolved within the page, and references that cannot be resolved are not counted. Valid share 95% or more = 3, 80% or more = 2, 50% or more = 1, below = 0. Not applicable when no checked value is present, or when no JSON-LD can be parsed. |
| 3-4 | Entity connections | 3 | JSON-LD parsing | @graph used = +1. A reference ({"@id": …}, including inside nested objects and arrays) that points to another entity’s @id on the page = +2; @id present without such a reference = +1. Not applicable without parseable JSON-LD, or with FAQ markup only. |
| 3-5 | Organization identifiable | 3 | JSON-LD validation | One point each, on any Organization-type entity (subtypes such as Corporation, LocalBusiness and its subtypes, NGO; nested entities and references included), for an address (addressLocality or addressRegion, or an address written as text), a telephone or contactPoint, and at least one sameAs reference to an external profile. On the top page, no Organization-type entity at all = 0 (fail): the entrance of a site should state whose site it is. Other pages are not applicable (organisation identity is asked on the top page). This differs from required properties: that check asks whether a type’s required fields are filled, this one asks whether the organisation can be told apart from a company with a similar name. |
| 3-6 | Required properties | 2 | JSON-LD parsing | Fill rate of the required properties on top-level entities, using the nearest type in the Schema.org hierarchy: Article and BlogPosting (and their subtypes) need headline, image, datePublished and author; FAQPage and QAPage need mainEntity, and every question needs a name and an answer text (counted as two more); BreadcrumbList needs itemListElement; WebPage needs name; WebSite needs name and url; LocalBusiness needs name and address; Organization needs name and url; Product and Person need name. Nested entities (such as a publisher) are not asked for these. 90% or more = 2, 50% or more = 1, below = 0. Not applicable (no penalty) when none of these types is at the top level, or when no JSON-LD can be parsed. |
| 3-7 | Open Graph metadata | 2 | Meta tag analysis | og:title, og:description, og:image and og:url all present = 2; two or more (or one plus twitter:card) = 1; otherwise 0. |
| 3-8 | Business stated up front | 3 | Title, description, h1 and the opening of the main text | Asked only on the top page (not applicable elsewhere). One point each for what you offer, who it is for and where you operate, found in the title, meta description, the main h1 or the opening of the main text (display width 600: about 300 Japanese or 600 Latin characters; navigation, header and footer excluded). Japanese pages are read with Japanese vocabulary and other pages with English vocabulary, matched as whole words. When no place is mentioned but the business reads as online (cloud, SaaS, app and so on), what and who alone give full marks. An AI reads the opening first; if the business is not clear there, structured data cannot make up for it. |
| 3-9 | Title | 3 | Meta tag analysis | A title element in the head (titles inside SVG icons are ignored) = 1; display width 20–120, counting a full-width character as 2 (about 10–60 Japanese or 20–120 Latin characters) = +1; not just a generic word (home, index, top, welcome, untitled, document, page, ホーム, トップ and similar) = +1. Missing = 0. |
| 3-10 | Meta description | 3 | Meta tag analysis | meta[name="description"] (any letter case) present = 1; display width 50–99 or 321–480 = 2; 100–320 (about 50–160 Japanese or 100–320 Latin characters) = 3. Missing = 0. |
| 3-11 | Information structure | 3 | Content structure analysis | Share of h2/h3 headings that are followed by real content: the text up to the next heading of the same or a higher level (text under h4–h6 included) reaches 40 on Japanese pages or 60 on other pages, counting a full-width character as 2 (about 20 Japanese or 60 Latin characters). Headings in navigation, the site header, footer, sidebars and breadcrumbs are not counted. 70% or more = 3; 40% or more = 2; otherwise 1 if there are two or more headings, 0 if only one; no h2/h3 = 0. |
| 3-12 | Internal links | 3 | Link analysis | Links to pages on the same site, resolved against the final URL after redirects and compared by registered domain (www and other subdomains count as the same site). #, javascript:, mailto: and tel: links are not counted. 10 or more = 3; 5 or more = 2; 1 or more = 1; none = 0. |
Reference-only items (not scored)
Recorded so the report can show them, but worth zero points. Reference bots are judged like scored bots (allowed / blocked). Agent-layer declarations are judged in three states: valid, malformed (present but not usable) and absent.
| # | Check | Points | Method | Judgement |
|---|---|---|---|---|
| — | Structured data vs. page text | Info | JSON-LD × page text | Whether values in the structured data also appear in the page text: company name (Organization name, legalName or alternateName — any one appearing is enough; corporate suffixes such as 株式会社 or Inc. and spacing are ignored), phone (telephone; brackets, hyphens, full-width digits and +81 are normalised), address (PostalAddress postalCode, addressLocality, streetAddress; 〒 and 丁目/番地 spellings are normalised), opening hours (openingHours / openingHoursSpecification; “10:00〜18:00” and “10時から18時” count as the same range) and price (Offer price, lowPrice or highPrice; ¥, 円, commas and 税込 are ignored). Apart from the company name, which is always compared, a kind of value is compared only when the page text contains at least one value of that kind, so a site-wide Organization block is not reported as “missing” on pages that never mention an address. The reverse direction flags phone numbers, postal codes and prices that appear in the text but not in the structured data, and FAQPage / QAPage questions that do not appear in the visible text. Currently a note only, not scored; scoring will be considered when the engine version is next raised. |
Standards: Schema.org · Google Search Central: Structured data · The Open Graph protocol · Google Search Central
4. Will AI use you in its answers? (14 Points)
Whether the page gives an AI grounds to use it in an answer: concrete figures with units, cited sources and quotations, content shaped like answers (questions, definitions, steps, comparisons), author and dates, and how recently it was updated. Research on generative engines (Aggarwal et al., KDD 2024) found these are what raise how often a source is cited. Sources and author and dates are asked on article pages; Q&A and answer shape on article and FAQ pages.
| # | Check | Points | Method | Judgement |
|---|---|---|---|---|
| 4-1 | Concrete figures | 3 | Body text analysis | Asked only on the top page and article pages (not applicable elsewhere: greetings, philosophy, lists and forms rarely carry figures). Count of figures with a unit in the visible main text (navigation, header, footer and breadcrumbs excluded). Dates and times are removed first, so “11 October 2026” or “2026年10月11日” do not count. Currency symbols ($ € £ ¥) and % count on any page; English units (years, customers, employees, million, kg, km …) are matched as whole words; Japanese units (円, 年間, ヶ月, 件, 社, 名, 人, 倍 …) count on Japanese pages only. 10 or more = 3, 5 or more = 2, 1 or more = 1, none = 0. Proper nouns are not counted. A study of generative engines (Aggarwal et al., GEO, KDD 2024) reports that methods such as adding statistics raised a page’s visibility in AI answers by up to 40%. |
| 4-2 | FAQ / Q&A content | 3 | Visible Q&A analysis | Asked only on FAQ and article pages (not applicable elsewhere). Counts visible question-and-answer pairs: a dt that reads as a question followed by a dd, a details element whose summary reads as a question, or a question-style heading followed by text. On article pages, question-style headings count only inside an FAQ section (under a heading such as FAQ or Frequently Asked Questions); other question-style headings in an article are judged under answer shape. Titles in lists of related articles (text that is entirely a link) are not counted. The answer needs at least 20, counting a full-width character as 2. A question ends in ? or ？, starts with Q / Q1 / Ｑ or a question word, or ends in a Japanese question form (ですか, ますか …). Pairs in navigation and footers are not counted. Two or more pairs = 2; five or more, or two or more with an FAQ heading on the page (FAQ, よくあるご質問, Frequently Asked Questions, Q&A …) = 3; one pair, or an FAQ heading with no readable pairs = 1; none = 0. FAQPage markup alone earns nothing here; whether it is correct and matches the page is checked under structured data. |
| 4-3 | Author and dates | 3 | Metadata and body analysis | Asked only on article pages (not applicable elsewhere). Author (meta author, an author property in JSON-LD, or rel="author") = +1; publication date (datePublished, a time element with datetime, or article:published_time) = +1; modification date (dateModified or article:modified_time) = +1. |
| 4-4 | Shaped like answers | 2 | HTML structure analysis | Asked only on FAQ and article pages (not applicable elsewhere). One point for two or more question-style h2–h4 headings (ending in a question mark; on English pages also headings such as “how to”, “what is”, “vs”, “comparison” or “step by step”; on Japanese pages とは, の違い, 比較, 手順, 選び方, メリット and similar), and one point for a definition list with two or more terms, an ordered list of three or more items, or a table with header cells and two or more columns. Navigation, header, footer, sidebars and breadcrumbs are not counted. These blocks are easy to lift into an answer as they are. |
| 4-5 | Sources and quotations | 2 | Link and markup analysis | Asked only on article pages (not applicable elsewhere). One point for a link to another site (compared by registered domain) whose text or surrounding sentence names a source (source, according to, reference, cited, 出典, 参考文献, 引用元 …; “open source” and “source code” do not count), and one point for a blockquote, cite or q element. Links and quotes in navigation, header, footer and sidebars are not counted. Showing sources lets readers, and AI, check where a claim comes from. |
| 4-6 | Recently updated | 1 | Date extraction | The most recent ISO-format date among dateModified in the structured data, time elements with a datetime attribute, article:modified_time and a last-modified meta tag; future dates (such as event dates) are ignored. Within 12 months of the diagnosis = 1, older = 0. Not applicable when no date can be read — the author and dates check already covers a missing date, so it is not counted twice. |
Standards: GEO: Generative Engine Optimization (Aggarwal et al., KDD 2024) · Google Search Central · Schema.org
AI agent readiness summary
Every report opens with a summary that answers “can AI agents reach this page, and can they find their way around?”. It is derived from the checks above and does not add or change points.
- Verdict
- Blocked — an Agent-type or Search-type bot is disallowed in robots.txt, or robots.txt cannot be read because the server fails (a missing robots.txt does not block). Nothing else matters until agents can get in.
- Partial — agents can reach the page, but the discovery layer has gaps: llms.txt missing, no working MCP server declaration, or no Markdown content negotiation.
- Ready — agents can reach the page and find guidance.
- First thing to fix
- The summary names exactly one blocker, checked in a fixed order — first what keeps an agent out (bots blocked, a broken robots.txt), then gaps in the signposts: an Agent-type bot blocked → robots.txt broken → a Search-type bot blocked → llms.txt missing → no working MCP server declaration → no Markdown negotiation.
- Reachability by bot type
- Reach is shown per bot group (Search, Training, Agent) from the individual bot checks. Bots that ignore robots.txt are marked so the reader knows the setting cannot control them.
Standards referenced
Standards and official documents the checks are derived from.
- 1. Can AI find you?
- 2. Can AI read your pages?
- 3. Does AI understand your business?
- 4. Will AI use you in its answers?
Measurement constraints
What the diagnostic can and cannot see.
- What is measured
- Only pages reachable over public HTTPS are measured. Pages behind Basic authentication, IP restrictions or a login cannot be diagnosed.
- The crawler reads the HTML the server returns and does not execute JavaScript. Content rendered only by scripts is invisible to it, which is what most AI crawlers also see.
- Each check is judged on the page as served to Tooru’s crawler (User-Agent “TooruBot”). Pages that serve different content to different clients may score differently from what a person sees. If the server refuses the crawler (401/403/406/429 or a bot-protection challenge), the same URL is fetched once more with a browser User-Agent, only to tell “the crawler is refused by User-Agent” apart from “every non-browser client is refused”.
- Accuracy
- robots.txt is interpreted with an RFC 9309-compliant parser. It shows what the file declares, not what the server or a CDN actually enforces.
- Edge-level bot management (for example CDN rules that challenge or block AI agents) is not visible in robots.txt. A page can allow a bot in robots.txt and still block it at the edge. Tooru detects this only for its own crawler, when the fetch itself is refused or challenged; rules that let a request through but alter the response are not visible.
- Structured-data “appropriateness” is a mechanical reading against Schema.org type definitions, not a judgement of business fit.
- Some content-quality checks (information structure, FAQ detection, author and date) use pattern matching and can miss unconventional markup.
- Timing
- Results are not cached. Every run fetches the page again at the time of the job and consumes credits per page.
- Server response time depends on network conditions at the moment of measurement and can vary between runs.
- Large jobs are processed in chunks; pages within one job may be measured minutes apart.
- What the score means
- The score measures conformance to published technical standards. It is an indicator, not a guarantee of business outcomes.
- How each AI platform retrieves, ranks and cites content is decided by that platform and is independent of this methodology.
- Reference-only items never affect the score. They are recorded so the report can show emerging standards before they are widespread enough to score.
Version history
| Version | Date | Changes |
|---|---|---|
| v3.0 | 2026-10-11 | Corrections based on observing about 6,400 companies’ sites. Checks that do not apply to a page (for example no structured data to validate, or no articles) no longer count as lost points: each stage is scored on the checks that apply. FAQ, answer shape, sources and author/date are asked only on article and FAQ pages; organisation identity, business clarity and feeds on the top page. FAQ is judged by the visible question-and-answer content rather than FAQPage markup alone. The text-to-HTML ratio was retired and its 3 points moved to JavaScript-free readability (+2) and heading structure (+1). Fixed judgement errors, including a missing robots.txt being treated as broken (it means everything is allowed), sitemaps declared in robots.txt being missed, Google Tag Manager noscript tags counted as page text, and English pages scoring low because vocabulary was Japanese-only. The MCP declaration is now checked by actually connecting to the declared server. Engine version 3.0.0. |
| v2.0 | 2026-10-09 | Corrected how 18 checks are described so that the text matches what the engine actually does (for example the llms.txt points, internal links, content structure, structured-data values and the conditions for refetching with a browser User-Agent). No check, threshold or point was changed. |
| v2.0 | 2026-09-08 | Scoring reorganised around four questions — the conditions a page has to meet before an AI can use it in an answer (find 19, read 29, understand 38, use 14) — replacing five categories of 20 points each. Points weighted by evidence: the llms.txt checks were reduced, and whether AI training bots are allowed became a policy shown without points. Six checks added: business clarity, organisation identity, specificity, evidence and sources, answer shape and freshness. Engine version 2.0.0. |
| v1.0 | 2026-09-05 | First Tooru release. Inherits the 39 checks and points of the untype.jp AI diagnostic (V1) methodology v2.3 (2026-06-05) unchanged, adds three reference-only agent-layer items (WebMCP Declarative API, A2A Agent Card, UCP profile) and publishes the rules of the AI agent readiness summary. The V1 methodology is frozen; later changes are made here. |
The methodology is revised as AI agent standards evolve. Changes are announced on this page with a new version number. Scored items and points only change when we release a new version of the Tooru checkup, so past scores stay comparable.
Feedback
Corrections, requests to support a new standard and opinions on point allocation are welcome at support@tooru.ai. Please mention “scoring methodology” in the subject.