An llms.txt file is a small map for an agent. It names a site, gives the shortest useful context,
and points to detailed resources that are easier for an agent to read than a full website shell.
That is useful, but only when the map can be found and its paths work.
The second version of the community proposal made discovery and scope clearer. A file can live at
the site root or inside a path such as /docs/. The most specific file applies. A page can point to
that file with rel=describedby, either in an HTML link element or an HTTP Link header. The file
starts with one H1 project title. It can then carry a short blockquote, useful context, and H2 file
lists made from named Markdown links.
IndieLLMs checks that shape. Enter a site or documentation URL and it reads the page’s discovery
link. If the link is absent, it walks from the page path to the site root and stops at the most
specific readable llms.txt. It checks the structure without rendering any Markdown. Then it
requests the first 20 unique linked resources and reports how many it skipped. A reachable Markdown
or plain-text resource is different from a reachable HTML page, and the answer names that difference.
The limits matter as much as the checks. llms.txt is a community proposal, not an IETF or W3C
standard. A valid file does not grant a crawler access. It does not force an agent to read the file,
include the site in an answer, or cite it. IndieCrawl answers the access-policy question. IndieLLMs
answers whether the proposed map is present, discoverable, structured, and connected to working
resources.
The check stays private and free. IndieLLMs stores no lookup history, asks for no account or card, and has no premium result waiting behind the useful one. Every rule links back to the current proposal, and every measured file or resource links back to the response that produced the result.