An llms.txt file is a small map for an agent. It names a site, gives the shortest useful context,
and points to detailed resources that are easier for an agent to read than a full website shell.
Publishing the file is not enough. An agent still needs to find the one that applies to the page,
understand its structure, and reach the resources it names.
IndieLLMs exists to check that whole path. Enter a site or documentation URL. The checker first reads
the page’s rel=describedby discovery link. If there is no link, it walks from the page path toward
the site root and stops at the most specific readable llms.txt. It checks the file against proposal
v2, then requests the first 20 unique linked resources and reports how many remain unchecked.
Every rule links to the current proposal. Every measured file or resource links to the response that produced the result. IndieLLMs distinguishes agent-friendly Markdown or plain text from an HTML page that merely returned HTTP 200, because a working link and a useful agent resource are different answers.
The limits matter as much as the checks. llms.txt is a community proposal, not an IETF or W3C
standard. A valid file does not grant a crawler access. It does not force an agent to read the file,
include the site in an answer, or cite it. IndieCrawl answers the access-policy question. IndieLLMs
answers whether the proposed map is present, discoverable, structured, and connected to working
resources.
The check stays private and free. IndieLLMs asks for no account, card, or premium plan. It stores no lookup history and publishes no list of checked sites. Analytics is currently off.
For the separate access-policy question, check published search and AI crawler rules. A valid resource list does not override robots.txt.
Read the proposal and its adoption limits. Use IndieLLMs to find the applicable llms.txt and inspect its linked-resource responses.