# Why IndieLLMs exists

> Find the llms.txt file that applies to one page, check proposal v2, and test the resources it names.

By turushan.

Published: 2026-09-03T00:00:00.000Z

Updated: 2026-09-18T19:57:15Z

Canonical: https://indiellms.com/blog/why-indiellms-exists/

An `llms.txt` file is a small map for an agent. It names a site, gives the shortest useful context,
and points to detailed resources that are easier for an agent to read than a full website shell.
Publishing the file is not enough. An agent still needs to find the one that applies to the page,
understand its structure, and reach the resources it names.

IndieLLMs exists to check that whole path. Enter a site or documentation URL. The checker first reads
the page's `rel=describedby` discovery link. If there is no link, it walks from the page path toward
the site root and stops at the most specific readable `llms.txt`. It checks the file against proposal
v2, then requests the first 20 unique linked resources and reports how many remain unchecked.

Every rule links to the current proposal. Every measured file or resource links to the response that
produced the result. IndieLLMs distinguishes agent-friendly Markdown or plain text from an HTML page
that merely returned HTTP 200, because a working link and a useful agent resource are different
answers.

The limits matter as much as the checks. `llms.txt` is a community proposal, not an IETF or W3C
standard. A valid file does not grant a crawler access. It does not force an agent to read the file,
include the site in an answer, or cite it. IndieCrawl answers the access-policy question. IndieLLMs
answers whether the proposed map is present, discoverable, structured, and connected to working
resources.

The check stays private and free. IndieLLMs asks for no account, card, or premium plan. It stores no
lookup history and publishes no list of checked sites. Analytics is currently off.

For the separate access-policy question, [check published search and AI crawler rules](https://indiecrawl.com/). A valid resource list does not override robots.txt.

Read [the proposal and its adoption limits](https://indiellms.com/#does-google-or-chatgpt-use-llms-txt). Use [IndieLLMs](https://indiellms.com/) to find the applicable llms.txt and inspect its linked-resource responses.

## Sources

- https://indiellms.com/
- https://llmstxt.org/
- https://llmstxt.org/changes.html
- https://indiecrawl.com/