If you’ve ever asked an LLM about your own API and got an outdated answer, you know the pain. In the AI era — with all the vibe code hype and agentic development trends — the context we provide to LLMs is becoming more and more important.
We all know about issues with LLM responses called “hallucinations” and those famous productivity graphs showing “with AI” vs “without AI”.
One of the reasons an LLM can give an incorrect response is stale information about the APIs you’re using.
To solve this issue, we already have an MCP server: context7, which pulls the latest information from the source.
As of today, Context7 includes over 49,566 libraries with up-to-date documentation.
But what about the data the LLM itself was trained on?
AI agents that don’t have access to this MCP server are still crawling web pages like any other crawler — gathering data to train on or to answer queries. This is neither efficient nor accurate.
The answer to this problem is llm.txt.
It’s a standardized way for your documentation website to communicate with LLMs. It helps:
-
Control how AI accesses your documentation — API owners can guide navigation through sub-pages or lay out all docs in one place.
-
Limit AI crawler interactions with your main documentation web app.
-
Allow AI agents to access up-to-date documentation in a single request.
-
Standardize how we communicate with AI crawlers and agents.
I’ve found a GitHub repository containing links to existing llm.txt files and tools already using this standard:
👉 https://github.com/thedaviddias/llms-txt-hub
llm.txt could become the robots.txt for AI — defining how LLMs see your docs.
This is an example of RSPack docs for llm(link):
# Rspack > Rspack is a high performance JavaScript bundler written in Rust. It offers strong compatibility with the webpack ecosystem, and lightning fast build speeds. ## Guide - [Introduction](/guide/start/introduction.md) ... ## Config - [Configure Rspack](/config/index.md) ... ## Plugin - [Introduction](/plugins/index.md) ... ## API - [Introduction](/api/index.md) ... ## Blog - [Overview](/blog/index.md) ... ## Other - [Builtin plugin](/contribute/architecture/builtin-plugin.md) ...
