Should I build an MCP server for my site now?

For most small-business websites it is premature. MCP is an open standard still being revised, and per its own specification, security practices such as explicit user consent are still evolving. It is worth evaluating mainly if you already operate an API.

Do AI agents obey robots.txt?

Automated crawlers generally do, but OpenAI's documentation states that ChatGPT-User, which fetches pages at a user's direct request, is not an automatic crawler, so robots.txt rules may not apply to those fetches.

How do I make my website readable and usable by AI agents?

The practical preparation for AI agents today is groundwork: structured data (JSON-LD), unambiguous state information like hours, prices, and availability, and a deliberate AI-crawler policy. Agents already read the live web on users' behalf; letting them act on your site through standards like MCP is still early, so build the fundamentals first.

Updated 2026-07-12

Frequently asked questions

How do I make my website readable and usable by AI agents?: Start with what works now: mark up facts with structured data, keep state information (hours, prices, availability) explicit and consistent, and set an intentional robots.txt policy per crawler type. Agent action standards like MCP are promising but still early.
Should I build an MCP server for my site now?: For most small-business websites it is premature. MCP is an open standard still being revised, and per its own specification, security practices such as explicit user consent are still evolving. It is worth evaluating mainly if you already operate an API.
Do AI agents obey robots.txt?: Automated crawlers generally do, but OpenAI's documentation states that ChatGPT-User, which fetches pages at a user's direct request, is not an automatic crawler, so robots.txt rules may not apply to those fetches.