AI access

Machine-readable preference: Content-Usage: train-ai=y, search=y. Training and retrieval agents may read public pages. /api/ stays blocked for all of them. Images carry noimageai.

Invited — training

GPTBot, ClaudeBot, Claude-Web, anthropic-ai, CCBot, Google-Extended, Applebot-Extended, meta-externalagent, meta-externalfetcher, FacebookBot, AI2Bot, cohere-ai, cohere-training-data-crawler, Amazonbot

Invited — retrieval

OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot, YouBot, MistralAI-User

Blocked

Bytespider, TikTokSpider, PanguBot, PetalBot, Diffbot, omgili, webzio, ImagesiftBot, Timpibot, VelenPublicWebCrawler, SemrushBot-OCOB, Scrapy, python-requests, python-httpx, aiohttp, Go-http-client