llm-crawler-access-check

Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot. Use when someone asks whether AI bots are blocked, whether to allow or block GPTBot, why a site never appears in AI answers, or wants a robots.txt review for AI crawlers. Reads only robots.txt, then returns a per-agent allow/block table, the exact rule responsible for each verdict, and the precise lines to change. Distinguishes training crawlers

Install

KitScout test notes

Not tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.

Similar tools

• x-research 107,474 ★
  X/Twitter public sentiment research for recent market, company, product, or community discourse. Use when the
• brightdata-automation 79,395 ★
  Automate Brightdata tasks via Rube MCP (Composio). Always search tools first for current schemas.
• browseai-automation 79,395 ★
  Automate Browseai tasks via Rube MCP (Composio). Always search tools first for current schemas.
• composio-search-automation 79,395 ★
  Automate Composio Search tasks via Rube MCP (Composio). Always search tools first for current schemas.
• contentful-graphql-automation 79,395 ★
  Automate Contentful Graphql tasks via Rube MCP (Composio). Always search tools first for current schemas.