Deep-crawl any website from start URLs, return per-page LLM-ready text/markdown/HTML plus metadata (title, description, author, language, canonical URL, OG) and in-scope outbound links. Use when user mentions deep crawl website, recursive crawl, crawl a whole site, scrape entire website, scrape docs site, scrape documentation, scrape knowledge base, scrape blog, build RAG corpus, build vector database from website, knowledge base for chatbot, GPT knowledge files, llms.txt, sitemap crawl, BFS cra
Install
KitScout test notes
Not tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.
Similar tools
- apiflash-automation 79,395 ★
Automate Apiflash tasks via Rube MCP (Composio). Always search tools first for current schemas. - Apify Automation 79,395 ★
Automate web scraping and data extraction with Apify -- run Actors, manage datasets, create reusable tasks, an - competitive-ads-extractor 79,395 ★
Extracts and analyzes competitors' ads from ad libraries (Facebook, LinkedIn, etc.) to understand what messagi - diffbot-automation 79,395 ★
Automate Diffbot tasks via Rube MCP (Composio). Always search tools first for current schemas. - clone-website 39,096 ★
Rebuild existing web pages as working local code, matching their real content, assets, responsive layout, and