Scraping Crawling Tools

Cloud platform for web automation, scraping, and actor-based data workflows.

Open-source crawler and scraper designed for LLM-friendly web extraction.

Open-source web scraping and browser automation library from Apify.

Tool and API for turning websites into LLM-ready markdown or structured data.

Python scraping library focused on resilient extraction from changing pages.

Open-source Python framework for building allowed crawlers and web data extraction pipelines.

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.