Scraping And Crawling Tools
Scraping Crawling AI ToolsAllowed web data extraction, browser automation, crawling, and content-to-markdown tooling for builders.
Use caseNeeds verification
Tool · Needs verification
Apify
Cloud platform for web automation, scraping, and actor-based data workflows.
ToolNeeds verificationOpen sourceSelf-hostedFreemiumGitHub linked
Review gate
Final scoring is withheld because this record is currently Needs verification. Official links, pricing, license, and publication summary need manual review before indexing.
Tool fit
Primary repository
Mapped use cases
Related repositories
apify/crawlee
TypescriptCrawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
RepositoryAuto enrichedTypeScriptApache-2.0Active signal
Related skills
API Literacy
EngineeringUnderstanding authentication, rate limits, request shapes, errors, and source attribution.
Role: Recommended
SkillNeeds verificationEngineeringRequired
Browser Automation
AutomationDriving browsers safely for testing, research, and user workflow automation.
Role: Recommended
SkillNeeds verificationAutomationRequired
Web Scraping Ethics
GovernanceUsing APIs, robots.txt, rate limits, attribution, and allowed collection methods.
Role: Recommended
SkillNeeds verificationGovernanceRequired