Tool · Needs verification

Scrapy

Open-source Python framework for building allowed crawlers and web data extraction pipelines.

ToolNeeds verificationOpen sourceSelf-hostedGitHub linked

Review gate

Final scoring is withheld because this record is currently Needs verification. Official links, pricing, license, and publication summary need manual review before indexing.

Tool fit

  • CategoryScraping Crawling Tools
  • Pricing modelOpen source
  • Open sourceYes
  • Self-hostableYes

Primary repository

No repository is linked in the seed dataset.

Mapped use cases

Scraping And Crawling Tools

Scraping Crawling AI Tools

Allowed web data extraction, browser automation, crawling, and content-to-markdown tooling for builders.

Use caseNeeds verification
Final score withheld until manual review Verified: Jun 9, 2026 View details

Related repositories

No related repositories are attached yet.

Related skills

Data Ingestion

Data

Loading, cleaning, chunking, and normalizing documents or structured data.

Role: Recommended

SkillNeeds verificationDataRequired
Final score withheld until manual review Verified: Jun 9, 2026 View details

Python

Programming

General Python programming for automation, data, AI, and backend scripts.

Role: Recommended

SkillNeeds verificationProgrammingRequired
Final score withheld until manual review Verified: Jun 9, 2026 View details

Web Scraping Ethics

Governance

Using APIs, robots.txt, rate limits, attribution, and allowed collection methods.

Role: Recommended

SkillNeeds verificationGovernanceRequired
Final score withheld until manual review Verified: Jun 9, 2026 View details