Crawl4AI

Category: OSS Agent / Web Crawler for LLM Pipelines

Pricing: Free and open source (Apache-2.0). No API keys required, no paywalls. A managed cloud API is in closed beta (launch date not announced as of 2026-06-30) with early access applications open, but the core library is and will remain free.

Official site: https://docs.crawl4ai.com

GitHub: https://github.com/unclecode/crawl4ai (70.5k stars, Apache-2.0)

---

What it is

Crawl4AI is an async Python web crawler purpose-built for feeding LLM pipelines. It converts pages to clean markdown, supports deep crawls (BFS/DFS), structured data extraction via CSS/XPath/LLM strategies, and ships a CLI (crwl) and Docker image. It runs Playwright/Chromium under the hood so it renders JavaScript pages.

Key facts:

• Zero forced API keys -- works fully offline once installed
• CLI-first: crwl <url> for quick one-off crawls
• Async Python SDK for programmatic use in agents and pipelines
• Deep crawl: BFS/DFS graph crawl, configurable max pages
• LLM extraction: pass a question, get structured answers extracted from pages
• Docker ready: docker pull unclecode/crawl4ai:latest
• Used by 3,100+ projects on GitHub

Safety and ToS notes

Crawl4AI does not enforce robots.txt compliance by default. You are responsible for respecting each target site's robots.txt and terms of service. Do not use it to scrape sites that prohibit crawling or to bypass paywalls. The library ships with a Privacy Policy and Terms of Service linked from the docs site.