Usage

Minimal Python example

The core pattern is an AsyncWebCrawler context manager. The result object carries .markdown (clean text), .html, .links, and more.

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com")
        print(result.markdown[:500])

if __name__ == "__main__":
    asyncio.run(main())

Running this prints the first 500 characters of the page rendered as clean markdown.

CLI quick reference

The crwl CLI ships with the package and handles the most common one-off cases:

# Basic crawl, output as markdown
crwl https://docs.crawl4ai.com -o markdown

# Deep crawl with BFS, up to 10 pages
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10

# LLM extraction: ask a question about the page content
crwl https://example.com/products -q "Extract all product prices"

Deep crawl example (Python)

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

async def main():
    config = CrawlerRunConfig(
        deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=2, max_pages=20)
    )
    async with AsyncWebCrawler() as crawler:
        results = await crawler.arun("https://docs.crawl4ai.com", config=config)
        for r in results:
            print(r.url, len(r.markdown))

asyncio.run(main())

Notes

• The crawler launches a real Chromium instance on first run. Expect a 2-3 second startup cost per process.
• For high-volume pipelines, reuse the AsyncWebCrawler context across multiple arun() calls rather than opening a new crawler per request.
• Pass headless=False to AsyncWebCrawler(headless=False) to watch the browser during debugging.