Launch Crawl

AI Assistant

Describe what you want to crawl and AI will configure the form for you

Project

URL to crawl

Enter the sitemap URL (e.g. https://example.com/sitemap.xml)

List of URLs to crawl

One URL per line. Only these pages will be crawled.

Crawl mode

Subdomains

Max URLs

Max depth

Render JavaScript

Required for SPAs and dynamically loaded sites

Ignore robots.txt

Crawl URLs blocked by robots

User Agent

Exclude patterns

URLs matching these patterns will not be crawled

Limit to patterns (optional)

If specified, only URLs matching these patterns will be crawled

Notification email (optional)

Crawl name (optional)