Launch Crawl

AI Assistant

Describe what you want to crawl and AI will configure the form for you

Project

Crawl configuration

URL to crawl

Enter the sitemap URL (e.g. https://example.com/sitemap.xml)

List of URLs to crawl

One URL per line. Only these pages will be crawled.

Crawl mode

Subdomains

Max URLs

Max depth

Render JavaScript

Only for SPAs (React/Vue without SSR). Not needed for standard audits — 404s, titles, metas live in the HTML.

Heads up: JS rendering is 5–15× slower and can deadlock on large sites (40k+ URLs). Only enable if the site is a single-page app and content isn't visible without JS. For everything else, leave it off.

Ignore robots.txt

Crawl URLs blocked by robots

User Agent

Exclude patterns

URLs matching these patterns will not be crawled

Limit to patterns (optional)

If specified, only URLs matching these patterns will be crawled

Notification email (optional)

Crawl name (optional)

Log in

AI Assistant

Crawl configuration

Crawl in progress

Crawl history