Crawler

The crawler is the core of the desktop app. Point it at a URL and it walks the site, inspecting every page it finds — collecting status codes, response times, meta tags, headings, links, images, structured data, and dozens of other signals that feed into the results panels. Three crawl modes cover the most common workflows, and the configuration panel lets you tune depth, speed, and scope to match the site.

Crawl modes

Full site crawl

Enter a starting URL and the crawler discovers every reachable page on the domain by following internal links. It begins at the URL you provide, parses the HTML for anchor tags, and queues each new internal URL it finds. The crawl continues until it has visited every discovered page, reached the maxUrls limit, or exhausted the configured maxDepth.

Single page analysis

Fetches and inspects a single URL without following any links. Useful for spot-checking a page after a deploy, investigating a specific finding, or auditing a landing page in isolation. The full suite of checks runs — on-page SEO, structured data validation, accessibility, performance — the same as during a site crawl, just scoped to one page.

URL list crawl

Paste or import a list of URLs and the crawler visits each one without discovering additional pages. This is the right mode when you already know which pages need inspecting — a set of URLs from Search Console, a staging environment where discovery would hit test harnesses, or a batch of pages flagged by a previous crawl that you want to re-check after fixes.

Configuration

The configuration panel controls how the crawler behaves. Defaults are tuned for a typical site audit — polite request rate, reasonable depth, robots.txt respected. Adjust them to match the site's size and your infrastructure's tolerance.

Setting	Default	Range	Description
`maxDepth`	10	1 – 20	Maximum link depth from the start URL. A depth of 1 means only pages linked directly from the start page are visited.
`maxUrls`	10,000	100 – 100,000	Hard cap on the number of URLs the crawler will visit in a single run. The crawl stops when this limit is reached, even if undiscovered URLs remain in the queue.
`concurrency`	5	1 – 20	Number of requests the crawler makes in parallel. Higher values finish faster but put more load on the target server.
`requestsPerSecond`	2	0.5 – 10	Rate limit applied across all concurrent connections. Works alongside concurrency to keep the crawl polite.
`respectRobotsTxt`	`true`	—	When enabled, the crawler fetches and obeys the site's `robots.txt` directives. Disallowed paths are skipped and noted in the results.
`includeSubdomains`	`true`	—	Treat subdomains of the start URL's domain as internal. When disabled, only the exact hostname is considered in scope.
`userAgent`	`Consuela SEO Spider/1.0`	—	The User-Agent header sent with every request. Some sites serve different content or block unknown agents; change this if needed.
`renderJavascript`	`false`	—	Enables headless Chrome rendering. When off, the crawler works with the raw HTML response. See Headless mode for details.
`includePatterns`	`[]`	—	A list of regular expressions. When non-empty, only URLs matching at least one pattern are crawled. Patterns are repeatable — add as many as needed.
`excludePatterns`	`[]`	—	A list of regular expressions. URLs matching any pattern are skipped. Applied after include patterns, so an exclude always wins.
`followExternal`	`false`	—	When enabled, the crawler follows links to external domains. External pages are fetched for status and response headers but are not crawled for further links.
`timeoutSeconds`	30	1 – 300	Per-request timeout. If a page does not respond within this window the request is recorded as a timeout failure.
`maxPageSizeMb`	10	1 – 100	Maximum response body size in megabytes. Responses larger than this are truncated to keep memory use in check.
`storeHtml`	`false`	—	Saves the raw HTML of every crawled page to the project database. Enables full-text search and historical diffing, but large crawls can grow to multiple gigabytes on disk.
`extractionRules`	`[]`	—	Custom extraction rules using CSS selectors, XPath expressions, or regular expressions. Each rule names a column that appears in the results and defines how to pull its value from the page.

Headless mode

Setting renderJavascript to true launches a headless Chrome instance behind the crawler. Every page is loaded in a real browser, JavaScript executes, and the fully rendered DOM is what gets inspected. This is essential for single-page applications and any site where content is injected client-side.

Headless mode unlocks a broader set of inspections that are not available from raw HTML alone:

Core Web Vitals — Largest Contentful Paint, Cumulative Layout Shift, and Interaction to Next Paint, measured in the browser.
Screenshots — a viewport capture of each page at both desktop and mobile sizes.
Accessibility audit — an automated accessibility inspection run against the rendered page.
Console errors — JavaScript errors, warnings, and failed resource loads captured from the browser console.
Resource analysis — a breakdown of every resource the page loads: scripts, stylesheets, images, fonts, with sizes and timing.
Rendering analysis — detection of content that only appears after JavaScript runs, useful for identifying indexability risks.
Security checks — mixed-content warnings, insecure resource loads, and Content Security Policy evaluation.
Mobile analysis — tap target sizing, viewport configuration, and font legibility at a mobile viewport.

Headless mode has its own configuration defaults:

Setting	Default	Description
`captureScreenshots`	`true`	Saves a viewport screenshot for each page at both desktop and mobile sizes.
`collectWebVitals`	`true`	Measures Core Web Vitals (LCP, CLS, INP) for each page.
`runAccessibilityAudit`	`true`	Runs an automated accessibility inspection against the rendered DOM.
Desktop viewport	1920 × 1080	Viewport size used for desktop screenshots and analysis.
Mobile viewport	375 × 812	Viewport size used for mobile screenshots and analysis.
`waitAfterLoadMs`	500	Milliseconds to wait after the page load event before capturing. Gives late-loading scripts time to finish rendering.

Headless crawls are significantly slower and more resource-intensive than HTML-only crawls. For large sites, consider running a standard crawl first to identify problem areas, then a targeted headless crawl on the pages that need the deeper inspection.

Controls

The crawl toolbar provides four controls:

Start — begins the crawl with the current configuration. Disabled while a crawl is running.
Pause — suspends the crawl. In-flight requests complete, but no new requests are dispatched. The queue is preserved.
Resume — continues a paused crawl from where it stopped.
Stop — ends the crawl immediately. In-flight requests are cancelled. Results collected so far are kept and available for inspection.

While a crawl is running, the progress bar displays live statistics: the number of URLs crawled, the number still queued, the number that failed, the current request rate per second, and the elapsed time. These update in real time and remain visible after the crawl finishes.