Custom Extraction

Custom extraction lets you define your own data-extraction rules and apply them across every page in a crawl. Each rule pulls specific content — a meta tag value, a price, a schema field, a pattern buried in inline script — and surfaces it in a dedicated results table where you can sort, filter, and export the findings.

Rule types

Three rule types cover the range from simple element selection to arbitrary pattern matching. Choose the one that fits what you need to extract.

Type	Expression	What it extracts	Example
CSS selector	Any valid CSS selector	The text content of each matching element, or the value of a specified attribute.	`meta[name="author"]` with attribute `content` returns the author meta value.
Regex pattern	A regular expression	Every match against the full page source. An optional capture group number isolates a specific part of the match.	`data-price="(\d+\.\d2)"` with capture group `1` returns the price value alone.
XPath	An XPath 1.0 expression	The text content of each node matching the expression.	`//h1[@class="product-title"]/text()` returns the text inside the product title heading.

 
CSS selectors work well for anything that lives in the DOM as a visible element or
    standard attribute. Use regex when the data you need is inside a script block,
    a comment, or any context where the DOM does not expose it as a node. XPath is
    useful when the selection logic requires traversal that CSS selectors cannot
    express, such as selecting a node based on its sibling or ancestor content.
 Creating rules
 
Open the Custom Extraction tab in the analysis panel. The rule builder sits at
    the top of the view with three fields:
  Name — a label for the rule. This becomes the column header in the results table. Names must be unique across the current set of rules.
 Type — one of CSS selector, Regex pattern, or XPath.
 Expression — the selector, pattern, or XPath to evaluate. For regex rules, an additional capture group field appears; leave it empty to use the full match.
 
 
Rules are validated before they are added. A CSS selector that cannot be parsed,
    a regex with unbalanced groups, or a malformed XPath expression will show a
    validation error and the rule will not be saved until the expression is corrected.
 
You can add rules before starting a crawl, during a running crawl, or after one
    has finished. Rules added mid-crawl apply to pages that arrive after the rule is
    created; pages already processed are not automatically re-evaluated. Rules added
    after the crawl has completed require an explicit extraction run to produce
    results.
 Running extraction
 
The toolbar above the results table offers three actions:
     Action  What it does  When to use it  
 
   Run Extraction  Evaluates all rules against the crawl results currently held in memory.  The crawl has finished and all rules were in place before or during the crawl. This is the fastest option since it works from data already loaded.  
  Re-run on stored HTML  Re-evaluates all rules against the raw HTML saved to disk during the crawl.  Rules were added or modified after the crawl completed. Requires that storeHtml was enabled in crawl settings before the crawl ran. Without stored HTML, this option is disabled.  
  Export CSV  Writes the current extraction results to a CSV file with one row per extracted value.  You need the data outside Consuela — in a spreadsheet, a reporting tool, or a script. The one-row-per-value format handles multi-match rules cleanly.  
 
 
 
 
When rules have been added after the crawl already ran, a warning banner appears
    at the top of the extraction view. It indicates which rules have no results yet
    and suggests using Re-run on stored HTML to evaluate them against
    the full page content from the completed crawl. If storeHtml
was not enabled, the banner notes that a new crawl is needed to populate those rules.
 Results and export
 
Extraction results appear in a table with a URL column followed
    by one column per rule. Each cell shows the values extracted from that page by
    that rule. When a rule produces multiple matches on a single page — several
    elements matching a CSS selector, or multiple regex hits in the source — the cell
    displays the first three values and a +N more indicator for any
    beyond that. Click the cell to expand and see the full list.
 
The table supports the same sorting and filtering as the main results table.
    Filter by a specific rule column to find pages where a value is present, absent,
    or matches a search term.
 
The CSV export writes one row per extracted value rather than one row per page.
    A page that matches three times for a given rule produces three rows, each with
    the same URL and a different value. This flat format avoids multi-value cells
    and is easier to work with in downstream tools that expect one observation per row.
  Related  Crawler Results table Export

Action	What it does	When to use it
Run Extraction	Evaluates all rules against the crawl results currently held in memory.	The crawl has finished and all rules were in place before or during the crawl. This is the fastest option since it works from data already loaded.
Re-run on stored HTML	Re-evaluates all rules against the raw HTML saved to disk during the crawl.	Rules were added or modified after the crawl completed. Requires that `storeHtml` was enabled in crawl settings before the crawl ran. Without stored HTML, this option is disabled.
Export CSV	Writes the current extraction results to a CSV file with one row per extracted value.	You need the data outside Consuela — in a spreadsheet, a reporting tool, or a script. The one-row-per-value format handles multi-match rules cleanly.