Duplicate Detection

The Duplicates tab in the analysis panel surfaces pages that share titles, meta descriptions, H1 headings, or body content. These overlaps cause search engines to choose which page to surface for a query, and the choice is rarely the one you would make yourself.

What it detects

The analysis runs four checks across every page in the crawl session:

Category	What it compares	Why it matters
Duplicate titles	The exact `<title>` element text, after whitespace normalisation.	The title tag is the primary text a search engine displays in results. Two pages sharing a title compete for the same impression, and the engine picks one.
Duplicate meta descriptions	The `<meta name="description">` content attribute.	A shared description makes two results look interchangeable. Search engines may rewrite one or both, often pulling a less useful snippet from the page body.
Duplicate H1 headings	The text content of the first `<h1>` element on each page.	The H1 is a strong on-page relevance signal. Identical H1s across pages dilute that signal and make it harder for crawlers to distinguish the pages by topic.
Near-duplicate content	A content hash of the page body, with configurable similarity thresholds for fuzzy matching.	Pages with nearly identical body content waste crawl budget. Search engines spend time fetching and indexing pages that add no new information to the index.

How it works

Duplicate detection is computed at session completion, after every page in the crawl has been fetched and stored. The full dataset is required because identifying duplicates means comparing each page against every other page in the session — a comparison that cannot be finalised while new pages are still arriving.

For titles, meta descriptions, and H1 headings, the comparison is an exact match after whitespace normalisation: leading and trailing whitespace is trimmed, and runs of internal whitespace are collapsed to a single space. Two values that differ only in whitespace are treated as duplicates.

For near-duplicate content, Consuela generates a content hash of each page's body text (after stripping navigation, headers, footers, and boilerplate). Pages whose hashes fall within the configured similarity threshold are grouped together. The default threshold catches pages that share the large majority of their content but differ in a date, a product name, or a few paragraphs. You can adjust the threshold in the crawl settings to be stricter (exact matches only) or looser (catching pages with more variation).

The stats bar at the top of the tab shows four counts: total pages analysed, duplicate titles found, duplicate meta descriptions found, and duplicate H1s found. These update once the analysis pass completes.

Reading the results

Results are organised into four sections, one per category. Within each section, duplicates are presented as expandable groups. The group header shows the shared value — the repeated title, description, H1 text, or content hash — and the number of pages that share it. Expanding a group reveals the full list of URLs.

Each section shows the first 20 groups, ordered by the number of affected pages (largest groups first). If a category has more than 20 groups, the remaining count is shown at the bottom of the section. For sites with widespread duplication, the largest groups are almost always the ones worth addressing first: a title shared by 200 product pages is a template issue, not a one-off oversight.

Clicking any URL in a group opens that page in the results table, where you can inspect its full metadata, headers, and other findings. This makes it straightforward to check whether the duplication is intentional (a paginated series, for instance) or an oversight in a template.

A few patterns to watch for when reviewing the results:

Template-generated titles. CMS templates that output the site name as the title for every page without appending the page-specific heading. These show up as one large group in the titles section.
Missing meta descriptions. Pages with no description tag are not grouped as duplicates (there is nothing to compare), but pages whose template emits the same fallback description for every page are. Check whether the "duplicate" is actually a default value.
Boilerplate pages. Terms-of-service pages, privacy policies, and cookie notices that are duplicated across subdomains or language variants. These are often intentional but still consume crawl budget.
Parameterised URLs. Pages that differ only by query parameters (sort order, session IDs, tracking codes) but serve the same content. Near-duplicate detection catches these even when the URLs are distinct.