HTTP Header Analysis

The Headers tab surveys every HTTP response header collected during a crawl. It surfaces security header gaps, cache control directives, and indexing signals that live in headers rather than in page markup.

Stats

Three counts appear at the top of the tab:

Stat	What it counts
Total pages	Every page included in the crawl that returned an HTTP response.
Noindex pages	Pages marked noindex via an `X-Robots-Tag` header, a `<meta name="robots">` tag, or both.
Missing canonical	Pages that returned a successful response but contain no `rel="canonical"` link, either in the HTML or in a `Link` header.

Header analysis

The analysis covers three groups of response headers:

Security headers

Each page is checked for the presence and value of:

Strict-Transport-Security (HSTS) — whether the server instructs browsers to use HTTPS for future requests, and the max-age value.
X-Content-Type-Options — whether the response sets nosniff to prevent MIME-type sniffing.
X-Frame-Options — whether the page restricts being embedded in frames via DENY or SAMEORIGIN.

Pages missing any of these headers are flagged in the findings. A missing security header is not necessarily wrong — it depends on the site — but it is worth knowing about, especially when it varies across pages on the same domain.

Cache control

The tab parses Cache-Control directives and groups pages by their caching behaviour: no-store, no-cache, max-age ranges, and absent headers. This makes it straightforward to spot pages that should be cached but are not, or pages serving stale content because of an overly generous max-age.

X-Robots-Tag

The X-Robots-Tag header carries the same directives as a <meta name="robots"> tag — noindex, nofollow, noarchive, and others — but it is set at the server level rather than in the HTML. It is commonly used for non-HTML resources (PDFs, images) and for pages where editing the markup is impractical.

Both X-Robots-Tag and <meta robots> can set noindex independently. A search engine honours both: if either one says noindex, the page is dropped from the index. They do not need to agree, and a meta robots tag saying index does not override an X-Robots-Tag: noindex header. The Headers tab reports the effective directive for each page, combining the two sources so you see the result a crawler would act on rather than one signal in isolation.

Noindex pages

Below the header analysis, the tab lists the first 20 pages carrying a noindex directive, whether that directive comes from an X-Robots-Tag header, a <meta robots> tag, or both. Each row shows the URL, the source of the directive, and the full directive value.

A separate list shows the first 20 pages that returned a successful response but have no canonical URL — no <link rel="canonical"> in the HTML and no Link header with rel="canonical". Pages without a canonical leave the choice of canonical URL to the search engine, which may not pick the one you intend.

Both lists are capped at 20 entries to keep the tab readable. The full set is available from the results table by filtering the relevant columns.