Crawl Endpoints

The /v1/crawls endpoints let you start a crawl, list past and in-progress crawls, and inspect individual crawl runs. A separate /v1/ping endpoint verifies that your API key is valid before you begin.

Start a crawl

POST /v1/crawls

Queues a new crawl and returns immediately with a 202 Accepted. The crawl runs asynchronously — poll the returned URL to track progress.

Request body

{
  "url": "https://example.com",
  "maxUrls": 500,
  "maxDepth": 3
}

Field	Type	Default	Description
`url`	string	—	Required. The absolute URL to begin crawling. Must be `https` or `http`.
`maxUrls`	integer	500	Maximum number of pages to fetch. Capped at 10,000.
`maxDepth`	integer	3	Maximum link depth from the start URL.

Every page actually fetched counts against the organization's monthly URL quota. The quota is metered on pages fetched, not pages discovered — a URL found in a link but never visited does not count.

Response

HTTP/1.1 202 Accepted

{
  "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "url": "/v1/crawls/a1b2c3d4-e5f6-7890-abcd-ef1234567890"
}

The url field is the path you poll until the crawl finishes.

Rate limit

Crawl starts are limited to 10 per minute per API key. Exceeding the limit returns 429 rate_limited with a Retry-After header.

Example

curl -X POST https://api.consuela.io/v1/crawls \
  -H "Authorization: Bearer csl_live_abc123..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "maxUrls": 1000,
    "maxDepth": 2
  }'

List crawls

GET /v1/crawls

Returns a cursor-paginated list of crawls for the authenticated organization, ordered by most recently started first.

Query parameters

Parameter	Type	Default	Description
`limit`	integer	50	Number of results per page. Maximum 200.
`cursor`	string	—	Opaque pagination cursor from a previous response.
`siteId`	UUID	—	Return only crawls for the given site.
`state`	string	—	Filter by crawl state: `running`, `done`, or `failed`.

Response

{
  "data": [
    {
      "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "host": "example.com",
      "state": "done",
      "urlsCrawled": 347,
      "findingsCount": 12,
      "startedAt": "2026-08-12T14:30:00Z",
      "finishedAt": "2026-08-12T14:32:18Z"
    }
  ],
  "nextCursor": "eyJzIjoiMjAyNi0wOC0xMiIsImlkIjoiYTFiMiJ9"
}

findingsCount is null while analysis is still running and a number once complete. A value of 0 means the analysis finished and found nothing — that is a clean result, distinct from null (not yet measured).

Pagination is cursor-based over (started_at, id). When nextCursor is null, there are no more results. Do not parse or construct cursors — they are opaque.

Example

curl "https://api.consuela.io/v1/crawls?limit=10&state=done" \
  -H "Authorization: Bearer csl_live_abc123..."

Get crawl detail

GET /v1/crawls/{id}

Returns the full record for a single crawl.

Response

{
  "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "host": "example.com",
  "state": "done",
  "urlsCrawled": 347,
  "findingsCount": 12,
  "error": null
}

Field	Type	Description
`id`	UUID	Unique identifier for the crawl.
`host`	string	The hostname that was crawled.
`state`	string	`running`, `done`, or `failed`.
`urlsCrawled`	integer	Number of pages fetched so far.
`findingsCount`	integer \| null	`null` until analysis completes. `0` means the analysis ran and the site is clean — not the same as `null`.
`error`	string \| null	Set only when `state` is `failed`. A human-readable description of the failure.

Example

curl https://api.consuela.io/v1/crawls/a1b2c3d4-e5f6-7890-abcd-ef1234567890 \
  -H "Authorization: Bearer csl_live_abc123..."

Auth check

GET /v1/ping

Returns the organization associated with the API key. Use it to verify credentials before starting work.

{
  "ok": true,
  "organizationId": "f9e8d7c6-b5a4-3210-fedc-ba0987654321"
}

Example

curl https://api.consuela.io/v1/ping \
  -H "Authorization: Bearer csl_live_abc123..."

Polling

Crawls run asynchronously. The recommended pattern is to start a crawl with POST /v1/crawls, then poll GET /v1/crawls/{id} until state reaches done or failed.

# 1. Start the crawl
CRAWL=$(curl -s -X POST https://api.consuela.io/v1/crawls \
  -H "Authorization: Bearer csl_live_abc123..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}')

CRAWL_ID=$(echo "$CRAWL" | jq -r '.id')

# 2. Poll until finished
while true; do
  RESULT=$(curl -s "https://api.consuela.io/v1/crawls/$CRAWL_ID" \
    -H "Authorization: Bearer csl_live_abc123...")

  STATE=$(echo "$RESULT" | jq -r '.state')

  if [ "$STATE" = "done" ] || [ "$STATE" = "failed" ]; then
    echo "$RESULT" | jq .
    break
  fi

  sleep 5
done

A five-second interval is a reasonable default. For large crawls with maxUrls above 5,000, consider polling every 15–30 seconds to stay well within read rate limits.

For production integrations that need real-time notification instead of polling, see the webhooks documentation.