Cloudflare AI crawlers

AI crawlers like GPTBot, ClaudeBot and PerplexityBot don't run JavaScript. They hit /robots.txt, sitemaps and raw HTML endpoints, so the TrustData JS SDK never sees them. A Cloudflare Worker deployed on your zone classifies every request at the edge and forwards a small log line to TrustData. Each forwarded hit is one of:

• AI bot visit: the user agent matches a known AI crawler
• AI referral visit: the referrer is an AI engine (chatgpt.com, perplexity.ai, claude.ai, …) and the visitor is human
• Traffic sample: a small anonymized share (~2% by default) of everything else, used only as the denominator for AI-share metrics

All other traffic never leaves your zone. The Worker matches against a bot list it syncs from TrustData every few hours, so brand-new crawlers are picked up without a redeploy.

How it works

Setup

1. Issue an API key

1. In your organization settings, open the Integrations tab → Log ingest keys
2. Pick Cloudflare Worker (or Cloudflare Logpush for Enterprise) as the provider, name the key, and click Issue ingest key
3. Optionally pin a default attribution ID. The Worker will tag events with this ID when the payload omits one
4. Copy the key immediately, as it is shown only once

Keys use the format td_cf_<random>. Only the prefix and a SHA-256 hash are stored in TrustData; the full key cannot be recovered. If lost, revoke it and generate a new one.

2. Send logs to TrustData

Pick your ingest method. Most zones use the Worker; Cloudflare Enterprise customers can forward the same data over Logpush with no Worker. Both authenticate with the td_cf_... key from step 1.

POST https://t.trustdata.tech/v1/logs/cloudflare_worker

[
  {
    "timestamp": 1740000000000,
    "attribution_id": "prop-uuid",
    "host": "example.com",
    "method": "GET",
    "pathname": "/blog/post",
    "query_params": { "utm_source": "chatgpt" },
    "ip": null,
    "user_agent": "Mozilla/5.0 (compatible; GPTBot/1.0; …)",
    "referer": "",
    "status": 200,
    "bytes": 4821,
    "country": "US",
    "asn": 13335,
    "verified": true,
    "verified_by": "signature"
  }
]

POST https://t.trustdata.tech/v1/logs/cloudflare_logpush

?header_X-API-Key=td_cf_...&attribution_id=<prop-uuid>

VerifiedBotCategory in [AI_CRAWLER, AI_ASSISTANT, AI_SEARCH]
  or ClientRequestReferer contains "chatgpt"
  or ClientRequestReferer contains "perplexity"
  or ClientRequestReferer contains "claude.ai"
  or ClientRequestReferer contains "gemini"
  or ClientRequestReferer contains "copilot"

3. Verify

From any terminal, simulate a GPTBot visit:

curl -X POST \
  https://your-domain.com/ \
  -H "User-Agent: Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)"

Within ~60 seconds the event should appear in the Live Events view for your attribution ID, tagged with an AI bot badge. Navigate via Analytics → Properties → your property → Attribution IDs → your attribution ID → Live Events. If not, see Troubleshooting.

What gets captured

The full bot and referrer lists are maintained on TrustData's side and synced into the Worker every few hours, so a new AI crawler is covered automatically with no redeploy. Over Logpush, coverage is retroactive only for the traffic your filter forwards (see the Logpush tab).

Viewing crawler activity

Crawler events surface in the Live Events view for your attribution ID, one row per hit:

• Rows with an AI bot badge, where the user agent matched a known AI crawler
• Rows with an AI referral badge, where a human visitor arrived from an AI engine
• Any other event with a small bot tag, which is traffic the server flagged as non-human from the browser SDK

Live Events shows the last 30 minutes, refreshing every few seconds. Aggregated per-bot counts by day and page surface in your dashboard and refresh daily.

Troubleshooting

401 Unauthorized

• Check the header is exactly X-API-Key: td_cf_... (no Bearer prefix)
• Verify the key is still Active under Organization → Integrations → Log ingest keys
• Confirm no extra whitespace was added to the key when you entered it in the guided deploy form

No events appear in the dashboard

1. Check the Worker is deployed and receiving traffic via its real-time logs: Workers & Pages → trustdata-ai-bot-collector → Logs → Begin log stream in the Cloudflare dashboard
2. Confirm the UA matches a known bot by trying a curl with User-Agent: GPTBot/1.0
3. Verify TRUSTDATA_ATTRIBUTION_ID is a real attribution ID UUID from Analytics → Properties → your property → Attribution IDs
4. Check that TRUSTDATA_INGEST_URL points to https://t.trustdata.tech/v1/logs/cloudflare_worker (no trailing slash)

Events arrive in Live Events but aggregate dashboards lag

Live Events is real-time. The aggregated per-day / per-page bot dashboards refresh once a day (around 3:00 AM UTC), so those break­downs lag by one run. If it's been longer than 24 hours, contact support.

Worker causes latency on the customer response

It shouldn't, because the Worker uses ctx.waitUntil() to fire-and-forget the TrustData POST after returning the origin response. If you see added latency, check the Worker's real-time Logs in the Cloudflare dashboard for an errored forward. Transient ingest failures are swallowed, but a bad TRUSTDATA_INGEST_URL can cause DNS resolution delays.

WebMCP hosting (automatic)

Once deployed, the same Worker also hosts your WebMCP manifest at /.well-known/webmcp.json. AI agents read this file to discover which tools your site exposes (search, add-to-cart, booking, contact…) before loading any page.

You don't need a separate build step. TrustData serves the signed manifest and the Worker caches it on the edge.

How the manifest is served

1. Agent requests https://<your-zone>/.well-known/webmcp.json
2. Worker checks Cloudflare KV for a cached copy (webmcp:v1:<attribution_id>, 1-hour TTL)
3. On cache miss, Worker fetches from TRUSTDATA_MANIFEST_URL + your attribution ID
4. Response is returned with Content-Type: application/json and Cache-Control: public, max-age=3600
5. Agent verifies the Ed25519 signature inside the JSON body and calls the declared tools

Enable it

WebMCP hosting is on by default in wrangler.jsonc:

"vars": {
  "TRUSTDATA_MANIFEST_URL": "https://app.trustdata.tech/api/v1/webmcp",
  "TRUSTDATA_ATTRIBUTION_ID": "<your property UUID>"
},
"kv_namespaces": [
  { "binding": "WEBMCP_CACHE", "id": "<your KV namespace id>" }
]

The Deploy to Cloudflare button provisions this KV namespace automatically and writes its ID into wrangler.jsonc, so there's nothing to create by hand. To add or change the binding later, use Workers & Pages → trustdata-ai-bot-collector → Settings → Bindings in the Cloudflare dashboard.

Declare the tools

In TrustData, go to Settings → Attribution Properties → your property → WebMCP and add 2–3 tools:

• search_products, input schema: { query: string }
• add_to_cart, input schema: { sku: string, quantity: number }
• contact_sales, input schema: { email: string, message: string }

Save. On the next Worker cache miss (at most 1 hour later), agents will see your updated manifest. Click Rotate keys to invalidate every cached signature immediately, which is useful if you suspect a leaked key.

Disable WebMCP hosting

Remove the kv_namespaces block and set TRUSTDATA_MANIFEST_URL to an empty string. The Worker falls through to your origin for /.well-known/webmcp.json, so any existing static file you serve there keeps working.

Integrations overview on trustdata.tech · Free trial