Why was that request blocked? Tracing one customer at 100% with Cloudflare Traces and Trace Rules

A customer writes in: their checkout request was blocked, or it took eight seconds, and they sent a screenshot with a Ray ID. Until now, answering that meant stitching together Security Events, cache analytics and your own origin logs by timestamp. Cloudflare Traces, announced in open beta on October 2, puts those steps into one trace: security rule evaluations, request transforms, cache decisions, routing, Workers and the origin call, as nested spans.

This is a recipe for the support case above: keep tracing cheap for normal traffic, capture 100% of requests for the one host or customer you are debugging, join Cloudflare's spans to your backend's, and ship them to an OpenTelemetry collector you run. It is built from Cloudflare's announcement and the Traces documentation; the feature is a beta, so check the dashboard labels against the docs links at the end if anything has moved.

What a trace contains

According to the docs, Traces currently covers spans for Rules, request routing, Cache, Workers and origin connections. The announcement shows span names such as http_request_transform and workers_routing, with cache, upstream and origin spans nested underneath. The four questions Cloudflare says it is built to answer are the ones support tickets usually ask:

• Which security rule blocked or challenged this request?
• Did a Transform Rule rewrite the URL before it reached the app?
• Which Page Rule, Snippet or Worker handled it?
• Was it a cache hit, and how was the time split between Cloudflare, the origin and the application?

One naming trap: Cloudflare Trace (singular) is an older tool that simulates how your configuration would treat a hypothetical request. It does not show production traffic. Cloudflare Traces records real requests. Use the second one for a customer complaint.

Step 1: enable tracing with a low baseline

Tracing is enabled per domain. The configuration page has a Default sample rate (%): at 10%, roughly 10 of every 100 requests are traced. For a small site, start low. Cloudflare's own example of normal operation is 1%, which still gives you a steady sample of what typical requests look like without filling your ingestion allowance.

Head sampling means the decision is made when the request arrives. A request that was not sampled is not recorded, even if it later turns out to be the one a customer complains about. That is why the next step matters.

Step 2: a Trace Rule that captures 100% of one slice

Trace Rules override the default rate for requests that match an expression written in the same Rules language you use for WAF custom rules and Transform Rules. Cloudflare uses the first matching rule, so put the narrow debug rules above any broad ones.

Three rules that cover most support cases. A whole hostname that is misbehaving:

http.host eq "checkout.example.com"

One customer's office or a test machine, by source IP:

ip.src eq 203.0.113.42

A debug header that you, or the customer's support contact, add on purpose:

any(http.request.headers["x-debug-trace"][*] eq "1")

Set each to a sample rate of 100%, deploy, and reproduce the problem. The header rule is the most useful one day to day: anyone on your team can force a full trace from a terminal without touching the configuration again.

curl -sS -o /dev/null -w "%{http_code} %{time_total}s\n" \
  -H "x-debug-trace: 1" \
  "https://checkout.example.com/api/cart"

Two cautions. A header anyone can send means anyone can raise your trace volume, so remove or narrow the rule when the investigation ends. And an IP rule only helps while that customer's address is stable.

Step 3: find the request

In the Traces view, filter by Ray ID to open the trace for one request. You get the full request path as a tree of spans; expanding a span shows its status, service, trigger, and the span and trace IDs. For a block, look for the security span with an error status and the rule that matched. For a slow request, compare the duration of the origin span with the total: if most of the time is under the origin, the problem is yours, not Cloudflare's.

Step 4: join the trace to your backend

Traces speaks W3C Trace Context, with two separate switches:

• Incoming trace context. The default policy is Reject: Cloudflare ignores a traceparent header sent by the client and starts its own trace. You can switch it to accept, so that a trace started in your frontend or mobile app continues through Cloudflare. The docs warn that incoming context is unverified, and joined traces are treated as untrusted. Accept only if you need client-side spans.
• Forward to origin. Turn this on and Cloudflare includes trace context in the request it sends to your origin. If your backend is instrumented with OpenTelemetry, its spans become children of Cloudflare's spans in the same trace.

Before you trust the join, check that the header actually arrives. A temporary log line at the origin is enough; for example, in an nginx log_format:

log_format traced '$remote_addr "$request" $status traceparent="$http_traceparent"';

Send a request with the debug header and confirm the log shows a value in the form 00-<trace-id>-<span-id>-01. If it is empty, forwarding is off or the request was not sampled.

Step 5: export over OTLP to your own collector

Export is configured in two places. At the account level you create a destination: a name (for example grafana-traces), the type Traces, the OTLP endpoint, and any authentication headers your backend needs. Then, in each domain's settings, you add that destination under Export destinations. Cloudflare lists Honeycomb, Grafana Cloud, Axiom, Sentry, Datadog, New Relic, SigNoz and others as supported providers.

One detail matters if you run your own collector: Cloudflare does not send binary (protobuf) OTLP, so the receiving endpoint must accept OTLP/HTTP with JSON. The OpenTelemetry Collector's OTLP HTTP receiver does. A minimal collector config that receives from Cloudflare, checks a bearer token, and forwards to a Tempo instance:

extensions:
  bearertokenauth:
    token: "${env:CF_TRACES_TOKEN}"

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
        auth:
          authenticator: bearertokenauth

processors:
  batch: {}

exporters:
  otlp:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  extensions: [bearertokenauth]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

The bearertokenauth extension ships in the collector's contrib distribution, so run the otel/opentelemetry-collector-contrib image. Put TLS in front of port 4318 (your reverse proxy is fine), use https://traces.example.com/v1/traces as the endpoint in the Cloudflare destination, and add the header Authorization: Bearer <token>. Your origin's own OpenTelemetry SDK can send to the same collector, which is what makes the joined trace show up as one tree in Grafana.

What it costs from December 1

Traces is free while in beta. Cloudflare's published pricing starts on December 1, 2026:

• Free plan: 0.5 GB of ingestion per day, 7 days of retention, no overage. When you hit the allowance, you stop ingesting.
• Paid and Enterprise: 50 GB of ingestion and 10 GB-month of storage included per billing cycle, retention up to one year, then $0.25 per GB ingested and $0.10 per GB-month stored.

Cloudflare prices by gigabytes, not by trace, and trace size depends on how many rules, Workers and subrequests a request touches. So measure, do not guess: run your normal baseline for a week during the beta and read the ingested volume. As plain arithmetic on the published rates, a Paid zone that ingests 2 GB a day (about 60 GB a month) would pay for 10 GB over the allowance, $2.50 a month for ingestion. If you export to your own collector and do not need traces kept in the Cloudflare dashboard, storage is the line to watch.

The short version

1. Enable Traces on the domain with a 1% default sample rate.
2. Add a 100% Trace Rule on a debug header (and, when needed, one host or one IP), placed first.
3. Reproduce with x-debug-trace: 1, open the trace by Ray ID, and read the security, cache and origin spans.
4. Turn on Forward to origin; leave incoming context on Reject unless you need client spans.
5. Export to an OTLP/HTTP JSON endpoint you run, then remove the broad debug rules when the ticket is closed.

Sources

• Cloudflare blog: Cloudflare Traces (open beta, pricing)
• Cloudflare docs: Traces overview and inspecting a trace
• Cloudflare docs: Traces configuration (sampling, Trace Rules, trace context)
• Cloudflare docs: OpenTelemetry export destinations
• Cloudflare docs: Rules language