Cloudflare Web Search API: choose a provider by privacy, cost, and result quality. Quick answer. Cloudflare Web Search API gives an agent one interface for Ceramic.ai, Exa, and Linkup. Do not treat them as interchangeable. Pin the provider, classify query sensitivity, measure accepted-answer cost, and keep a tested fallback. For sensitive searches, use the stricter documentation: Cloudflare's provider table marks Ceramic and Linkup as Zero Data Retention, but Exa as not ZDR. Who this is for. This guide is for developers adding live web grounding to an agent, chatbot, research workflow, or automation, especially on Workers or AI Gateway. If you are also routing model inference through AI Gateway, pair this checklist with the AI Gateway model-routing validation guide. What changed and why it matters. Cloudflare launched Web Search API in open beta on October 2, 2026. A backend can call a REST endpoint, while a Worker can call env.AI.websearch. Every request passes through an AI Gateway and returns a normalized list containing URLs, titles, and provider-dependent descriptions plus request metadata. The API supports Ceramic, Exa, and Linkup. ceramic is the default. Queries can contain up to 1,024 characters and return at most ten results. Payment uses AI Gateway credits or a stored BYOK provider key. The normalized schema does not equalize privacy, price, coverage, freshness, or relevance. Those differences belong in the routing policy. Provider contract and evidence boundary. | Provider | Current documented route | ZDR in provider table | Published list price | Documented result behavior | | Ceramic.ai | Default; independent web index | Yes | $0.25 / 1,000 requests | Long descriptions, up to 8,000 characters | | Exa | provider: "exa" No | $7.00 / 1,000 requests | auto search with query-relevant highlights | | Linkup | provider: "linkup" Yes | $5.00 / 1,000 requests | fast depth with raw search results | These are Cloudflare's terms, not IndieSeek tests. Verify the live documentation, account terms, billing, and result quality before production use. There is also a documentation conflict worth treating as a release gate: the launch changelog says all three providers support Zero Data Retention through Cloudflare, while the detailed provider page says Exa does not. Until Cloudflare reconciles those pages or confirms your account terms, keep sensitive queries away from Exa and record the source used for the decision. A seven-step rollout workflow. 1. Classify the query before search. Separate public research from customer data, credentials, private repository text, internal URLs, and personal data. Redact or refuse sensitive input before search. ZDR is not permission to disclose data. 2. Pin the provider and credential path. Always set provider; do not let a future default change cost or handling. Decide whether each route uses Gateway credits or a named BYOK alias. A missing byokAlias fails with 400 rather than falling back to credits - use that fail-closed behavior for cost ownership. 3. Normalize only the transport. Map the response into your schema, but preserve provider, query, request ID, latency, URL, title, description presence, and retrieval time. Never turn a missing field into an empty-but-trusted fact. 4. Build a frozen query canary. Test navigational queries, exact docs, recent announcements, obscure entities, non-English queries, ambiguity, and zero-result cases. Keep expected domains and freshness windows, not one fragile ranking. 5. Measure accepted-answer cost. Search price is only one term: accepted-answer cost = (search requests + model tokens + retries + review + correction) / accepted grounded answers A cheap provider can become expensive if long descriptions inflate context or weak results trigger repeats. Cap result count and description bytes before the model. 6. Verify citations and freshness. Require the answer to cite returned URLs. Re-fetch or reject high-impact claims when a date is missing, the source is secondary, or the page no longer supports the statement. A successful call proves retrieval, not truth. 7. Canary one low-risk workflow. Start with read-only public research. Track no-result rate, expected-domain hits, citation support, latency, accepted-answer cost, and privacy violations. Your application must implement provider fallback. Roll back when any gate fails. Provider decision tree. Does the query contain non-public or sensitive data? yes -> redact or refuse; if search remains necessary, require a verified ZDR route no -> Is low request price the main constraint? yes -> canary Ceramic first no -> Do you need concise query-relevant highlights? yes -> evaluate Exa only if its privacy terms fit no -> compare Linkup fast results with Ceramic Did the selected provider miss the quality or freshness gate? yes -> retry once with a pre-approved fallback and record both attempts no -> pass bounded results to the model and verify citations Search receipt template. policy_version: web-search-v1 query_class: public-release-check provider: ceramic credential_path: ai-gateway-credits result_limit: 5 request_id: <gateway-request-id> latency_ms: 612 returned_urls: 5 expected_domain_hit: true freshness_gate: passed context_bytes: 9400 model_answer_accepted: true fallback_used: false Common mistakes. * Omitting provider and accidentally making the current default a permanent policy. * Treating one normalized response schema as equal privacy, quality, or freshness. * Repeating the launch post's ZDR statement without checking the conflicting provider table. * Comparing request prices while ignoring description size and model-token cost. * Logging sensitive raw queries merely because AI Gateway provides observability. * Calling URLs citations without checking that they support the generated claim. * Retrying across providers without a budget, privacy gate, or deduplication key. Make your Mac notch useful with SuperNotch - more than 20 native tools for music, clipboard, focus, screenshots, system controls, and more. Faq. Is Cloudflare Web Search API generally available? No. Cloudflare labels it open beta. Keep a versioned adapter and rollback for changing request shape, providers, pricing, and behavior. Which provider should I use by default? There is no universal winner. Ceramic is Cloudflare's current default and cheapest published option, but your choice should come from sensitivity, query mix, result quality, context cost, and fallback tests. Does the API fetch full pages? The response contains titles, URLs, and optional descriptions and metadata. For page evidence, fetch and validate the selected source separately under its access rules. Can AI Gateway automatically fail over between search providers? The Web Search request selects one provider. Implement and test search-provider fallback in your application unless Cloudflare documents a native mechanism later.

Cloudflare's Pay Per Crawl: what it means for your site. Cloudflare's Pay Per Crawl lets websites charge AI bots for scraped content. Here is what it means for small business owners, and what to check today. AI consultant and marketing strategist, 10+ years in marketing and AI AI-readable Starting September 15, 2026, Cloudflare will block AI training and agent crawlers by default on ad-supported pages. AI companies can still get in, but only if they pay through Cloudflare's Pay Per Crawl marketplace. If you run a small business website, that block-allow-or-charge decision now lives in a dashboard setting most owners have never opened. This isn't a future problem. It's a setting that already exists, sitting on default, deciding right now whether ChatGPT, Claude, and Gemini can read your site, and whether you get anything back for it. What changed on September 15, 2026? Cloudflare's new default blocks AI training and AI agent crawlers on any page that carries ads, unless the crawler's owner pays. The block applies to what Cloudflare calls "mixed-use crawlers," bots that combine search indexing, AI agent actions, and AI model training in one crawler. Cloudflare names Googlebot, Applebot, and Bingbot as examples (Cloudflare, "Your site, your rules"). Search crawling stays allowed by default. Training and agent crawling do not. Cloudflare says it's encouraging those companies to split their bots into three separate crawlers, so site owners can allow search while still blocking training. The new default applies to new Cloudflare customers, new sites on existing accounts, and every free-tier account. If you already run a paid Cloudflare account with live sites, your current settings stay put until you change them. This builds on Pay Per Crawl, the marketplace Cloudflare opened in private beta on July 1, 2025. It lets site owners charge AI crawlers a per-request fee using the HTTP 402 "Payment Required" status code (Cloudflare, "Introducing pay per crawl"). That program is now expanding into "Pay Per Use," which pays publishers when their content actually shows up in an AI answer, not just when a crawler fetches the page. Cloudflare's first two named partners on Pay Per Use are Ceramic.ai and You.com (TechCrunch, July 1, 2026). Why is Cloudflare doing this now? Cloudflare is acting because bots now outnumber humans on the open web, and AI crawlers take far more than they give back. Cloudflare Radar data shows bots made up 57.5% of all HTTP requests to HTML content in early June 2026, against 42.5% from humans. Cloudflare hadn't expected that crossover until 2027 (Forbes, June 4, 2026). The imbalance is clearest in Cloudflare's own "crawl-to-refer ratio." It measures how many times a platform crawls a page for every visitor it actually sends back to that page, comparing January 2025 to July 2025: * Anthropic (Claude): 286,930:1 in January, down to 38,065:1 by July. Still extreme. * OpenAI (GPTBot): 1,217:1 in January, 1,091:1 by July. Roughly flat. * Perplexity: 54:1 in January, worsening to 194:1 by July. * Google: 3.8:1 in January, worsening to 5.4:1 by July. Cloudflare's own read on that data is blunt. Training now drives nearly 80% of AI bot activity, while referrals to publishers, especially from Google, are falling. As the company puts it: "creators face a paradox: feeding AI systems without gaining traffic in return" (Cloudflare, "The crawl-to-click gap"). Does this actually affect your small business site? Probably not through the September 15 default itself, but the underlying decision applies to you either way. The new default only changes behavior on pages that carry ads. A typical small business site without a programmatic ad network won't get auto-blocked by this specific change. But the setting Cloudflare is drawing attention to, which bots can read your site and on what terms, already applies to every website, ad-supported or not. If your site runs on Cloudflare, you already have an AI Crawl Control panel. It lists every AI crawler hitting your domain and lets you allow, block, or charge each one on its own. Most small business owners have never opened it. If you're not on Cloudflare, the same choice sits in your robots.txt file, where GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are either allowed, blocked, or simply never addressed. Should you block AI bots, allow them free, or charge through Pay Per Crawl? There's no single right answer. Each of the three options costs you something different. * Block everything. You stop the scraping and protect bandwidth. But you also disappear from AI answers entirely. If GPTBot or ClaudeBot can't read your site, they can't cite it when a customer asks ChatGPT or Claude who does what you do. Kindly Creative, LLC. has written about why that access matters for getting cited in AI search results. * Allow everything free. This is what most sites run today, and it's exactly the arrangement the crawl-to-refer numbers above describe: heavy crawling, little traffic back. You stay eligible for AI citations, but you're not compensated for what you feed those answers. * Charge through Pay Per Crawl or Pay Per Use. Set a price per request, or opt into the model that pays when your content is actually used in an answer. Adoption is still thin (Cloudflare currently lists two named partners), so don't expect meaningful revenue from Pay Per Crawl this quarter. Treat it as a switch worth flipping on, not a new income line. Most small businesses will land in the middle. Allow the AI crawlers that plausibly send customers your way, the ones tied to AI search visibility, and block or price the ones that only take. What does Gemini Spark have to do with any of this? It's proof that "AI bot traffic" no longer means only model training. Google shipped Gemini Spark on Mac in beta on July 1, 2026. It's an AI agent that books restaurant tables, orders groceries, and completes tasks across apps like Instacart, OpenTable, and Zillow Rentals on a person's behalf (TechCrunch, July 1, 2026). Every one of those actions is a bot request landing on somebody's website or booking system, not a human clicking through a menu. That's what flat "block all bots" advice misses. Some AI traffic hitting your site now is an agent trying to book your service or check your hours for a real customer, not a crawler training a model. Blocking indiscriminately can cut off a genuine booking attempt along with the training crawler you meant to stop. It's exactly why Cloudflare's new default separates search, agent, and training instead of treating "AI bot" as one bucket. How to check and set your AI bot policy this week. Five steps, and none of them require a developer. * Find out if you're on Cloudflare. Check your DNS provider or ask whoever manages your hosting. If your nameservers point to Cloudflare, you already have this control. * Open Bot Management or AI Crawl Control in the Cloudflare dashboard. Review what's set for Search, AI Training, and AI Agent categories on each domain. * If you're not on Cloudflare, open your robots.txt file. Check whether GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are allowed, blocked, or simply absent (absent usually means allowed). * Decide per category, not per bot. Keep search and citation-relevant crawling open if AI visibility matters to your business. Block or price pure training crawlers that offer no plausible referral value. * Revisit this quarterly. The rules, the named partners, and the pricing tools are all moving faster than a typical website review cycle. Faq. What is Cloudflare's Pay Per Crawl? Pay Per Crawl is a marketplace Cloudflare launched in private beta on July 1, 2025. It lets website owners charge AI crawlers a per-request fee using the HTTP 402 "Payment Required" status code, instead of allowing free scraping or blocking access outright. What changes on September 15, 2026? Cloudflare's default settings will block AI training and AI agent crawlers on any page that displays ads, for new customers, new sites, and free-tier accounts. Search crawling stays allowed by default. Existing paid accounts keep their current settings unless changed. Does this affect my small business website if I don't run ads? The September default specifically targets ad-supported pages, so a typical small business site may not be auto-blocked. But the crawler-access decision behind it, which AI bots can read your site and on what terms, already applies to every website through robots.txt or a host's bot-management settings. Should I block AI bots from my website? It depends on whether AI visibility matters to your business. Blocking protects bandwidth and proprietary content but removes you from AI-generated answers on ChatGPT, Claude, Gemini, and Perplexity. Most businesses do better allowing crawlers tied to search and citation while blocking or pricing pure training crawlers. What is a crawl-to-refer ratio? It's the number of times an AI platform crawls a page compared with how often it sends a visitor back to that page. Cloudflare's own data shows this ratio topped 280,000 crawls per referral for one platform in early 2025, the imbalance now driving its new policy. Sources. Is the bot reading your site right now training a model, or booking a customer? Most site owners have no idea, and until this month, most had no way to tell the two apart.