Data Analytics Intern

Deadline set: Cloudflare to give website owners more power over AI crawlers. Image: Generated with ChatGPT Jul 6, 2026 eWeek content and product recommendations are editorially independent. Legal Tech LLC may make money when you click on links to its partners. Learn More The free ride AI bots have enjoyed across the web just got bumpier. Cloudflare is tightening the rules around how AI companies collect content from websites, introducing new controls that could reshape the relationship among publishers, search engines, and AI companies. The company announced that beginning Sept. 15, 2026, new default settings on its platform will block AI Training and Agent crawlers from pages that display ads, while traditional Search crawlers will remain allowed. The move also affects mixed-purpose bots, or crawlers that combine search indexing with AI training or agent functions. Cloudflare said those mixed-use systems will be judged by all of their behaviors, not just one. In practice, that means a crawler used for both search and AI training could be blocked even if search access is still permitted. The changes apply to new customers, new sites created by existing customers, and existing free-tier users who have not adjusted their settings. A shift away from the old web bargain. Cloudflare argues that the internet's long-standing exchange - websites allow crawling and receive visitors in return - no longer works as cleanly in the AI era. Traditional search engines historically indexed websites and redirected users back to publishers. AI systems increasingly absorb information, summarize it, and deliver answers directly to users, reducing the need to visit sources. The company described this challenge as a difficult trade-off for content owners who want online visibility without freely handing over their work to AI systems. "Content owners still want to be able to protect their content, and they should be compensated for the original content that they work hard to create, curate, and share. We also know that locking down content isn't a one-size-fits-all solution; website owners want more options than resorting to 'block all automation, every time,'" wrote Cloudflare Product Manager Jin-He Lee and Director of Product Bryan Becker. Cloudflare also renewed its criticism of large search platforms, suggesting that major players gain an advantage when the same crawling systems handle both search indexing and AI-related tasks. New tools beyond simple blocking. Instead of treating all AI traffic the same, Cloudflare is splitting automation into three categories: Search, Agent, and Training. Website owners can now create separate rules for each category, giving them more control over how content is accessed. Cloudflare also introduced BotBase, a searchable database for Enterprise customers that identifies bot behavior and classification. The company is also expanding its content controls to include usage categories such as "Immediate," "Reference," and "Full," allowing publishers to signal how their material should be stored or reused after being crawled. Why this matters. The announcement reflects a larger fight over who benefits from AI-generated answers. Publishers depend on traffic, advertising, subscriptions, and referrals. AI companies depend on access to large amounts of content. As AI systems increasingly answer questions directly, fewer users may visit the original websites that produced the information. Cloudflare is also evolving its previous Pay Per Crawl system into "Pay Per Use," an approach designed to compensate publishers when their content contributes value to AI-generated responses. Pressure shifting toward AI companies. The biggest impact of Cloudflare's move may not be blocking bots; it may be forcing transparency. Companies running one crawler for search, AI agents, and model training now face pressure to separate those functions if they want uninterrupted access. That could make it easier for publishers to understand who is collecting content and why. The challenge is that Cloudflare cannot control the entire internet. Large technology firms already have their own opt-out mechanisms, and competitors could pursue alternate methods of gathering data. Still, because Cloudflare sits in front of a large share of internet traffic, its policy changes could influence how AI companies structure their systems going forward. Also read: Portugal launched Amália, an open-source AI model built for European Portuguese as Europe pushes for more control over AI infrastructure. Aminu Abdullahi is a B2C and B2B technology and finance writer with more than six years of experience covering enterprise IT, cybersecurity, cloud computing, artificial intelligence, fintech, business software, and emerging technologies. His work has appeared in publications including TechRepublic, eWEEK, Channel Insider, Geekflare, Enterprise Networking Planet, eSecurity Planet, CIO Insight, and Webopedia. With a technical background in computer science, he specializes in translating complex technology topics into clear, accessible content for business leaders and decision-makers.

Cloudflare unveils new AI content control and monetisation framework to power the Agentic Internet. July 3, 2026 2:14 pm By IBG NEWS Bureau India, July 3, 2026: Global connectivity cloud company Cloudflare, Inc. (NYSE: NET) has announced a comprehensive suite of new AI-focused tools, analytics, and commercial partnerships aimed at empowering website owners to control how artificial intelligence systems access, use, and monetise online content. The initiative is designed to create a transparent and sustainable ecosystem for the rapidly evolving Agentic Internet, where AI-powered agents increasingly serve as the primary interface for information discovery and online commerce. Cloudflare said that more than half of all web traffic is now generated by automated bots and AI agents, fundamentally changing how digital content is discovered and consumed. To address growing concerns among publishers and content creators, the company has introduced a philosophy centred on "Your Content, Your Rules," giving website owners greater authority over whether AI systems can crawl, train on, or utilise their content. New default AI crawling policy. Cloudflare announced that beginning September 15, 2026, new default settings for websites with advertising will permit AI search indexing while blocking AI training and autonomous agent usage unless website owners choose otherwise. Mixed-purpose crawlers that do not clearly distinguish between search, AI training and agent activities will be restricted on ad-supported pages by default. Existing free customers who do not modify their settings before the deadline will also receive the updated default protections. Enhanced AI analytics for publishers. To help businesses better understand how AI platforms interact with their content, Cloudflare has introduced the Attribution Business Insights dashboard. The platform enables publishers to monitor AI bot activity, measure referral traffic generated by AI systems, and evaluate how frequently their content is cited in AI-generated responses. The company also highlighted the emergence of Answer Engine Optimization (AEO) - a new discipline focused on improving content visibility in AI-generated answers, similar to how Search Engine Optimization (SEO) transformed digital publishing during the traditional search era. AI search optimisation and creator compensation. Cloudflare is also working to improve AI search efficiency by reducing unnecessary crawling of unchanged webpages. According to the company, more than 50% of AI crawler traffic is spent repeatedly fetching content that has not changed, resulting in wasted computing resources and bandwidth. Building on its Pay Per Crawl initiative launched last year, Cloudflare has introduced Pay Per Use, a model under which publishers are compensated when their content actively contributes to AI-generated answers rather than merely being crawled. The company is collaborating with AI firms including Ceramic.ai and You.com to develop scalable commercial frameworks that fairly compensate content creators while enabling AI innovation. Industry partnerships. Cloudflare's latest initiative has received support from several leading organisations, including beehiiv, Ceramic.ai, Condé Nast, and Patreon, all of which emphasised the need for greater transparency, creator control, and equitable compensation in the AI ecosystem. CEO's perspective. Announcing the initiative, Matthew Prince, Co-founder and CEO of Cloudflare, said the internet has reached a stage where non-human traffic now dominates online activity, making it essential to establish a balanced framework that protects intellectual property while enabling AI innovation. He noted that the company's new tools provide website owners with enhanced visibility, commercial opportunities, and greater control over AI access, while encouraging AI companies to adopt transparent and clearly identifiable crawling practices. Towards a sustainable AI economy. Cloudflare stated that its latest offerings are built around three guiding principles - control, transparency, and monetisation - to ensure that publishers, creators, businesses, and AI developers can thrive together in the emerging agentic economy. With AI increasingly reshaping the future of search, commerce, and digital interactions, Cloudflare's initiative seeks to establish a collaborative framework where content creators retain ownership, AI companies operate transparently, and innovation continues to flourish without compromising intellectual property rights.

OpenAI Seeks Ad Format Innovation; Monetizing The Post-Click Web. Choice Analyst Choice Technical Labs OpenAI's hiring priorities hint at the nascent ad platform's future direction. Plus: Cloudflare unveils tools to categorize crawlers. The post OpenAI Seeks Ad Format Innovation; Monetizing The Post-Click Web appeared first on AdExchanger. Executive Technical summary. In a groundbreaking move, OpenAI is spearheading the innovation of ad formats aimed at monetizing the Post-Click Web, as reported by AdExchanger. This initiative signifies a pivotal shift in the digital advertising landscape, particularly affecting YouTube creators, MCNs, and content agencies. By focusing on privacy, trust, and policy compliance, OpenAI is poised to redefine the dynamics of user interaction and revenue generation. This shift necessitates a recalibration of current strategies for content monetization and rights management, presenting both opportunities and challenges for stakeholders in the digital content ecosystem. Structural deep-dive. Impact on creator workflows. The introduction of new ad formats by OpenAI necessitates a reevaluation of existing content workflows for YouTube creators. The adaptation to these formats requires a seamless integration with Content ID systems and rights management frameworks to ensure compliance and maximize ad placement efficacy. * Content Integration: Creators must align their content with new ad specifications, which may involve modifying video metadata and descriptions to enhance ad relevance and placement. * Workflow Automation: Automation tools within CMS need enhancements to accommodate dynamic ad formats and ensure real-time compliance with OpenAI's privacy and safety standards. The evolving ad landscape mandates an overhaul of rights management procedures within Content Management Systems (CMS). This includes: * Policy Compliance: Updating CMS frameworks to reflect OpenAI's privacy and safety guidelines, ensuring that all uploaded content adheres to the new standards. * Dynamic Licensing: Implementing flexible licensing models that account for the dynamic nature of ad interactions, particularly in LLM-driven environments. Revenue & strategic implications. Creator payouts. The shift towards innovative ad formats introduces new variables in revenue calculation for creators: * Ad Performance Metrics: Redefining performance metrics to include post-click engagement metrics, which directly influence revenue streams. * Revenue Distribution: Adjusting revenue models to account for variable ad engagements, impacting the payout structure for creators and agencies.