Cloudflare and GoDaddy are joining forces to give website owners back control of their content in the age of AI agents. But the stakes go far beyond bots and crawlers. This is a fight over who owns the future of the web.

by Kasun Illankoon, Editor in Chief at Tech Revolt

[For more news, click here]

Picture this: you run a small bakery in Columbus, Ohio. You have a website. Nothing fancy, just your menu, your story, a few photos of your sourdough. For years, Google sent occasional crawlers to index your pages, and in return, curious locals found you through search. It was a simple, if imperfect, bargain.

Now imagine hundreds of AI systems scraping your site every week, training on your recipes, reproducing your about-page word for word in generated summaries, and sending you nothing in return. No traffic. No credit. No revenue. Just silence while your content quietly fuels someone else's product.

This is not a hypothetical. It is happening right now, at scale, across the entire web. And two of the internet's most powerful infrastructure companies say they have a plan to stop it.

Cloudflare, the connectivity and cybersecurity giant that handles traffic for a significant portion of the global web, and GoDaddy, the domain and small business platform used by millions of website owners, announced a major strategic partnership this week aimed at bringing identity, trust, and access controls to what they are calling the "agentic open web." The collaboration targets one of the most underreported but consequential shifts in modern technology: the explosion of AI agents operating autonomously across the internet, often with no accountability, no transparency, and no permission.

What Is an AI Agent, and Why Should You Care?

To understand why this matters, it helps to understand what an AI agent actually is, and why it is fundamentally different from the search bots that have crawled the web for decades.

A traditional search crawler, like Googlebot, is relatively simple. It visits a page, reads it, indexes it, and moves on. Its identity is publicly documented. Website owners can block it using a standard file called robots.txt. The relationship is understood by both sides.

AI agents are a different animal entirely. These are autonomous software systems that do not just read content, they act on it. An AI agent might browse a product page, compare prices across dozens of sites, make a purchase decision, fill out a form, and book a reservation, all without a human touching a keyboard. As large language models become more capable and more embedded in everyday software, these agents are multiplying rapidly.

The problem is that most of them are invisible. Website owners often cannot tell whether traffic is coming from a human, a known crawler, or an unknown AI agent with unclear intentions and unknown operators. There is no standard way to verify who is running the agent, what it is doing with the data it collects, or whether it has any right to be there at all.

"The Internet is evolving into a high-velocity, AI-driven ecosystem, and that requires a new kind of transparent infrastructure," said Stephanie Cohen, Chief Strategy Officer at Cloudflare. "By putting tools like AI Crawl Control and open standards into the hands of website owners, we are providing essential underpinnings for a new Internet business model. We want to ensure that every creator has the tools to verify who is interacting with their site, while giving legitimate AI agents a secure, transparent way to participate in a thriving open web."

The Two-Pronged Plan

The Cloudflare-GoDaddy partnership tackles this problem from two directions simultaneously, which is what makes it more substantive than most industry announcements.

The first prong is about visibility and control. GoDaddy is integrating Cloudflare's AI Crawl Control tool directly into its website hosting platform. This means that millions of GoDaddy website owners, many of them small businesses and independent creators with little to no technical expertise, will be able to see which AI crawlers are visiting their sites and choose what to do about them. They can allow access, block it entirely, or signal that access requires payment. This is not a feature buried in a developer dashboard. It is designed to be accessible to the bakery owner in Columbus who just wants to know who is eating her digital bread.

The second prong is more ambitious, and in some ways more important. It addresses the question of AI agent identity. Cloudflare and GoDaddy are throwing their weight behind emerging open standards designed to make AI agents verifiable and accountable by default.

GoDaddy has introduced something called the Agent Name Service, or ANS, a global open standard that draws on the existing infrastructure of the domain name system and public key cryptography to give AI agents verifiable identities. Think of it as a passport system for bots. An agent operating under ANS can prove who operates it, what it is authorized to do, and whether its claimed identity is legitimate, rather than a spoofed impostor trying to look like a trusted system.

Cloudflare is supporting ANS and has also introduced its own complementary tools: Web Bot Auth, which uses cryptographic signatures to verify bot and agent traffic, and a Signature Agent Card, which allows agent developers to transparently publish their agent's identity and purpose.

"By working with Cloudflare on AI Crawl Control and championing the Agent Name Service, an open standard giving every agent a verifiable identity built on DNS, we are providing our customers the transparency they need to thrive in an AI-first world," said Jared Sine, Chief Strategy Officer at GoDaddy. "We move at the speed of the Internet, and we are working with the broader industry to ensure the agentic open web does too."

The Deeper Economic Crisis This Is Trying to Solve

None of this would matter quite so much if it were only about blocking bad bots. But the partnership is really a response to something much larger: the collapse of the web's foundational economic model.

For roughly three decades, the internet ran on a relatively stable feedback loop. Content creators and publishers produced content. Search engines indexed and surfaced it. Users clicked through. Traffic generated advertising revenue. Everyone, more or less, had a reason to participate.

Generative AI has disrupted this loop in a fundamental way. As AI-powered answer engines become more capable of summarizing and synthesizing information directly in the interface, users no longer need to click through to the original source. The content gets used. The traffic never arrives. The creator sees nothing.

This is already showing up in the data. Publishers and independent website owners across the web are reporting declining organic traffic even as AI tools trained on their content grow more popular. The value is being extracted from one end of the chain while the economics are collapsing at the other.

What Cloudflare and GoDaddy are proposing, at its core, is the beginning of a permission-based and potentially payment-based framework for how AI systems access human-generated content. The tools they are announcing today do not yet include a functioning payment system for AI content access. But by combining audit logs that track exactly which agents accessed which content with cryptographic identity standards that make those agents accountable, they are building the plumbing that such a system would require.

Who This Is Really For

It is worth being clear-eyed about who benefits most immediately from this partnership, because the beneficiaries are not evenly distributed.

Large media organizations and enterprise websites already have teams of engineers and legal departments that can negotiate with AI companies, implement technical protections, and pursue violations. They have leverage.

The people who have historically had the least protection are the ones who need it most: independent bloggers, small business owners, local news sites, niche content creators, and the vast middle of the web that produces the vast majority of the internet's actual human knowledge and experience. These are exactly the people GoDaddy primarily serves.

By embedding these controls directly into GoDaddy's hosting interface, the partnership is making a bet that protection does not have to require technical sophistication. A restaurant owner should not need to understand public key infrastructure to decide whether an AI company can train on her menu. She should be able to click a button.

Whether that vision translates cleanly into reality depends on execution. Standards are only as useful as their adoption, and adoption in the agentic ecosystem is still far from guaranteed. ANS and Web Bot Auth are open standards, which is the right approach, but the history of the web is littered with technically sound standards that were ignored by dominant players who had no incentive to comply.

The Road Ahead

The announcement is a beginning, not a resolution. The challenges ahead are genuinely hard.

Enforcement is the central problem. Cryptographic standards and audit logs can identify well-behaved agents operated by companies that want to be identified. They do less to stop bad actors who deliberately spoof identities or operate outside any governance framework. The same way that email spam persisted long after authentication standards like SPF and DKIM were introduced, some portion of malicious or extractive AI crawling will likely persist regardless of what standards are established.

There is also the question of market power. The companies most aggressively deploying AI agents at scale are also among the wealthiest and most powerful in the technology industry. Whether they choose to comply with a permission-based framework, or simply continue operating as they have been while the standards ecosystem develops around them, remains to be seen.

Still, the direction of this partnership reflects something real: the internet is undergoing a structural transformation, and the infrastructure that governs how information flows, who can access it, and on what terms is genuinely up for renegotiation right now. The decisions made in the next two or three years about agent identity, content access, and value exchange will shape what the web looks like for the next two or three decades.

Cloudflare and GoDaddy are not the only parties at the table. But they are two of the most consequential pieces of infrastructure the web runs on, and they are now pointing in the same direction. For millions of website owners who have watched their content get consumed by AI systems without consent or compensation, that is at least a start.