Will Cloudflare’s “Pay per crawl” keep media companies profitable?

Some of the strongest sentiments following the growth of generative AI has been that traditional media companies are going to get killed.

The reason?

Generally, people believe that since AI makes content creation easy, and access to information “cheaper” that these media services would no longer be needed as:

—more individuals will create content and publish on social platforms, creating a shift from Tradmedia to individual-media.

—and less people will visit traditional media websites.

The reality however, is that AI cannot kill the media companies even if it replaces them because AI doesn't manufacture its data, it gets them from somewhere.

For it to be able to report on current events, it needs to have credible sources to retrieve data from.

Secondly, less people visiting traditional media websites is something that's mostly going to be temporary because people will increasingly need to fact-check information from AI applications.

Whether or not people will grow to trust without verifying is something we can only wait to see but it's to be expected that this cannot happen without a system that makes users trust AI data without digging deep.

The most basic example of such systems is a fact-checked icon besides AI outputs, letting users know that all data is verified. That still, has to have gone through a process where the fact-checking engine has been rigorously studied by multiple companies and individuals to determine its validity.

We can expect some integration with blockchain at some point, but does all of this mean that traditional media will get killed?

The truth is that the only traditional media companies that will get killed are those that cannot adapt to AI advancements. AI doesn't only empower individual creators, it also helps companies cut costs in numerous ways.

But that's besides the focus since what we are discussing is mostly how the increasing use of generative AI such as ChatGPT will affect the attention-dependent business model of media platforms.

The shift from human attentions to AI-attention

So let's say that traditional media websites lose anywhere between 60-75% of their traffic due to the rise of AI. How do they stay in business?

The answer is quite simple. If humans no longer surf the web, then monetization shifts to whatever does, and in this case, that's AI agents.

Did we ever imagine a time where bots will be actively tracked as valuable web visitors and charged for it?

That's what comes with the rise of AI because although users may not directly visit or interact with web pages again, these AI bots are doing it for them, so they are essentially a proxy of the attention, hence, will need to have their visits monetized!

When Coinbase launched stablecoin payments over HTTP, I briefly discussed this, and now, Cloudflare has launched a “Pay per crawl” solution that will enable website owners to charge AI agents that attempt to access their web content.

Pay per crawl integrates with existing web infrastructure, leveraging HTTP status codes and established authentication mechanisms to create a framework for paid content access.

Each time an AI crawler requests content, they either present payment intent via request headers for successful access (HTTP response code 200), or receive a 402 Payment Required response with pricing. Cloudflare acts as the Merchant of Record for pay per crawl and also provides the underlying technical infrastructure.

An important mechanism here is that even if a crawler doesn’t have a billing relationship with Cloudflare, and thus couldn’t be charged for access, a publisher can still choose to ‘charge’ them. This is the functional equivalent of a network level block (an HTTP 403 Forbidden response where no content is returned) — but with the added benefit of telling the crawler there could be a relationship in the future. — Cloudflare

So I guess traditional media isn't dead afterall?

The reality is that AI agents were never going to get away with just gaining access to free data. They were always going to either pay for it or get blocked and every major internet infrastructure company is going to move to enable website owners to charge these agents.

One could argue that infrastructure companies are the shovel sellers in the age of AI as they offer the middleware for information transfer between sources and AI systems which then leverages it in the broader economic system.

There remains a possibility that even advertisements can become AI-targeting, completely changing how companies looking to promote their products and services approach bot traffic, insane right?

This is the future.

Posted Using INLEO