Open the access logs of a large retail or media site and the user-agent column now reads like a list of AI products: ChatGPT-User, Meta-ExternalAgent, PerplexityBot. DataDome counted nearly 25 billion AI agent requests on its network in the third quarter of 2026, up 41.5% from 17.7 billion in the second. That brings the 2026 total to nearly 55 billion requests, and nearly half of it arrived in those three months.
Most bot defenses in production today were designed before that shift. They were tuned for scrapers, credential stuffing scripts and DDoS botnets, traffic that gave itself away through sheer volume or a crude browser. Security teams are now applying those same rules to software that reads pages and fills in forms the way a person would.
What Bot Management Was Built To Stop
Bot management is the practice of detecting automated traffic and deciding, request by request, whether to allow it, challenge it or block it. Allowing certain bots in has always been part of the job. Search crawlers and uptime monitors need access, while scalpers and account takeover scripts do not.
The classic toolkit grew around that split. Signatures match known attack tools and malformed requests. IP reputation lists flag addresses tied to data centers, open proxies or past abuse. Rate limits cap how many requests one address can send in a given window, and CAPTCHAs ask a suspicious visitor to prove it is human. User-agent rules let named crawlers through and hold unknown ones to closer scrutiny. DataDome’s guide What is bot management? sets out the full definition, including when a bot manager lets a bot through, when it shows a CAPTCHA and when it blocks the request outright.
Each of those tools makes four assumptions about the bot it faces: that it operates from a small pool of IP addresses, sends requests faster than a human could, runs a browser that leaves automation traces, and identifies itself honestly.
Five Places The Old Model Breaks
Allowlists That Trust A Name
An allowlist is the set of agents a site has chosen to let through, usually matched on the user-agent string each request carries. That string is self-declared. Trusted names such as ChatGPT or Perplexity get lighter scrutiny, which makes them the easiest names to borrow. DataDome reports that about 80% of AI agents do not properly identify themselves. When the company sent a spoofed ChatGPT-style user agent to 698,214 reachable sites, 79.7% let the request through without a block or a challenge. Agent spoofing rose 45% between February and July 2026, and Meta-ExternalAgent was the most impersonated agent, with 12.7 million spoofed requests in July.
Google’s documentation describes a way to verify Googlebot that does not rely on the user-agent string. Site owners run a reverse DNS lookup on the requesting IP address, then a forward lookup to confirm the hostname resolves back to the same address.
One Label For Very Different Agents
Sorting traffic into good bots and bad bots worked when a crawler either helped a site or attacked it. Two AI companies show why a single “AI bots” category no longer works. In the third quarter of 2026, Meta’s two agents sent 12.4 billion requests, 49.7% of the AI agent traffic DataDome observed, against about 51,000 referrals from Meta AI. That is roughly one visitor sent back for every 243,000 requests. ChatGPT sits at the other end of the scale: it drove 83.4% of the 89.4 million AI-referred visits DataDome measured in the first half of 2026.
Rate Limits, CAPTCHAs And Browser Checks
Volume thresholds assume an attacker concentrates its traffic. In April 2026, a DDoS campaign against a content platform sent 2.45 billion requests in five hours without triggering a single rate limit. It spread the load across 1.2 million IP addresses, so each address sent about one request every nine seconds, a pace closer to a person browsing than to a botnet.
Browser signals are becoming less reliable too. DataDome’s testing found that since Playwright 1.53, an AI agent driving the browser through Playwright MCP reports navigator.webdriver as false, the value a human visitor’s browser returns. A CAPTCHA usually appears only after another signal has raised suspicion, so an agent whose signals all look human never sees one.
Sensitive Pages
AI agents sent 605.6 million requests to login, form, cart, payment and account creation pages in the first half of 2026. Login pages took 51.7% of that traffic, up from 23% in 2025. Account takeover and fake signups start on exactly these pages, so an unverified agent on a login form carries more risk than one reading a blog post.
Coverage On Real Websites
DataDome tested 10 bot and spoofed AI agent types against 21,491 popular websites. It found that 65.3% of the sites stopped none of them and only 2.4% stopped all 10.
| Failure point | Figure |
|---|---|
| User-agent allowlists | 79.7% of about 700,000 sites let a spoofed ChatGPT-style user agent through |
| Good vs bad bot binary | 12.4 billion Meta requests vs about 51,000 referrals (1 per 243,000) |
| Rate limits | 2.45 billion requests, 1.2 million IPs, no rate limit triggered |
| Sensitive pages | 605.6 million AI agent requests to login, cart, payment and forms; login 51.7% |
| Protection coverage | 2.4% of 21,491 sites stopped all 10 bot types; 65.3% stopped none |
Identity Comes Before The Allow Or Block Decision
The replacement model checks who actually sent a request, independently of the name it gives, then reads what the request is trying to do from its behavior: which pages the agent touches, in what sequence, and whether it heads for forms and logins. A policy applies only after both checks.
Signed requests are the most promising way to settle identity. They rest on RFC 9421, HTTP Message Signatures, the IETF standard that lets a sender sign parts of an HTTP request with a key the receiving site can verify. Web Bot Auth, the set of proposals that applies those signatures to bots and agents, is still at the Internet-Draft stage at the IETF. Until it matures, network checks such as Google’s DNS method remain the practical way to verify the crawlers that support them.
Blocking every AI agent cuts off referral traffic your site may want, and allowing every agent leaves room for the spoofing described above. The pressure on that choice keeps rising: bad bot traffic grew 124% between July 2025 and June 2026, while human traffic grew 13.2%, according to DataDome’s annual report. DataDome’s research recommends an access policy set agent by agent, stating which agents are permitted, what they can access and under what conditions.
What A Per-Agent Policy Looks Like
In practice, that policy can be as simple as a short table kept current alongside the WAF rules. Four agents show how far apart the answers can be.
| Agent | What it does | Value to the site | Policy question |
|---|---|---|---|
| Meta-ExternalAgent | Collects content to train Meta’s models | None measured (12.4B Meta requests vs about 51K referrals in Q3) | Limit or block? |
| Meta-WebIndexer | Indexes pages for Meta AI answers | Low so far | Rate-limit? |
| ChatGPT-User | Fetches pages live for user questions | High: ChatGPT drives 83.4% of AI referrals | Allow, and verify identity |
| Muse | Browses from its own cloud VM; 8.9% of requests on sensitive flows | Unclear | Decide deliberately, watch forms and login |
Muse is the hardest of the four to classify. It is Meta’s personal AI agent, and it runs inside its own dedicated cloud virtual machine, so its traffic looks more like a remote user than a crawler. In a sample of Muse traffic DataDome tracked from September 22 to October 3, 2026, 8.9% of requests touched forms, login, payment, account creation or cart endpoints. For a team writing its policy now, the open question is how much of that checkout and login traffic it can verify as acting for a real customer, and what it does with the rest.


