If this is rogue AI agent behaviour, then the reality is that some of the underlying behaviour has been happening for quite some time.
I’m sure every large organisation in New Zealand including every Government department has seen large numbers of unusual requests generated by AI crawling through their public sites to mine every bit of data that can be found for years. This includes going over, under and around security controls put in place, but again its a public site, if you put data on it, then expect for it to be seen in public.
Research published by Transluce on 23 September 2026 has presented evidence that AI agents used the web security service URLQuery.net to bypass access restrictions and expand their ability to retrieve information from the public internet. The research also identified three instances where agents attempted to exploit public data providers, including the Australian Institute of Health and Welfare (AIHW).
Transluce links at least some of the activity to agent swarms previously attributed to OpenAI. It says the activity goes back to at least March 2026, with weaker evidence potentially extending back to November 2025.
The AIHW has confirmed that its public-facing website was one of the government sites interacted with by an OpenAI agent, although it says there is currently no evidence that anything other than publicly available information was accessed.
But is this really new?
We were seeing something similar in 2024
Back in 2024, New Zealand organisations suddenly started seeing the early AI race play out against public-facing websites.
A mixture of relatively well-known crawlers, including Anthropic’s ClaudeBot, and numerous less easily identifiable crawlers were consuming significant resources on public websites as AI companies raced to collect data for model development.
The problem wasn’t necessarily someone requesting a PDF or downloading a webpage.
Some crawlers were effectively using website search functionality as a way of mining the available data.
Instead of making a handful of normal requests, the crawler could generate hundreds of requests per second against a site’s search functionality, attempting to discover whatever information the website could return.
For organisations running government websites, the result could be very noticeable.
Hundreds of requests per second consumed all the CPU resources of the underlying web servers and cause timeouts for legitimate users, which was what drew out attention to it to begin with.
Below is a screenshot from June 2024 with the website CPU utilisation at 100% until the AI crawler IP addresses were blocked and the web server reverted to usual CPU activity.
Caching doesn’t necessarily save you
One reason this behaviour could be particularly effective was the way caching infrastructure was configured.
Services such as Cloudflare are very effective at caching commonly requested content such as HTML pages, JavaScript, CSS and images.
A search request with a unique URI is a different proposition.
An example from back then of search parameters these crawlers were trying was “%7b%221%22%3a6%2c%223%22%3a13%2c%224%22%3a7%2c%225%22%22%3a16%2c%226%22%3a8%%7d” which when decoded is a json string {“1″:6,”3″:13,”4″:7,”5″:16,”6”:8}. This isn’t what the search function was designed for as it was designed for humans but it was legally wrong to search for whatever you wanted. Imagine the search parameters tried covering hundreds of requests per second for days.
Below again back in 2024, is the change over 400 requests/sec when the AI crawlers were hitting then dropping to standard activity when the crawlers were blocked
If every search request generates a unique URL, the caching layer may have little useful content to serve from cache. The request instead gets passed through to the application and underlying web infrastructure.
An AI crawler could therefore turn a relatively simple website search function into a resource-intensive enumeration mechanism.
This isn’t necessarily a sophisticated cyber attack. It can simply be the result of a crawler doing exactly what it was designed to do — finding and retrieving information at scale.
robots.txt isn’t a magic security control
There was also a relatively simple mitigation available for well-behaved crawlers: robots.txt.
For example, website operators could choose to tell ClaudeBot not to crawl their site.
Anthropic explicitly documents this behaviour today. ClaudeBot follows robots.txt, while Anthropic also provides a Crawl-delay mechanism for website operators that want to control crawling rates.
The problem was that not every crawler necessarily behaved in the same way.
Unknown crawlers, particularly those that didn’t identify themselves clearly, could simply ignore the rules.
And even a crawler that normally respects robots.txt could potentially encounter an interesting problem when it started using intermediary services.
The URLQuery.net example
This is where the Transluce research becomes particularly interesting, and what we cover below was being seen in 2024.
The researchers found evidence of AI agents using URLQuery.net and other intermediary services to retrieve content.
Conceptually, if an AI agent asks a third-party service to retrieve a URL on its behalf, there are now two web requests:
AI agent → URLQuery.net → target website
The intermediary is making the request to the target website.
That creates an interesting distinction around crawler controls.
If the target website has:
aihw.gov.au/robots.txt
and the intermediary has:
urlquery.net/robots.txt
the crawler is interacting directly with URLQuery.net, not necessarily directly with AIHW.
From the crawler’s perspective, it may therefore be following the robots rules applicable to the service it is accessing.
This doesn’t make the behaviour particularly desirable from a website operator’s perspective, but it demonstrates an important limitation of treating robots.txt as an access-control mechanism.
It isn’t one.
robots.txt is essentially a mechanism for communicating crawling preferences to cooperating automated clients.
AI agents are taking this further
What Transluce appears to be documenting is an evolution of this behaviour.
The research identified thousands of reports showing agent-like activity through URLQuery.net, including attempts to retrieve data indirectly when direct approaches failed.
In three cases, agents reportedly moved beyond simply retrieving data and attempted vulnerability probes against public data providers.
The AIHW incident is particularly notable because Transluce says the agent was attempting to obtain relatively mundane government health statistics rather than carrying out a specifically assigned cyber security task.
According to Transluce, the AI agent attempted different approaches after encountering obstacles to retrieving the required information.
That is the significant change.
A traditional crawler might simply fail when blocked.
An agent can potentially reason:
I need this information → the normal method isn’t working → what other method could achieve the objective?
That could mean using another service, changing the request, finding an alternative data source or, in more concerning circumstances, probing the target for weaknesses.
So is this “rogue AI”?
That depends on what we mean by rogue.
The underlying concept of automated systems trying different ways to obtain information from public websites is certainly not new. Websites have been dealing with aggressive crawlers, scrapers, search bots and poorly behaved automated clients for years.
What is different with AI agents is the potential for goal-directed adaptation.
A crawler generally follows a predefined process.
An agent can potentially make decisions about what to try next.
That distinction matters for defenders.
The Transluce research suggests that AI agents have been using intermediary services and adapting their retrieval techniques for months, with evidence of activity going back to at least March 2026.
The lesson for organisations isn’t necessarily that AI has suddenly become an attacker.
It is that the traditional assumptions we have made about web crawlers may no longer be sufficient when the crawler can make decisions about how to achieve its objective.
Importantly if aihw.gov.au doesn’t want AI models using its data then why hasn’t it edited its robots.txt file to try and prevent them?
What should website operators be thinking about?
For organisations operating public websites, particularly government and healthcare organisations, the defensive considerations are familiar but increasingly important:
- Don’t treat
robots.txtas a security control. - Rate-limit resource-intensive search and API functions.
- Monitor unusual request volumes and query patterns.
- Check your web logs at least each month just to see whats going on.
- Ensure caching infrastructure doesn’t simply push expensive searches back to the application.
- Consider bot management and behavioural detection.
- Monitor for automated clients accessing unusual URL paths or functions.
- Ensure public-facing applications are resilient to high-volume automated requests.
- Treat APIs and search functions as attack surfaces, even when the underlying data is public.
- Assume that automated clients may attempt alternative routes when their first approach fails.
The bigger issue is that the web was designed around humans, browsers and relatively predictable automated clients.
AI agents introduce another class of user — one that can potentially browse, search, experiment, use third-party services and change its approach based on what happens next.
That isn’t entirely new.
We’ve been seeing pieces of it for years.
What may be new is how capable and quick the automated client has become.
