Rules for AI Crawlers

[peq42]

Motivation

AI Crawlers are a massive issue on the web nowadays. Bad acting ones, which disrespect robots.txt, normally act basically as a DDoS on the application/website.

Proposed solution

I'd like to suggest adding new rules, specifically targeted at detecting malicous crawlers(through more than just user agent, as that's easy to lie about) in order to better protect against such cases.

Alternatives

I've considered trying tools such as https://github.com/fabriziosalmi/patterns that add on top of OWASP but I think it'd be best if this project had it

Additional context

AI Crawlers have been a huge problem, specially for smaller websites. Most of those rely on OWASP for defense and things like cloudflare.

Cloudflare however is extremely bad/basic, specially on the free tier, so it'd be great if OWASP did some level of bot/ai crawler protection

[airween]

Hi @peq42,

thanks for bringing up this idea - I'm sure that's a pretty good feature.

I took a quick look and it seems that there is no any plugin for CRS, but your suggestion is a huge help. We definitely need to take a review that repository.

Thanks, we will get back soon.

[peq42]

Hi @peq42,

thanks for bringing up this idea - I'm sure that's a pretty good feature.

I took a quick look and it seems that there is no any plugin for CRS, but your suggestion is a huge help. We definitely need to take a review that repository.

Thanks, we will get back soon.

In my opinion, a way to detect and block excessive bot requests is important enough nowadays that it should be a main feature and not a plugin.

It's worth remembering that half the internet traffic is bots, and that was last year, it must be even worse nowadays.

[peq42]

Maybe making https://github.com/coreruleset/dos-protection-plugin-modsecurity a main feature could be a good start?

[etiennemunnich]

Some quickly searched known sources that can be quickly added to rules.

https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
https://developers.openai.com/api/docs/bots/
https://docs.perplexity.ai/docs/resources/perplexity-crawlers
https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers
https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
https://ja4db.com

This is an ever evolving area - these should be a good set of resources to start, each have a different approach; IPs, user-agents, ASN matching etc. This gets more complicated as you dig into it. This is why companies that specialise in data gathering and validation exist - it's an on going effort and requires some rigor.

I agree, a plugin is likely in order; to allow for data feeds from vendors you'd want to use, all the way to open sourced data/lists. robots.txt is way to basic, but can help too (when bots obey the rules that is.

If you're a fan of a simple availability control, then rate limiting use is okay.

[airween]

It's worth remembering that half the internet traffic is bots, and that was last year, it must be even worse nowadays.

I think probably you're right, but CRS tries to be generic. Everything else, which is a specific protection should go into a plugin.

Maybe making https://github.com/coreruleset/dos-protection-plugin-modsecurity a main feature could be a good start?

This plugin is a part of CRS 3.3, and it has been removed from 4.0, moved into a plugin, as you can see now.

I'm not sure we want to put it back :).

I know the border is very thin around these topics, I mean what the generic means exactly. It's not my decision, it's the team's decision.

The only thing is what I can do now that I put this issue in the list of next monthly chat's topic, and we will discuss this idea. You should join to the chat here, and choose #coreruleset channel.

[EsadCetiner]

Someone already proposed a PR a while back to block AI crawlers, it might be a good place to start if somebody wants to work on it coreruleset/coreruleset#3852. We also had other rules for blocking scripting bots and similar, but it was removed in the v4 release. https://github.com/coreruleset/coreruleset/blob/v3.3.8/rules/REQUEST-913-SCANNER-DETECTION.conf

I agree that a rule like this doesn't belong in CRS since AI crawlers may be desirable in some cases, it's not like blocking tools like sqlmap where in almost all cases there's a clear malicious intent. This would be a great candidate for a plugin that focuses on blocking bot traffic an admin decides is undesirable for their site.

[peq42]

Someone already proposed a PR a while back to block AI crawlers, it might be a good place to start if somebody wants to work on it #3852. We also had other rules for blocking scripting bots and similar, but it was removed in the v4 release. https://github.com/coreruleset/coreruleset/blob/v3.3.8/rules/REQUEST-913-SCANNER-DETECTION.conf

I agree that a rule like this doesn't belong in CRS since AI crawlers may be desirable in some cases, it's not like blocking tools like sqlmap where in almost all cases there's a clear malicious intent. This would be a great candidate for a plugin that focuses on blocking bot traffic an admin decides is undesirable for their site.

I disagree.

If an AI Crawler massively hits the server and ignores robots.txt, then it deserves to be blocked no matter the source/case. Thats why I think there should be a core rule for blocking them.

Obviously: It shouldn't be hyper aggressive. It should be broad, with as little false positives as possible. Something like the DoS plugin would be a good start, but I think a bit more could be added. Also, using user agents shouldn't be even considered, as any bot(or person) can change their user agent easily.

[EsadCetiner]

Obviously: It shouldn't be hyper aggressive. It should be broad, with as little false positives as possible. Something like the DoS plugin would be a good start, but I think a bit more could be added. Also, using user agents shouldn't be even considered, as any bot(or person) can change their user agent easily.

ModSecurity is not well-suited to handle a DoS attack as mentioned within the plugin's description. Using the rate limiting modules for your HTTP server such as mod_evasive for Apache and ngx_http_limit_req_module for NGINX would be better suited for this use-case.

ModSecurity and CRS are suited to protect against small-scale, application level denial-of-service (DoS) attacks. DoS attacks at the network level, distributed DoS (DDoS) attacks, and larger DoS attacks are hard to fight with ModSecurity. The reason for this is that the means of ModSecurity degrade under load and hence its ability to effectively handle DoS attacks degrades as well.

[fzipi]

@peq42 There is no one size fits all here. We moved away the DoS feature because it was unreliable for ModSecurity.

We would love to have other options here, but for now you can work to enhance the plugin.

What I really want is for webservers to expose ja3/ja4 fingerprints to WAF modules like Coraza/ModSecurity so we can just use that to block.

[HackingRepo]

you can also detect at PL3, browsers send some headers like Sec-Fetch-Dest and Sec-Fetch-Mode and Set-Fetch-Site and TE: trailers, but bots rarely do that, maybe that can be also reelayed on