Only follow robots.txt redirects on the same host

[rzo1]

HttpRobotRulesParser resolved the Location header of a robots.txt redirect and re-fetched it without checking scheme, host or port, so the fetch could end up on any host the redirect named.

A redirect which only replaces http with https on the same host and port is still followed, whatever this is configured to, so the usual http://…/robots.txt -> https://…/robots.txt is unaffected. Anything else is now followed only when the target is http or https and shares scheme, host and port with the URL it was reached from; http.robots.redirect.crossorigin.allow (default false) restores the old behaviour for http and https targets.

getCacheKey derives the default port from the scheme, so an upgraded target has a different key although no other host is involved, and refusing it would leave that very common redirect with no rules at all. Only that direction is exempt — https to http would put the configured request headers on the wire in clear and let rules read over plain text replace the ones the redirect was reached from.

A redirect which is not followed no longer ends as allow-all in the success cache. It yields no rules for the host, cached in the error cache like a 429 or a 5xx, so nothing is crawled there until the redirect is resolved; http.robots.redirect.refused.allow: true restores allow-all. The refusal is logged at warn level with the target, the reason and the setting that changes it.

Following a redirect to another origin re-fetches robots.txt with the headers configured through http.basicauth.* and http.custom.headers, as any other request does. That is now stated where the setting is defined and logged once at configuration time when either is set. http.filter.ipaddress.exclude gained a linklocal rule to restrict which addresses those fetches may reach — link-local is matched by neither loopback nor sitelocal, so the documented "localhost,sitelocal" example did not cover it.

Sites serving robots.txt through a redirect to another host, a CDN for instance, are affected. Documented in crawler-default.yaml and configuration.adoc.

For all changes

• Is there a issue associated with this PR? Is it referenced in the commit message? - no issue

• Does your PR title start with #XXXX where XXXX is the issue number you are trying to resolve? - no issue

• Has your PR been rebased against the latest commit within the target branch (typically main)?

• Is your initial contribution a single, squashed commit?

• Is the code properly formatted with mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false?

For code changes

• Have you ensured that the full suite of tests is executed via mvn clean verify? - core module tests only
• Have you written or updated unit tests to verify your changes? - new HttpRobotRulesParserRedirectTargetTest; the existing redirect test sets the new key, since its chain crosses ports
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0? - no new dependencies
• If applicable, have you updated the LICENSE file, including the main LICENSE file? - n/a
• If applicable, have you updated the NOTICE file, including the main NOTICE file? - n/a

[dpol1]

Not convinced by the default. As it stands http://h/robots.txt -> https://h/robots.txt (same host, only the scheme changes) gets refused, and from there we crawl the site with no rules at all. Tried it on a scratch test. Threads.

[dpol1]

LGTM. The one-way scheme-upgrade exemption and the refused redirect going to the error cache instead of allow-all cover both points, and refused.allow keeps the old behaviour one line away for whoever needs it. Only the description: it still opens with "shares scheme, host and port", the http -> https exemption deserves the first sentence since that's what most people will hit.