refactor(sessions): move blocked_status_codes from Session to crawler - #2160
Draft
aryansk wants to merge 1 commit into
Draft
refactor(sessions): move blocked_status_codes from Session to crawler#2160aryansk wants to merge 1 commit into
aryansk wants to merge 1 commit into
Conversation
Deciding whether a response status means "blocked" is the crawler's concern, not the session's - the crawler already owns additional_http_error_status_codes and ignore_http_error_status_codes. Move blocked status code handling out of Session/SessionPool and onto BasicCrawler, matching the JS implementation (apify/crawlee#3496). - BasicCrawler gains a `blocked_status_codes` option (default [401, 403, 429]); the blocked check in _raise_for_session_blocked_status_code now consults the crawler's set minus ignore_http_error_status_codes instead of Session - Session drops the blocked_status_codes parameter, the is_blocked_status_code method, and the model field; get_state no longer serializes it - docs: session examples configure blocked_status_codes on the crawler - tests: session/model tests updated; new crawler tests cover the default set, custom values, and ignore-code interaction Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2139
What
blocked_status_codeswas configured perSessioneven though deciding whether a response means "blocked" is the crawler's concern — the crawler already ownsadditional_http_error_status_codesandignore_http_error_status_codes. This moves it to a crawler option and keeps the handling internal to the crawler, matching the JS implementation (apify/crawlee#3496).Changes
BasicCrawlergains ablocked_status_codesoption, defaulting to[401, 403, 429]. The blocked-status check in_raise_for_session_blocked_status_codenow consultsself._blocked_status_codes - self._ignore_http_error_status_codesinstead of asking theSession.Sessiondrops theblocked_status_codesparameter, theis_blocked_status_code()method, and the internal set;get_state()no longer serializes the field.SessionModeldrops theblockedStatusCodesfield.blocked_status_codeson the crawler instead of insideSession/create_session_settings.ignore_http_error_status_codesinteraction.Validation
ruff checkandruff format --checkcleanty checkclean on the touched modulestests/unit/sessions/(29) andtests/unit/crawlers/_basic/+_http/(196, 1 pre-existing skip)