Skip to content

Add --date parameter to load_bigquery_stats job#5369

Open
stormKx18 wants to merge 3 commits into
masterfrom
load_bigquery_stats_by_date_v1
Open

Add --date parameter to load_bigquery_stats job#5369
stormKx18 wants to merge 3 commits into
masterfrom
load_bigquery_stats_by_date_v1

Conversation

@stormKx18

@stormKx18 stormKx18 commented Jul 16, 2026

Copy link
Copy Markdown

This PR updates load_bigquery_stats.py to accept an optional --date flag (in YYYY-MM-DD format), allowing cluster operators and developers to execute the BigQuery stats loading job against an arbitrary target date (useful for backfilling or re-processing past stats).

Key Highlights

  • Optional --date Flag: Supports loading stats for a specified date (e.g. --date 2026-06-20).
  • Backward Compatible: Defaults to yesterday's date (_utc_now().date() - datetime.timedelta(days=1)), maintaining standard daily cron execution behavior.
  • Design Parity: Follows the existing command-line argument pattern established in aggregate_fuzzer_stats.py.
  • Unit Test Coverage: Added test_execute_with_date to verify that target partition tables ($YYYYMMDD) and GCS source URIs (/date/YYYYMMDD/) are correctly generated when a custom date is provided.
  • Unit Test Refactoring: Extracted BigQuery dataset, table, and load job assertions into a shared _assert_load_calls(expected_date_str) helper method to avoid code duplication across default and date-specific test cases.

Testing Instructions

  • Unit Tests:
    python butler.py py_unittest -t appengine -p load_bigquery_stats_test.py
    
  • Unit Tests results:
image

@stormKx18
stormKx18 requested a review from dylanjew July 16, 2026 17:14
@stormKx18 stormKx18 self-assigned this Jul 16, 2026
@stormKx18
stormKx18 requested a review from a team as a code owner July 16, 2026 17:14

@dylanjew dylanjew left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice! Mostly LGTM. Just one nit.

I think you can go ahead and test this on dev by running the cron job with butler. You may need to verify that the args are passed in correctly, I remember that being a bit annoying when trying to run the cron

@dylanjew dylanjew left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants