Upload Fuzzer Logs to GCS only if not in a uworker#5363
Conversation
dylanjew
left a comment
dylanjew
left a comment
There was a problem hiding this comment.
drive by, for context I was looking into a similar issue in crbug.com/534369736
| target=testcase_manager.run_testcase_and_return_result_in_queue, | ||
| args=(temp_queue, thread_index, testcase_file_path, gestures, | ||
| env_copy, True)) | ||
| env_copy, not environment.is_uworker())) |
There was a problem hiding this comment.
should this just be environment.is_tworker? what about swarming and off swarming bots (like macs)?
It would be nice if there was a single function we could call with confidence that determined whether we have access to GCS/metadata and cleanly handled all environments :)
There was a problem hiding this comment.
Tworkers only pick up pre/post tasks, hence they won't end up executing this workload.
Heres where it gets tricky, there are 2 kind of bots that can reach this code path in the fuzzing task workflow:
- uworkers &
- pre-emptible bots, these are bots that only execute fuzzing tasks, and they execute them locally(see how fuzz is a local Utask) using a trusted service account
Pre-emptibles don't have either the uworker nor the tworker flags in place.
So to make this work for both types of bots, we only check, if its its a uworker bot or not. There are upcoming PRs to change this a little bit more(see b/473624472 for more context)
There was a problem hiding this comment.
This explains why although this issue appears on all uworkers, it doesnt completely hindered our fuzzing hours metric.
There was a problem hiding this comment.
Got it, thanks for the explanation!
2 participants
Bug: b/473624472
Overview
After reverting this PR
PERMISSION_DENIEDerrors started to reappear in CF. Its clear that we can't follow the same approach of doing a short-circuit in the DB calls.This instance of
PERMISSION_DENIEDerrors occurs onuworkersthat perform thefuzztask, in this workflow theres a logic to upload the Fuzzer logs to GCS at theutask_mainstage. As we knowuworkerscant access nor DB nor GCS, so when auworkerreaches this code it fails. Now instead of always uploading the Fuzzer logs to GCS, we only upload them if not in auworker, this allows regular bots that perform the fuzz task(pre->main->pos) locally to continue to work.Tests performed
Changes have been in
devsince Jul-13. Since then error logs related to this instance ofPERMISSION_DENIEDissues have been resolved.Note how this error dramatically decreases. Other instances of this error like this one:
Are unrelated to this changes and seem intermittent errors due to external reasons.
Note:
This PR is the first of 3(maybe 4) PRs to tackle the PERMISSION_DENIED issues