Skip to content

fix: use pipeline code_hash for FrameworkProcessor sourcedir upload - #6494

Open
hippi345 wants to merge 2 commits into
aws:masterfrom
hippi345:fix/framework-processor-pipeline-cache-6493
Open

hippi345 wants to merge 2 commits into
aws:masterfrom
hippi345:fix/framework-processor-pipeline-cache-6493

Conversation

@hippi345

Copy link
Copy Markdown

Issue #, if available: #6493

Description of changes:

Fixes #6493. FrameworkProcessor._package_code() always used job_name in the sourcedir S3 key, so pipelines never got cache hits for jobs with source_dir. It now uses _pipeline_config.pipeline_name and _pipeline_config.code_hash when they're set, matching ScriptProcessor._upload_code(). Added the unit test test_package_code_with_pipeline_config, which fails on current master (the S3 key contains test-job) and passes with the fix. The 5 local-mode failures in test_processing.py also fail on unmodified master because the test environment has no AWS region set.

Note: prepared with AI assistance and reviewed by the author.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

FrameworkProcessor._package_code() now uses _pipeline_config.code_hash
in the S3 key when available, matching the behavior of
ScriptProcessor._upload_code(). This enables pipeline cache hits for
FrameworkProcessor jobs with source_dir.

Fixes aws#6493
@hippi345
hippi345 marked this pull request as ready for review October 16, 2026 17:53
@Nimisha987

Copy link
Copy Markdown

The new hash-based s3_payload URI no longer contains sourcedir.tar.gz. However, _pack_and_upload_code() derives the runproc.sh and install_requirements.py URIs using s3_payload.replace("sourcedir.tar.gz", ...). When the pipeline code hash is used, both replacements return the original URI, causing both helper uploads to target the same S3 object as the source archive.

Could we preserve the sourcedir.tar.gz filename under the hash-specific prefix and add a regression test covering _pack_and_upload_code() with a pipeline code hash? This would ensure the source archive and helper scripts are uploaded to distinct S3 locations.

Preserve the sourcedir.tar.gz filename under the pipeline code-hash
prefix so _pack_and_upload_code helper replace() URIs stay distinct.
Add a regression test covering _pack_and_upload_code with two hashes.
@cursor
cursor Bot requested a deployment to manual-approval October 17, 2026 09:22 Waiting
@cursor
cursor Bot requested a deployment to manual-approval October 17, 2026 09:22 Waiting
@cursor
cursor Bot requested a deployment to manual-approval October 17, 2026 09:22 Waiting
@hippi345

Copy link
Copy Markdown
Author

Thanks, that was a real collision. _pack_and_upload_code() still derives helper URIs with s3_payload.replace("sourcedir.tar.gz", ...), so the pipeline hash key now keeps that filename: {pipeline_name}/source/{code_hash}/sourcedir.tar.gz. Added test_pack_and_upload_code_with_pipeline_config to assert the hash prefix, distinct helper uploads, and that two hashes produce different keys.

This branch is waiting to be deployed

1 waiting deployment
manual-approval — 5cab44bd Waiting Oct 17, 2026 by cursor[bot] via wait-for-approval #2057
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FrameworkProcessor._package_code ignores _pipeline_config.code_hash, so ProcessingStep caching never hits when source_dir is used

2 participants