Repository navigation
fix: pass the configured session to ModelBuilder's Model reloads - #6487
Merged
mohamedzeidan2021 merged 1 commit intoOct 16, 2026
Conversation
ModelBuilder.build() creates the SageMaker Model through the configured sagemaker_session, but reloaded it with a session-less Model.get(). That helper builds its own boto3 session from the ambient credential chain, so DescribeModel is signed with the default account's credentials: in a cross-account setup CreateModel succeeds in the target account and the reload then fails with "Could not find model". Single-account setups hide the defect because both sessions resolve to the same account. Pass sagemaker_session.boto_session to Model.get() at all three Model reload/lookup sites in model_builder.py (_create_sagemaker_model, _find_reusable_model and _get_model_for_endpoint - the latter two read the model name from the configured session's client and then described it session-lessly, the same defect on build()'s reuse path), matching the pattern the Endpoint.get() call sites already use after aws#6069. Model.get() also records the session on the resource it returns, so refresh, delete and wait calls on the Model that build() hands back stay in the configured account as well. Fixes aws#6459
There was a problem hiding this comment.
🤖 AI code review
This PR fixes a cross-account bug in ModelBuilder by passing the configured sagemaker_session.boto_session to three Model.get() reload/lookup call sites (_create_sagemaker_model, _find_reusable_model, _get_model_for_endpoint), matching the already-fixed Endpoint.get() pattern elsewhere in the same file. I confirmed Model.get() accepts a session kwarg and that the existing Endpoint.get() sites use exactly this self.sagemaker_session.boto_session form, so the change is correct and consistent. New unit tests cover all three paths plus the PipelineSession early-return. The change looks clean; no issues found.
Reviewed commit 8249ac5. Automated review; verify before acting.
aviruthen
approved these changes
Oct 16, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available: Fixes #6459
Description
ModelBuilder.build()created the SageMaker Model through the configuredsagemaker_sessionand then reloaded it with a session-lessModel.get():Model.get()resolves its client throughBase.get_sagemaker_client(), and withsession=Nonethat helper constructs a brand-newboto3.Sessionfrom the ambient credential chain (SageMakerClient.__init__: "No boto3 session provided. Creating a new session."). SoCreateModelis signed with the configured session's credentials while the follow-upDescribeModelis signed with the process default ones. Cross-account, the model is created in the target account and the reload fails withValidationException: Could not find model ..., exactly as reported. Single-account setups hide the defect because both sessions resolve to the same account. This is the same class of bug as #6069; the fixes for that issue (#6107 / #6120) covered theEndpoint.get()call sites on the deploy path but not the Model reload on the build path.The fix passes
self.sagemaker_session.boto_sessiontoModel.get(), which is the pattern the already-fixedEndpoint.get()call sites in this same file use. It is also more than cosmetic for the returned object:Model.get()calls_set_client_context(session=..., region=...), so theModelthatbuild()hands back now keeps the configured session for its ownrefresh(),delete()andwait_*calls instead of silently falling back to the default account.Widened beyond the one-line patch in the issue. The suggested diff fixes
_create_sagemaker_model()only. Grepping every resource lookup on the build path found two moreModel.get()calls with the identical shape — the model name is read throughself.sagemaker_session.sagemaker_clientand then described with no session, so they are guaranteed to be cross-account inconsistent:_create_sagemaker_model()build()→_create_model()(the reported path)_find_reusable_model()build(reuse_resources=True)_get_model_for_endpoint()All three are fixed here. Nothing depends on the old behaviour: no unit test in the repo asserts a session-less
Model.get()call, and thePipelineSessionbranch returns before the reload, so pipeline builds are untouched (covered by a new test).Deliberately not widened, to keep this change on one path:
model_builder.pystill has session-lessHubContent.get()(×2),TrainingJob.get()(×3),ModelPackage.get()(×1), and, on the Nova/LoRA model-customization deploy path,Endpoint.get()(×2, incl._does_endpoint_exist) andInferenceComponent.get()(×2). Those are the same bug class on different resources and a different path, and fixing them needs their own tests; happy to follow up in a separate PR if you'd like them bundled.Testing
New file
sagemaker-serve/tests/unit/test_model_builder_session_propagation.py, 5 tests, modelled on the existing issue-scoped filetest_model_builder_pipeline_variable_image_uri.py. It uses two real, distinguishableboto3.Sessionobjects (a "configured" one and a "default" one) so the assertion is about which session object the reload actually receives:Model.get()receivessession=<configured boto session>afterCreateModel(_create_sagemaker_model).Base.get_sagemaker_clientintercepted, the client that issuesDescribeModelis built from the configured session and the right region — this is the assertion that fails withsession=Noneonmaster.PipelineSessionbuild still returns before the reload and never callsModel.get()(guards the path that already worked)._find_reusable_model()passes the configured session._get_model_for_endpoint()passes the configured session.Negative check (the installed
sagemaker3.25.0model_builder.pyrestored to its released content, new tests unchanged):(The one that passes either way is the
PipelineSessionregression guard, by design.)Full package regression,
sagemaker-serve/tests/unit:The two
FAILEDlists are identical; the +5 is the new tests. All 70 are pre-existing and environmental in this sandbox: 65 fail withbotocore ClientError: ExpiredToken(they call live AWS) and 5 withModuleNotFoundError: No module named 'tensorflow'.Lint on both changed files:
black --line-length 100 --checkclean,flake8 --max-line-length 120clean,pydocstyleclean on thesrcfile.pylint --rcfile=.pylintrc:model_builder.py9.84/10 before and after with an identical message set (one pre-existingE0203line number shifts); package-widesrc/sagemaker9.89/10 before and after (+0.00), 83 messages in both runs — no new findings.Not verified: there are no usable AWS credentials in my environment, so I did not run a live cross-account round trip of the reproduction script from the issue. The evidence above is offline and is about which boto3 session object the reload's client is constructed from; the account-level failure mode is inferred from that plus the CloudTrail evidence in the issue.
Merge Checklist
Put an
xin the boxes that apply.General
Tests
On the region item: this change initializes no S3 or STS clients. The only client it affects is the SageMaker client created inside
Model.get(), which now receives both the configured session andregion=self.region. The documentation item is unticked because the change is a bug fix with no user-facing API or doc surface, and no integ tests were added (hence the last item).By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.