Repository navigation
fix: normalize ModelTrainer Tag objects inherited by HyperparameterTuner - #6482
Conversation
ModelTrainer stores tags as sagemaker.core.shapes.Tag objects, while
HyperparameterTuner carries boto-style {"Key": ..., "Value": ...} dicts.
_prepare_tags_for_tuning() merged the trainer's Tag objects into that list
without converting them, so the resulting list held both shapes. Both
consumers of that list subscript each tag:
* add_jumpstart_uri_tags() -> tag_key_in_array() reads tag["Key"]
* _start_tuning_job() builds Tag(key=tag["Key"], value=tag["Value"])
Either path raised TypeError: 'Tag' object is not subscriptable before the
tuning job was created, so any tagged ModelTrainer broke tune().
Normalize each tag to boto-style form before merging, which fixes both
call sites and lets the dedup check compare equivalent tags. Tags are left
as None when neither the tuner nor the trainer defines any.
Fixes aws#6472
There was a problem hiding this comment.
🤖 AI code review
This PR fixes a TypeError: 'Tag' object is not subscriptable in HyperparameterTuner.tune() when a ModelTrainer has tags set. The underlying problem is that ModelTrainer.tags holds pydantic Tag objects while the tuner expects boto-style {"Key": ..., "Value": ...} dicts, and the merge mixed the two shapes before downstream code subscripted every element. The fix adds a _to_boto_tag() helper that normalizes tags before the merge and defensively at the _start_tuning_job call site, and adds focused unit tests covering inheritance, dedup, the no-tags case, the JumpStart path, and the helper itself. The change is correct, minimal, backward-compatible, and well-covered; I found no blocking issues.
Reviewed commit dd2fb22. Automated review; verify before acting.
Issue #, if available: Fixes #6472
Description
HyperparameterTuner.tune()raisedTypeError: 'Tag' object is not subscriptablefor anyModelTrainerthat had tags set, before the tuning job was created.ModelTrainer.tagsis typedOptional[List[sagemaker.core.shapes.Tag]], so its tags are pydanticTagobjects.HyperparameterTuner.tagscarries boto-style{"Key": ..., "Value": ...}dicts (format_tagspasses lists through untouched)._prepare_tags_for_tuning()merged the trainer'sTagobjects into that dict list without converting them, producing a mixed-shape list. Both consumers of the list then subscript every element:add_jumpstart_uri_tags()→tag_key_in_array()readstag["Key"](tuner.py:328, reached first wheninclude_jumpstart_tagsis on — the default)_start_tuning_job()buildsTag(key=tag["Key"], value=tag["Value"])(tuner.py:1280, the traceback in the issue)There was no workaround:
ModelTrainer's tags field is strictly typed, so passing plain dicts to the trainer still yieldsTagobjects. The only way to avoid the crash was to leave the trainer untagged, which drops the tags from the training jobs.This change adds a
_to_boto_tag()helper and normalizes tags to boto-style form before the merge, which fixes both call sites in one place and lets the existingif tag not in self.tagsdedup compare equivalent tags across shapes.tuner.tagsstaysNonewhen neither the tuner nor the trainer defines tags, so the no-tags behavior is unchanged.Verified against the reproduction script from the issue on
3.25.0: before, it raised attuner.py:1280; after,tuner.tagsis[{'Key': 'project', 'Value': 'tuner-tag-reproduction'}]and the call proceeds toCreateHyperParameterTuningJobwith the tag preserved.Note for follow-up (not changed here to keep this focused): the underlying gap is that
format_tags()insagemaker-core/.../common_utils.pyreturns any list unchanged, so it never normalizes v3Tagobjects at v2/v3 boundaries. Teaching it aboutTagobjects would address this class of bug at its ~40 call sites.Testing
Added
TestHyperparameterTunerTagInheritancetosagemaker-train/tests/unit/train/test_tuner.py(8 tests): inheritingTagobjects, merging with the tuner's own boto-style tags, dedup of equivalent tags, tags stayingNonewhen unset, normalization before the JumpStart tag helper,_start_tuning_jobtoleratingTagobjects, and the_to_boto_taglowercase-dict and pass-through cases.The 5 behavior tests fail on
masterand pass with the fix; the 3 helper tests cover the new function.black --check,flake8,pydocstyleclean on both changed files;pylint --rcfile=.pylintrcscore unchanged at 9.90/10 with no new findings.Merge Checklist
Put an
xin the boxes that apply.General
Tests
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.