Skip to content

fix: normalize ModelTrainer Tag objects inherited by HyperparameterTuner - #6482

Merged
mohamedzeidan2021 merged 1 commit into
aws:masterfrom
mohamedzeidan2021:fix/hyperparametertuner-fails-on-v3-modeltrainer-tags-6472
Oct 16, 2026
Merged

mohamedzeidan2021 merged 1 commit into
aws:masterfrom
mohamedzeidan2021:fix/hyperparametertuner-fails-on-v3-modeltrainer-tags-6472

Conversation

@mohamedzeidan2021

Copy link
Copy Markdown
Collaborator

Issue #, if available: Fixes #6472

Description

HyperparameterTuner.tune() raised TypeError: 'Tag' object is not subscriptable for any ModelTrainer that had tags set, before the tuning job was created.

ModelTrainer.tags is typed Optional[List[sagemaker.core.shapes.Tag]], so its tags are pydantic Tag objects. HyperparameterTuner.tags carries boto-style {"Key": ..., "Value": ...} dicts (format_tags passes lists through untouched). _prepare_tags_for_tuning() merged the trainer's Tag objects into that dict list without converting them, producing a mixed-shape list. Both consumers of the list then subscript every element:

  • add_jumpstart_uri_tags() → tag_key_in_array() reads tag["Key"] (tuner.py:328, reached first when include_jumpstart_tags is on — the default)
  • _start_tuning_job() builds Tag(key=tag["Key"], value=tag["Value"]) (tuner.py:1280, the traceback in the issue)

There was no workaround: ModelTrainer's tags field is strictly typed, so passing plain dicts to the trainer still yields Tag objects. The only way to avoid the crash was to leave the trainer untagged, which drops the tags from the training jobs.

This change adds a _to_boto_tag() helper and normalizes tags to boto-style form before the merge, which fixes both call sites in one place and lets the existing if tag not in self.tags dedup compare equivalent tags across shapes. tuner.tags stays None when neither the tuner nor the trainer defines tags, so the no-tags behavior is unchanged.

Verified against the reproduction script from the issue on 3.25.0: before, it raised at tuner.py:1280; after, tuner.tags is [{'Key': 'project', 'Value': 'tuner-tag-reproduction'}] and the call proceeds to CreateHyperParameterTuningJob with the tag preserved.

Note for follow-up (not changed here to keep this focused): the underlying gap is that format_tags() in sagemaker-core/.../common_utils.py returns any list unchanged, so it never normalizes v3 Tag objects at v2/v3 boundaries. Teaching it about Tag objects would address this class of bug at its ~40 call sites.

Testing

Added TestHyperparameterTunerTagInheritance to sagemaker-train/tests/unit/train/test_tuner.py (8 tests): inheriting Tag objects, merging with the tuner's own boto-style tags, dedup of equivalent tags, tags staying None when unset, normalization before the JumpStart tag helper, _start_tuning_job tolerating Tag objects, and the _to_boto_tag lowercase-dict and pass-through cases.

The 5 behavior tests fail on master and pass with the fix; the 3 helper tests cover the new function.

pytest tests/unit/train/test_tuner.py tests/unit/train/test_tuner_phase5.py \
       tests/unit/train/test_tuner_driver_channels.py tests/unit/train/test_model_trainer.py
193 passed

black --check, flake8, pydocstyle clean on both changed files; pylint --rcfile=.pylintrc score unchanged at 9.90/10 with no new findings.

Merge Checklist

Put an x in the boxes that apply.

General

  • I have read the CONTRIBUTING doc
  • I certify that the changes I'm introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
  • I have checked that my tests are not configured for a specific region or account (if appropriate)
  • I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

ModelTrainer stores tags as sagemaker.core.shapes.Tag objects, while
HyperparameterTuner carries boto-style {"Key": ..., "Value": ...} dicts.
_prepare_tags_for_tuning() merged the trainer's Tag objects into that list
without converting them, so the resulting list held both shapes. Both
consumers of that list subscript each tag:

* add_jumpstart_uri_tags() -> tag_key_in_array() reads tag["Key"]
* _start_tuning_job() builds Tag(key=tag["Key"], value=tag["Value"])

Either path raised TypeError: 'Tag' object is not subscriptable before the
tuning job was created, so any tagged ModelTrainer broke tune().

Normalize each tag to boto-style form before merging, which fixes both
call sites and lets the dedup check compare equivalent tags. Tags are left
as None when neither the tuner nor the trainer defines any.

Fixes aws#6472

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI code review

This PR fixes a TypeError: 'Tag' object is not subscriptable in HyperparameterTuner.tune() when a ModelTrainer has tags set. The underlying problem is that ModelTrainer.tags holds pydantic Tag objects while the tuner expects boto-style {"Key": ..., "Value": ...} dicts, and the merge mixed the two shapes before downstream code subscripted every element. The fix adds a _to_boto_tag() helper that normalizes tags before the merge and defensively at the _start_tuning_job call site, and adds focused unit tests covering inheritance, dedup, the no-tags case, the JumpStart path, and the helper itself. The change is correct, minimal, backward-compatible, and well-covered; I found no blocking issues.

Reviewed commit dd2fb22. Automated review; verify before acting.

@mohamedzeidan2021
mohamedzeidan2021 merged commit c65ad99 into aws:master Oct 16, 2026
26 of 27 checks passed

This branch was successfully deployed

1 active deployment
auto-approve — dd2fb225 Deployed Oct 15, 2026 by mohamedzeidan2021 via wait-for-approval #1764
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HyperparameterTuner fails when it inherits v3 ModelTrainer.tags

2 participants