Repository navigation
Expand documentation for importlib.metadata custom providers to include caveats regarding backports #123976
Description
Documentation
As importlib_metadata provides a backport of importlib.metadata and asserts primacy when imported, for consumers of the API, it provides a consistent interface, except when a custom provider serves objects as imported from importlib.metadata (not using the backport). For example:
- project A implements and installs a DistributionFinder to
sys.meta_pathprovidingimportlib.metadata.Distributioninstances. - project B calls
importlib_metadata.distributions()to return all distributions, including that of project A. - A user installs project A and project B.
- When project B queries for the distributions, expecting it to return
Iterable[importlib_metadata.Distribution]objects, their expectation is violated when theDistributionFinderinstalled by project A emitsimportlib.metadata.Distributionobjects.
python/importlib_metadata#486 captures this concern. python/importlib_metadata#505 promises to correct this concern by wrapping and transforming any objects from custom providers referencing importlib.metadata to instead use objects from importlib_metadata.
However, for importlib.metadata.Distribution objects, it's not possible to replace safely the objects of an arbitrary custom provider, so the proposed implementation emits a warning when such an object is encountered.
Therefore, we'd like to add some documentation to the custom provider docs to advise custom providers on what to do to avoid their users encountering the warning. In particular:
- Custom providers should emit
importlib_metadata.Distributionobjects whenimportlib_metadatais present, or- Custom providers should register their objects with
importlib_metadatato bypass the warning.
Update: the protocols landed, and need syncing
python/importlib_metadata#505 merged 2026-10-23 and released in importlib_metadata 10.0.0. It takes the shared-protocol approach settled on in python/importlib_metadata#486 rather than either recommendation above:
IDistributionandIPackagePathprotocols join the existingPackageMetadata, so providers and consumers have one interface that the classes of both libraries satisfy.- Objects a provider supplies from
importlib.metadataare adapted to this package's classes where they can be. An arbitraryDistributionsubclass cannot be, so it is passed through, and the warning now fires only where the class omits membersIDistributionrequires, naming them, rather than merely because it derives fromimportlib.metadata.
So neither recommendation above is needed: a provider implements the interface, and a conforming one draws no warning. gh-160344 drafts the documentation on that basis.
That documentation names the protocols, so this issue now also covers getting them into the stdlib. Under the development methodology, _meta.py carries the protocols and syncs downstream, while the compat and _compat packages holding localize and its warning are excluded from the port. Only the protocols arrive here, which is what gh-160344 assumes.
Lib/importlib/metadata/ was last synced at importlib_metadata 8.7 (gh-132947, 3.15.0a1). The current release is 10.1.0, so a sync spans 9.0.0, 9.0.1, 9.1.0, 10.0.0 and 10.1.0 — among them the MetadataNotFound and KeyError changes, lazy EntryPoint.value validation, the metadata directory preference added in 9.1.0 and reverted in 10.1.0, and the Prepared.normalized and metadata detection performance work. Some of those originated in CPython and have already been ported back, so the diff wants tracing rather than taking wholesale.
Tasks
- Sync
Lib/importlib/metadata/with importlib_metadata 10.1.0 - Document the custom provider caveats (gh-123976: Document the protocols shared with the importlib_metadata backport #160344), which depends on the sync
Activity
jaraco commentedon Sep 11, 2024
As I write that recommendation, I'm already uncomfortable with the recommendation. It's unclear what "when
importlib_metadatais present" means. Does it mean the module is available for import? Does it mean it's been imported previously (e.g. insys.modules)?I think it has to mean it's available for import, because the provider can't tell if it will later be imported by a consumer. But that also means the provider might activate
importlib_metadatain an environment that would otherwise have never had it activated. Perhaps that's not a huge concern - if it's installed, it's likely to be activated.That also means that users of
importlib.metadatawill be gettingimportlib_metadata.Distributionobjects, violating the type spec. But that's the case already today, so perhaps that's not so bad.Also, "register with
importlib_metadatato bypass the warning" sounds messy at best. There probably aren't a lot of providers out there, but even so, having to make new releases that are required to support their custom provider does not sound like a lot of fun for anyone.jaraco commentedon Sep 11, 2024
All of this has me re-thinking the strategy.
Maybe instead of having two distinct class trees (
importlib.metadata.Distributionandimportlib_metadata.Distributionand subclasses), there should be a protocol that's shared by both versions of the library, explicitly modeling what's currently implicit - that there's an interface expected byDistributionobjects (alsoPackagePathandMessageobjects), and those protocols are what the two libraries are coded against.pythongh-123976: Refresh docs around custom providers.
pythongh-123976: Refresh docs around custom providers.
gh-123976: Refresh docs around custom providers. (#123977)
hugovk commentedon Sep 15, 2024
Triage: #123977 has been merged, ready to close this?
jaraco commentedon Sep 15, 2024
Thanks, but no. The PR only did some minor refreshing of the docs. The core concern of this issue remains.
pythongh-123976: Warn only on an incomplete provider
pythongh-123976: Separate the consumer and provider caveats
pythongh-123976: Document the protocols shared with the backport