Repository navigation
Add a paginated Python repository package catalog #1451
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
c89cef4
9d5e237
95012ae
6956bb3
6f0157e
26dce87
227b70b
58c4b0e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Added paginated package catalogs for repositories and repository versions, excluding yanked versions and reporting when files were added to the repository. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| # Browse the package catalog | ||
|
|
||
| The package catalog returns one row per normalized package name. Use the | ||
| repository URL to read its latest complete repository version, or a repository | ||
| version URL to read a specific complete version. `${REPO_PK}` is the repository UUID. | ||
|
|
||
| ## List packages | ||
|
|
||
| ```bash | ||
| http GET "${BASE_ADDR}/pulp/api/v3/repositories/python/python/${REPO_PK}/packages/?limit=10" | ||
| ``` | ||
|
|
||
| To read a specific repository version, set `${VERSION_NUMBER}` to its number: | ||
|
|
||
| ```bash | ||
| http GET "${BASE_ADDR}/pulp/api/v3/repositories/python/python/${REPO_PK}/versions/${VERSION_NUMBER}/packages/?limit=10" | ||
| ``` | ||
|
|
||
| `count` is the number of distinct packages, not files. `summary`, `description`, | ||
| `author`, `author_email`, and `license` come from the highest package version in | ||
| PEP 440 order. Metadata comes from an sdist of that version when available. | ||
| Otherwise, it comes from that version's newest file. | ||
|
|
||
| `versions` has one entry per stored version string, in descending PEP 440 order. | ||
| A wheel and an sdist of the same version share one entry. Different version | ||
| strings, such as `1.0.0` and `1.0.0+test.1`, produce separate entries. | ||
|
|
||
| Each version's `last_updated` is the latest `pulp_created` timestamp among its | ||
| repository membership records in the selected repository version. This records | ||
| when a file was added to the repository. Removing and re-adding a file updates | ||
| this time. Historical repository versions retain the earlier addition time. | ||
|
|
||
| Package versions with a yank marker in the selected repository version are excluded. | ||
| Packages with no remaining versions are excluded from both the results and `count`. | ||
|
|
||
| ```json | ||
| { | ||
| "count": 1, | ||
| "next": null, | ||
| "previous": null, | ||
| "results": [ | ||
| { | ||
| "name": "shelf-reader", | ||
| "name_normalized": "shelf-reader", | ||
| "summary": "A small example package", | ||
| "description": "", | ||
| "author": "", | ||
| "author_email": "", | ||
| "license": "", | ||
| "versions": [ | ||
| {"version": "0.1", "last_updated": "2026-11-11T12:00:00Z"} | ||
| ] | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| Results are ordered by `name_normalized`. Use `limit` and `offset` to page | ||
| through the catalog. Use a repository version URL to keep every page on the | ||
| same repository version. Missing or incomplete repository versions return HTTP 404. | ||
|
|
||
| Filter names with `name_normalized`, `name_normalized__icontains`, or | ||
| `name_normalized__istartswith`. Use `ordering=-name_normalized` to list names | ||
| in descending order. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| """Helpers for the repository package catalog.""" | ||
|
|
||
| from django import VERSION as DJANGO_VERSION | ||
| from django.contrib.postgres.aggregates import JSONBAgg | ||
| from django.db.models import Case, Exists, F, IntegerField, OuterRef, Q, Value, When | ||
| from django.utils.dateparse import parse_datetime | ||
| from packaging.version import InvalidVersion, Version | ||
|
|
||
| from pulp_python.app.models import PackageYank, PythonPackageContent | ||
|
|
||
|
|
||
| def package_catalog_content(repository_version): | ||
| """Select unyanked files and their addition times in the repository version.""" | ||
| content = PythonPackageContent.objects.all() | ||
| if repository_version is None: | ||
| return content.annotate(repository_added=F("pulp_created")).none() | ||
| yanks = repository_version.get_content(PackageYank.objects.all()).filter( | ||
| name_normalized=OuterRef("name_normalized"), version=OuterRef("version") | ||
| ) | ||
| return ( | ||
| content.filter( | ||
| Q(version_memberships__version_removed__isnull=True) | ||
| | Q(version_memberships__version_removed__number__gt=repository_version.number), | ||
| version_memberships__repository_id=repository_version.repository_id, | ||
| version_memberships__version_added__number__lte=repository_version.number, | ||
| ) | ||
| .annotate(repository_added=F("version_memberships__pulp_created")) | ||
| .filter(~Exists(yanks)) | ||
| ) | ||
|
|
||
|
|
||
| def _version_sort_key(version): | ||
| """Sort invalid version strings after valid PEP 440 versions.""" | ||
| try: | ||
| return True, Version(version) | ||
| except InvalidVersion: | ||
| return False, version | ||
|
|
||
|
|
||
| class PackageVersions(JSONBAgg): | ||
| """Aggregate values by version, keeping the last value for duplicate keys.""" | ||
|
|
||
| function = "JSONB_OBJECT_AGG" | ||
| arity = 2 | ||
|
|
||
|
|
||
| def package_catalog_queryset(content, ordering=("name_normalized",)): | ||
| """Collect version timestamps and metadata IDs for each normalized package name. | ||
|
|
||
| Wheels and sdists with the same version share an aggregate entry. | ||
| Each timestamp is the latest file addition to the repository. Use sdist metadata when available. | ||
| """ | ||
| # Aggregate ordering selects the latest addition time and prefers sdist metadata. | ||
| ordering_argument = "order_by" if DJANGO_VERSION >= (5, 2) else "ordering" | ||
| file_ordering = { | ||
| ordering_argument: ( | ||
| "version", | ||
| Case( | ||
| When(packagetype="sdist", then=Value(1)), | ||
| default=Value(0), | ||
| output_field=IntegerField(), | ||
| ), | ||
| "pulp_created", | ||
| "-pk", | ||
| ) | ||
| } | ||
| return ( | ||
| content.order_by(*ordering) | ||
| .values("name_normalized") | ||
| .annotate( | ||
| versions=PackageVersions( | ||
| "version", | ||
| "repository_added", | ||
| **{ordering_argument: ("repository_added",)}, | ||
| ), | ||
| metadata_ids=PackageVersions( | ||
| "version", | ||
| "pk", | ||
| **file_ordering, | ||
| ), | ||
| ) | ||
| ) | ||
|
|
||
|
|
||
| def assemble_package_index(package_rows): | ||
| """Sort each package's versions and fetch metadata for its highest version in one query.""" | ||
|
gerrod3 marked this conversation as resolved.
Resolved
|
||
| if not package_rows: | ||
| return [] | ||
|
|
||
| for row in package_rows: | ||
| row["versions"] = sorted( | ||
| ( | ||
| {"version": version, "last_updated": parse_datetime(updated)} | ||
| for version, updated in row["versions"].items() | ||
| ), | ||
| key=lambda unit: (_version_sort_key(unit["version"]), unit["version"]), | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Prefer an sdist of the highest PEP 440 version when choosing package metadata. Metadata selection reads only the top stored version string. The tie-break at line 96 orders by the raw string after PEP 440 equality. When an sdist and a wheel share the highest PEP 440 version under different strings, such as Select metadata from all files whose PEP 440 version equals the highest version. Prefer an sdist among them. Keep the per-string
|
||
| reverse=True, | ||
| ) | ||
|
|
||
| fields = ( | ||
| "name", | ||
| "name_normalized", | ||
| "summary", | ||
| "description", | ||
| "author", | ||
| "author_email", | ||
| "license", | ||
| ) | ||
| metadata = { | ||
| str(unit["pk"]): {field: unit[field] for field in fields} | ||
| for unit in PythonPackageContent.objects.filter( | ||
| pk__in=[row["metadata_ids"][row["versions"][0]["version"]] for row in package_rows] | ||
| ) | ||
| .order_by() | ||
| .values("pk", *fields) | ||
| } | ||
| return [ | ||
| { | ||
| **metadata[row["metadata_ids"][row["versions"][0]["version"]]], | ||
| "versions": row["versions"], | ||
| } | ||
| for row in package_rows | ||
| ] | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Document how pruning affects
last_updated.If v1 adds a file, v2 removes it, and v3 re-adds it, pruning v2 can change v3’s
last_updatedfrom the v3 membership time to the v1 membership time. Document this limitation of the membership timestamp. Add a functional test that reads v3 before and after pruning.