diff --git a/package-structure-code/declare-dependencies.md b/package-structure-code/declare-dependencies.md index 56519a44..75827f2b 100644 --- a/package-structure-code/declare-dependencies.md +++ b/package-structure-code/declare-dependencies.md @@ -506,3 +506,232 @@ Why you specify dependencies How to specify dependencies When you use different specifiers ::: + +## Dependency Locking + +In addition to declaring dependencies in `pyproject.toml`, it is common for +packages to lock down exact versions of all their dependencies in a separate +lock file. A lock file provides benefits of reproducibility, security, and +potentially faster installs, among other things. Pinning the exact dependency +versions used in a project eliminates "works on my machine" bugs and gives CI a +reproducible baseline. For applications meant to be run rather than imported, +lock files also ensure anyone installing the project gets a known-good set of +dependencies instead of whatever happens to be latest. + +### `pyproject.toml` vs lock file +* `pyproject.toml`: defines all supported environments for users importing +your package into their project. +* **lock file**: defines a specific environment used for development + +:::{admonition} Standardized Lock File +:class: note +As of March 2025, [PEP 751](https://peps.python.org/pep-0751) defined a standard +`pylock.toml` format to unify the various lock file formats in use by other +package managers (e.g. `uv.lock`, `poetry.lock`, `pdm.lock`). Most package +managers provide ways to generate a PEP 751 compatible file. See [PyPA +specification](https://packaging.python.org/en/latest/specifications/pylock-toml/) +for up-to-date formatting info on `pylock.toml` +::: + +### How to work with lock files? + +Lock files are not written by hand. Package managers and IDEs provide tools +to create, update, and reformat lock files as needed. + +1) **Create** - Package managers often do this automatically though it can be +done manually. For example, calling `uv add numpy` will automatically create a +`uv.lock` file, setup the environment, and install numpy. +2) **Update** - This is not done automatically by package managers. +Maintainers can choose to do this manually or setup their own automated +workflow. Updates can be for specific packages or all dependencies. +3) **Reformat** - Package managers currently use native formats (e.g. +uv uses `uv.lock`) and provide tools for converting into `pylock.toml` and other +formats (e.g. `requirements.txt`) when needed + +Below are common package manager CLI workflows for lock files: + +::::{tab-set} + +:::{tab-item} uv (recommended) +```sh +# Create a uv.lock file based on pyproject.toml +> uv lock + +# Update uv.lock +> uv lock --upgrade +> uv lock --upgrade-package pandas + +# Install packages into environment based on uv.lock +> uv sync + +# PEP 751 pylock.toml support +> uv export --format pylock.toml -o pylock.toml # export uv.lock -> pylock.toml +> uv pip sync pylock.toml # install from pylock.toml +``` +See [official docs](https://docs.astral.sh/uv/concepts/projects/sync/) for more details +::: + +:::{tab-item} Poetry +```sh +# Create a poetry.lock file based on pyproject.toml +> poetry lock + +# Update poetry.lock +> poetry update +> poetry update pandas numpy + +# Install packages into environment based on poetry.lock +> poetry sync + +``` +PEP 751 pylock.toml not yet supported (track progress on [GitHub](https://github.com/python-poetry/poetry/issues/10356)) + +See [official docs](https://python-poetry.org/docs/basic-usage/#installing-dependencies) for more details +::: + +:::{tab-item} PDM +```sh +# Create a pdm.lock file based on pyproject.toml +> pdm lock + +# Update pdm.lock +> pdm update +> pdm update pandas numpy + +# Install packages into environment based on pdm.lock +> pdm sync + +# PEP 751 pylock.toml support +> pdm export -f pylock -o pylock.toml # export pdm.lock -> pylock.toml +``` +See [official docs](https://pdm-project.org/latest/usage/lockfile/) for more details +::: + +:::: + +### Should I use a lock file? + +Most package managers will generate a lock file automatically for you (e.g. uv, +Poetry, PDM). The real question is when you version control the lock file as +part of your package. + +:::{admonition} Recommendation: Versioning a lock file +:class: tip +If your project is an application others use directly, include a lock file as +the recommended environment. + +If your project is a library to be used in other projects and it is mature +enough to have CI, include a lock file for CI and contributors. For a small +library only you maintain that is shared amongst people you know, waiting to add +a lock file is not an issue. +::: + +There is some maintenance cost from lock files. Maintainers should aim to update +the lock file neither too rarely nor too often. +* Too rarely means you risk missing updates with bugfixes, security patches, +performance improvements, etc. +* Too often means you may introduce bugs or even security vulnerablilites before +maintainers of your dependencies catch them. Package managers are starting to +support dependency cooldowns to mitigate this. + +:::{admonition} Recommendation: Updating a lock file +:class: tip +Update lock files frequently (e.g. weekly) but configure a dependency cooldown +of several days to avoid automatically installing the latest packages. Only +override the cooldown if a new package has a needed bug fix or security +patch. +::: + +::::{dropdown} Dependency cooldowns +:icon: info +:color: primary +[Dependency cooldowns]( +https://blog.pypi.org/posts/2026-04-02-incident-report-litellm-telnyx-supply-chain-attack/#dependency-cooldowns +) are strongly encouraged by security experts to avoid automatically downloading +the latest package updates that may have been compromised with malware. Package +manager tools are starting to support configurations for cooldowns +```sh +> uv lock --exclude-newer "3 days"` +``` +or in `pyproject.toml` +```toml +[tool.uv] +exclude-newer = "3 days" +``` + +Integrating cooldown constrained lock files into CI is important since this is +where new packages are commonly tested first. Automated testing code that +resolves `[project.dependencies]` every time +```sh +> python -m pip install . +``` +can be replaced with +```sh +> uv lock --exclude-newer "3 days"` +``` +or can be replaced with lock file based installations +```sh +> uv sync --frozen +``` +Support for this varies across automated testing frameworks (e.g. hatch, nox) so +consult their documentation for how to install dependencies from lock files with +dependency cooldowns. +:::: + +When you decide to update a lock file, make sure to test that the resulting +environment works before committing. If it fails because of some dependency +update, then it may be necessary to update `pyproject.toml` to cap the supported +versions of that dependency unless/until the code can be updated to support it. + +It can also be good, though not necessary, to double check what changed when +updating a lock file. The diff can be noisy so the main changes to focus on are +1) major version updates (e.g. `pandas 2.X.X` -> `pandas 3.X.X`) +2) new transitive dependencies (i.e. not part of your `pyproject.toml`) + +:::{tip} +A lock file captures one tested environment, not the full compatibility range +declared in `pyproject.toml`. Projects that use lock files should still have CI +test other environments such as + +1) the latest packages consistent with your `pyproject.toml`, subject to +dependency cooldowns. This lets you know if a dependency update breaks your +package. +2) older supported versions of Python to let you know if a recent change to your +package no longer works with an older Python release. + +::: + + +::::{dropdown} What about `requirements.txt` +:icon: info +:color: primary + +Older approaches to locking used `pip freeze` to generate a `requirements.txt` +that got used as a lock file. These are minimal lock files that pin a specific +version for the system on which the command was run. They might look like +``` +# requirements.txt +numpy==2.4.6 +plotly==6.7.0 +pyzmq==27.1.0 +``` + +However, this minimal level of specificity has several downsides making lock +files the preferred format: +* The versions satisfying `pyproject.toml` may differ between your Windows +laptop and the Linux server your CI runs on. A single lock file contains the +information needed to build platform specific and Python version specific +environments. In contrast, a separate `requirements.txt` files is needed to +store this information (e.g. `requirements.ci.txt`, +`requirements.py313-macos.txt`) +* Packages can get updated without a version update for both legitimate and +malicious reasons. Lock files include package hashes to catch this. A +[hash](https://en.wikipedia.org/wiki/Hash_function) is a unique signature +computed from the code and any change to the code +will cause the release to have a different hash even if is given the same +release version number. +* Other metadata determined during resolution of `pyproject.toml` (e.g. which +dependencies are transitive, where the packages were downloaded from, etc.) that +can help speed up future installs is lost. + +::::