Add packaging guide entry on dependency locking

[alexarmstrongvi]

Addresses Issue #491. First PR to the project. Open to feedback if I am missing general style and teaching philosophy or if it's preferred that this be shorter/longer.

[sneakers-the-rat]

maybe before the examples, we give a sense of the kinds of operations that need to be done in prose:

• create the lockfile
• update the lockfile
• update a single package in the lockfile
• creating different forms of the lockfile for different python versions, OSes, etc.

[alexarmstrongvi]

Changed: cd2c12f
Doesn't seem like PDM supports updating a single package in the lockfile like uv and poetry apart from some workaround using the --constraint arg so I left that out of the examples. I figure people will go to the linked docs if they really are going to use lock files and want to see tool specific workflows.

[sneakers-the-rat]

It does!

https://pdm-project.org/en/latest/reference/cli/#update_2

Positional Arguments:

packages: If packages are given, only update them

It looks like the links in the docs are a little wonky, where the narrative docs link to the wrong place in the CLI docs.

I'll double check this later when at keyboard

[alexarmstrongvi]

Ah. The docs said "Update package(s) in pyproject.toml" instead of in lock file so I skipped over that but I see now what it means. Added: bd49ceb

[sneakers-the-rat]

with dep cooldowns, i think only the 'too rare' case is a problem. I would probably frame this as "one risk when developing using lockfiles is that your lockfile will fall too far behind the most recent versions of your dependencies. When people install your package, they typically will not be using your lockfile, and will install the latest versions of the packages supported by their environment. If your dependencies update and break something you rely on, you might not notice it until someone reports it to you."

[sneakers-the-rat]

to be more specific, i basically think we should just say "you should use dependency cooldowns" as an unambiguous recommendation, but then on the other end give a bit of discussion to build intuition about why not updating frequently would be bad, which is the more common problem to have.

[alexarmstrongvi]

I agree "too rare" is the more likely situation. As far as falling behind on package updates, is there anything beyond bugfixes, security patches, and performance improvements that should be mentioned as reasons to update?

Regarding people installing your package and ignoring lock files (presumably for the case of a library), I intended the tip at the end about CI testing different environments to address this.

A discussion on dependency cooldowns seems best added into sections on building packages and CI as it's something that should be added to tool configs instead of invoked when generating a lock file

[sneakers-the-rat]

reasons to update

well the main one w.r.t. keeping lockfiles updated is detecting backwards incompatible API changes in the deps. Different reasons for updating for package consumers at install time vs. updating locked deps at Dev time, and different considerations for libs vs. Apps. But for the sake of this section I think we can give simple guidance like "use dep cooldowns and update & test lockfile frequently"

I intended the tip at the end about CI testing different environments to address this.

I'll take another full read later to see how reading flow goes, on phone rn

it's something that should be added to tool configs instead of invoked when generating a lock file

I'm not sure what you mean, dep cooldowns are something that can both be done by the consuming installer (ignoring lockfile) and configured for the lockfile resolver, and both should be done. Since goal here is to give intuition and "best practices" guidance, we should just say "you should configure your package to use dep cooldowns when locking" - I think all the major package managers support this now, I just pulled this into pdm a week or two ago

[alexarmstrongvi]

But for the sake of this section I think we can give simple guidance like "use dep cooldowns and update & test lockfile frequently"

Gotcha. Added a new recommendation box: 7fb2aa9

I'm not sure what you mean

I just meant I don't normally see people call sync with cooldown CLI args uv sync --exclude-newer "3 days" and instead put it in a user or project config

[tool.uv]
exclude-newer = "3 days"

so that its not something you have to remember every time you make/update lock files. In either case the recommendation to use cooldowns was added above.

[sneakers-the-rat]

I personally don't read the lockfile when it updates, and so we might not want to give the impression to newbies that it's normal to read lockfile changes since it can be daunting. My pattern is basically "update the lockfile, run the tests." maybe i'll keep a loose eye for transitive dependencies if e.g. something suddenly starts pulling in a huge dep like scipy or something. maybe the thing to communicate is like "what to do if you update and a new version breaks your stuff" and how to handle adding e.g. temporary version caps

[alexarmstrongvi]

Clarified its not necessary and that testing is more important: a1664bc

[sneakers-the-rat]

I like the changed language, when others get around to reviewing this I imagine there will be more said on version capping, which is a hot topic for us lol, but this is one case where it makes sense - the version range should reflect the range of versions that work for a given state of the package. Ideally one would just adapt to the changes in the upstream dep (and maybe scooch up the dep version floor if the changes aren't backwards compatible), but cutting a release with a version cap temporarily until you can make the fix so that it actually runs when people install it is a reasonable case for a cap.

[sneakers-the-rat]

I have done this, where i have one branch of the tests run with a pip install and the rest run with the lockfile as a sentinel, we might want to add a bit more scaffolding in the form of an example CI action for this. Usually I just to that on linux with latest python.

[alexarmstrongvi]

Seems good to reference the testing section here and then add an example there. To avoid the combinatoric explosion of maintaining guides on local vs CI testing, for various tools (e.g. nox, hatch, GitHub Actions), with and without lock files, what are your thoughts on making a high level point that testing can setup environments from a lock files instead of resolving at runtime, showing one example replacing run: python -m pip install ... with run: uv sync --frozen, and the linking to documentation for various tools?

[sneakers-the-rat]

Yeah, good point, agreed. Don't need a full CI example for this, having just one CI example for updating the lockfile is plenty for one page. Just having like a two line thing like

# install most recent dependencies compatible with [project.dependencies]
python -m pip install .

# install exact versions in lockfile
uv sync --frozen

As you say is probably plenty. I can imagine that helping generally with "wait when would I do one command vs the other" and "what does pip do/what does uv do" confusion as well.

[alexarmstrongvi]

Added a dropdown box giving more detail on cooldowns: 69d49dd

I am not familiar with hatch and nox workflows but after a quick search I wasn't seeing how to enable cooldowns or just calling uv instead of pip. The most recent pip releases have a PIP_UPLOADED_PRIOR_TO env var that can be used so perhaps that is the solution to recommend for the Tests section.

[sneakers-the-rat]

I think right above this we need a section like "why are lockfiles good" or "why do they exist" to orient people to what they are doing and why before getting into how to work with them.

something like "lockfiles create a predictable development environment that helps reduce problems where code runs on one person's machine but crashes on another. together with CI result logs, they are a versioned record of the exact set of code that passed or failed the tests. for applications (as opposed to libraries) that are intended to be used as-is rather than depended on by other packages, they allow someone to install and run it and be confident that it will work, without accidentally installing a more recent version of some dependency that is incompatible"

[alexarmstrongvi]

Added more motivation for lock files in line with your points in the intro paragraph: 88d6b78

[sneakers-the-rat]

platform and python version deps are the main things that my lockfiles vary by: maybe generalized like "lockfiles can be customized to include all the different versions that are applicable across different operating systems, python versions, and other platform markers, while that information can only be stored in requirements.txt files through file naming conventions."

idk that text isn't very clear either, but something like that

[alexarmstrongvi]

Clarified: 88f7445

[sneakers-the-rat]

nice, thank you for drafting this! we have needed this for sure. i think the main thing that's TODO here is providing a recommended workflow, getting a lockfile updating setup can have some weird pitfalls that aren't obvious. i am pretty bad about doing this across all my packages because usually i am very stingy with deps in the first place, and otherwise will be touching them frequently enough that i relock, but the thing that i end up doing is doing a weekly automated PR that just updates the lockfiles and runs the tests. check for any added dependencies, if no new deps, merge it if tests pass. if there are new deps, list them in the PR.

I am not sure the current state of the tooling but the final straw for me that pushed me to switch away from using poetry was when i requested and drafted a command that would just print the deps that would change from a lock update, but they didn't want to do that for some reason, so i just ended up writing some heinous bash scripts for it, but if it's more reasonable now we should include an example CI workflow for that. Not saying mine is correct or the best, so that can look like anything, but i think that's sort of the major thing that is a pain in the ass about lockfiles and so would be one of the main things we want to provide in the guide.

[sneakers-the-rat]

Suggested change

	malicious reasons. Lock files include package hashes to catch this. A hash
	malicious reasons. Lock files include package hashes to catch this. A [hash](https://en.wikipedia.org/wiki/Hash_function)

[alexarmstrongvi]

Added: 42d5c8a83bbc0a303c1bd351bcf405e26ca41188

[sneakers-the-rat]

Suggested change

	number is a unique signature computed from the code and any change to the code
	is a unique signature computed from the code and any change to the code

(Sorry can't select multiple lines on mobile)

[alexarmstrongvi]

Removed: 42d5c8a83bbc0a303c1bd351bcf405e26ca41188