Dependency management improvements (#188)

* Correctly use a constraints file

Turns out I've been using the constraint file incorrectly for years. I
love learning moments like this!

The `constraints.txt` file **is** the lockfile for the project. It
contains all the versions that *should* be used *if* they are needed. Is
is not the source of **what** should be installed, only what version.

So, instead of having N different `requirements*.in` files with N
different `requirements*.txt` generated from `pip-compile`; we have a
single `constraints.txt` that all of our `requirements*.txt` files
reference. Glorious.

This solves for the scaling problem that I ran into when, for example,
`requirements-doc.txt` had a version conflicct with something in
`requirements-test.txt`. Initially, I thought to somehow cross-reference
all the `.txt` files with themselves in some horrible chained constraint
system. That worked about as well as it sounds.

Note, I have to remove the dynamic dependencies here from the
`pyproject.toml`. That's fine. Eventually, I'll be moving to `uv` which
simplifies most of this juggling. For now, however, I want to understand
the underlying machanics before abstracting them away.

* Improve noxfile for dev install and management

Moving toward simplicity of use, the `noxfile.txt` now has a session
called `dev`. This session creates, if needed, the `.venv` and installs
all development requirements.

Additionally, the `update` and `upgrade` sessions have been renamed to
`update-deps` and `upgrade-deps` respectively. They now use the new,
awesome pattern of a single constraint file for dependencies.

Finally, added the `--no-annotations` to the `pip-compile` commands and
what a difference that makes! I have yet to understand when and why
`pip-compile` will expand `requirements/requirements.txt` to
`/home/preocts/project-name/requirements/requirements.txt`. In practice,
the results are different depending on the OS being used as well. Since
it all lead to noisy diffs without much value, removing it seems ideal!

* Correct nox sessions for CI actions

Okay, this is the ugly part of the path I'm taking. It is now required
to install the correct requirement*.txt files for each session.
Previously it was a simple `.[test]` reference when installing the
package.

This too shall pass.

Already thinking about pivoting into listing all requirements in the
`pyproject.toml`. With dependency groups being supported this should be
amazing. It will hindge on `pip-compile` being able to extract from
dependency groups. That's the next PR!

* Update documenation

Adjust the contributing docs with the new expected nox workflows. Add
some leading links at the top of the readme pointing to the secondary
files.

Reading, strangely, isn't the strongest skill of many.

Author: Preocts

CONTRIBUTING.md

@@ -35,7 +35,6 @@ prior to submitting a pull request.
   - No test should be dependent on another
   - No test should be dependent on secrets/tokens
 
-
 ---
 
 # Local developer installation
@@ -52,81 +51,23 @@ git clone https://github.com/[ORG NAME]/[REPO NAME]
 cd [REPO NAME]
 ```
 
-### Virtual Environment
-
-Use a ([`venv`](https://docs.python.org/3/library/venv.html)), or equivalent,
-when working with python projects. Leveraging a `venv` will ensure the installed
-dependency files will not impact other python projects or any system
-dependencies.
-
-**Windows users**: Depending on your python install you will use `py` in place
-of `python` to create the `venv`.
-
-**Linux/Mac users**: Replace `python`, if needed, with the appropriate call to
-the desired version while creating the `venv`. (e.g. `python3` or `python3.12`)
-
-**All users**: Once inside an active `venv` all systems should allow the use of
-`python` for command line instructions. This will ensure you are using the
-`venv`'s python and not the system level python.
-
-### Create the `venv`:
-
-```console
-python -m venv .venv
-```
-
-Activate the `venv`:
-
-```console
-# Linux/Mac
-. .venv/bin/activate
-
-# Windows
-.venv\Scripts\activate
-```
+### [Install nox](https://nox.thea.codes/en/stable/index.html)
 
-The command prompt should now have a `(venv)` prefix on it. `python` will now
-call the version of the interpreter used to create the `venv`
+It is recommended to use a tool such as `pipx` or `uvx` to install nox. nox is
+needed to run the provided sessions for developer setup, linting, tests, and
+dependency management. It is optional, but these instructions will not cover
+manually steps outside of nox.
 
-To deactivate (exit) the `venv`:
 
-```console
-deactivate
-```
-
----
+## Nox Sessions
 
-## Developer Installation Steps
+### Developer Install
 
-### Install editable library and development requirements
+This builds the `/.venv`, installs the editable
+package, and installs all dependency files.
 
-```console
-python -m pip install --editable .[dev,test]
-```
-
-### Install pre-commit [(see below for details)](#pre-commit)
-
-```console
-pre-commit install
-```
-
-### Install with nox
-
-If you have `nox` installed with `pipx` or in the current venv you can use the
-following session. This is an alternative to the two steps above.
-
-```console
-nox -s install
-```
-
----
-
-## Pre-commit and nox tools
-
-### Run pre-commit on all files
-
-```console
-pre-commit run --all-files
+```bash
+nox -s dev
 ```
 
 ### Run tests with coverage (quick)
@@ -151,23 +92,23 @@ nox -e build
 
 ## Updating dependencies
 
-New dependencys can be added to the `requirements-*.in` file. It is recommended
+New dependencys can be added to the `requirements-*.txt` file. It is recommended
 to only use pins when specific versions or upgrades beyond a certain version are
 to be avoided. Otherwise, allow `pip-compile` to manage the pins in the
-generated `requirements-*.txt` files.
+generated `constraints.txt` file.
 
 Once updated following the steps below, the package can be installed if needed.
 
 ### Update the generated files with changes
 
 ```console
-nox -e update
+nox -s update-deps
 ```
 
 ### Upgrade all generated dependencies
 
 ```console
-nox -e upgrade
+nox -s upgrade-deps
 ```
 
 ---

README.md

@@ -7,6 +7,11 @@
 
 # python-src-template
 
+- [Contributing Guide and Developer Setup Guide](./CONTRIBUTING.md)
+- [License: MIT](./LICENSE)
+
+---
+
 A template I use for most projects and is setup to jive with my environment at
 the company I work with.
 

noxfile.py

@@ -11,13 +11,11 @@
 MODULE_NAME = "module_name"
 TESTS_PATH = "tests"
 COVERAGE_FAIL_UNDER = 50
-DEFAULT_PYTHON_VERSION = "3.12"
+DEFAULT_PYTHON = "3.12"
 PYTHON_MATRIX = ["3.9", "3.10", "3.11", "3.12", "3.13"]
 VENV_BACKEND = "venv"
 VENV_PATH = ".venv"
-REQUIREMENT_IN_FILES = [
-    pathlib.Path("requirements/requirements.in"),
-]
+REQUIREMENTS_PATH = "./requirements"
 
 # What we allowed to clean (delete)
 CLEANABLE_TARGETS = [
@@ -43,35 +41,72 @@
 ]
 
 
+@nox.session(python=False)
+def dev(session: nox.Session) -> None:
+    """Setup a development environment by creating the venv and installs dependencies."""
+    # Use the active environement if it exists, otherwise create a new one
+    venv_path = os.environ.get("VIRTUAL_ENV", VENV_PATH)
+
+    if sys.platform == "win32":
+        py_command = "py"
+        venv_path = f"{venv_path}/Scripts"
+        activate_command = f"{venv_path}/activate"
+    else:
+        py_command = f"python{DEFAULT_PYTHON}"
+        venv_path = f"{venv_path}/bin"
+        activate_command = f"source {venv_path}/activate"
+
+    if not os.path.exists(VENV_PATH):
+        session.run(py_command, "-m", "venv", VENV_PATH, "--upgrade-deps")
+
+    python = f"{venv_path}/python"
+    requirement_files = get_requirement_files()
+
+    session.run(python, "-m", "pip", "install", "-e", ".")
+    for requirement_file in requirement_files:
+        session.run(python, "-m", "pip", "install", "-r", requirement_file)
+
+    session.run(python, "-m", "pip", "install", "pre-commit")
+    session.run(f"{venv_path}/pre-commit", "install")
+
+    if not os.environ.get("VIRTUAL_ENV"):
+        session.log(f"\n\nRun '{activate_command}' to enter the virtual environment.\n")
+
+
 @nox.session(python=PYTHON_MATRIX, venv_backend=VENV_BACKEND)
 def version_coverage(session: nox.Session) -> None:
     """Run unit tests with coverage saved to partial file."""
     print_standard_logs(session)
 
-    session.install(".[test]")
+    session.install(".")
+    session.install("-r", "requirements/requirements.txt")
+    session.install("-r", "requirements/requirements-test.txt")
     session.run("coverage", "run", "-p", "-m", "pytest", TESTS_PATH)
 
 
-@nox.session(python=DEFAULT_PYTHON_VERSION, venv_backend=VENV_BACKEND)
+@nox.session(python=DEFAULT_PYTHON, venv_backend=VENV_BACKEND)
 def coverage_combine(session: nox.Session) -> None:
     """Combine all coverage partial files and generate JSON report."""
     print_standard_logs(session)
 
     fail_under = f"--fail-under={COVERAGE_FAIL_UNDER}"
 
-    session.install(".[test]")
+    session.install(".")
+    session.install("-r", "requirements/requirements.txt")
+    session.install("-r", "requirements/requirements-test.txt")
     session.run("python", "-m", "coverage", "combine")
     session.run("python", "-m", "coverage", "report", "-m", fail_under)
     session.run("python", "-m", "coverage", "json")
 
 
-@nox.session(python=DEFAULT_PYTHON_VERSION, venv_backend=VENV_BACKEND)
+@nox.session(python=DEFAULT_PYTHON, venv_backend=VENV_BACKEND)
 def mypy(session: nox.Session) -> None:
     """Run mypy against package and all required dependencies."""
     print_standard_logs(session)
 
     session.install(".")
-    session.install("mypy")
+    session.install("-r", "requirements/requirements.txt")
+    session.install("-r", "requirements/requirements-dev.txt")
     session.run("mypy", "-p", MODULE_NAME, "--no-incremental")
 
 
@@ -83,7 +118,7 @@ def coverage(session: nox.Session) -> None:
     session.run("coverage", "report", "-m")
 
 
-@nox.session(python=DEFAULT_PYTHON_VERSION, venv_backend=VENV_BACKEND)
+@nox.session(python=DEFAULT_PYTHON, venv_backend=VENV_BACKEND)
 def build(session: nox.Session) -> None:
     """Build distribution files."""
     print_standard_logs(session)
@@ -92,49 +127,41 @@ def build(session: nox.Session) -> None:
     session.run("python", "-m", "build")
 
 
-@nox.session(python=False, venv_backend=VENV_BACKEND)
-def install(session: nox.Session) -> None:
-    """Setup a development environment. Uses active venv if available, builds one if not."""
-    # Use the active environement if it exists, otherwise create a new one
-    venv_path = os.environ.get("VIRTUAL_ENV", VENV_PATH)
-
-    if sys.platform == "win32":
-        py_command = "py"
-        venv_path = f"{venv_path}/Scripts"
-        activate_command = f"{venv_path}/activate"
-    else:
-        py_command = f"python{DEFAULT_PYTHON_VERSION}"
-        venv_path = f"{venv_path}/bin"
-        activate_command = f"source {venv_path}/activate"
-
-    if not os.path.exists(VENV_PATH):
-        session.run(py_command, "-m", "venv", VENV_PATH, "--upgrade-deps")
-
-    session.run(f"{venv_path}/python", "-m", "pip", "install", "-e", ".[dev,test]")
-    session.run(f"{venv_path}/pre-commit", "install")
-
-    if not venv_path:
-        session.log(f"\n\nRun '{activate_command}' to enter the virtual environment.\n")
-
-
-@nox.session(python=DEFAULT_PYTHON_VERSION, venv_backend=VENV_BACKEND)
-def update(session: nox.Session) -> None:
-    """Process requirement*.in files, updating only additions/removals."""
+@nox.session(python=DEFAULT_PYTHON, venv_backend=VENV_BACKEND, name="update-deps")
+def update_deps(session: nox.Session) -> None:
+    """Process requirement*.txt files, updating only additions/removals."""
     print_standard_logs(session)
 
-    session.install("pip-tools")
-    for filename in REQUIREMENT_IN_FILES:
-        session.run("pip-compile", "--no-emit-index-url", str(filename))
-
+    requirement_files = get_requirement_files()
 
-@nox.session(python=DEFAULT_PYTHON_VERSION, venv_backend=VENV_BACKEND)
-def upgrade(session: nox.Session) -> None:
-    """Process requirement*.in files and upgrade all libraries as possible."""
+    session.install("pip-tools")
+    session.run(
+        "pip-compile",
+        "--no-annotate",
+        "--no-emit-index-url",
+        "--output-file",
+        f"{REQUIREMENTS_PATH}/constraints.txt",
+        *requirement_files,
+    )
+
+
+@nox.session(python=DEFAULT_PYTHON, venv_backend=VENV_BACKEND, name="upgrade-deps")
+def upgrade_deps(session: nox.Session) -> None:
+    """Process requirement*.txt files and upgrade all libraries as possible."""
     print_standard_logs(session)
 
+    requirement_files = get_requirement_files()
+
     session.install("pip-tools")
-    for filename in REQUIREMENT_IN_FILES:
-        session.run("pip-compile", "--no-emit-index-url", "--upgrade", str(filename))
+    session.run(
+        "pip-compile",
+        "--no-annotate",
+        "--no-emit-index-url",
+        "--upgrade",
+        "--output-file",
+        f"{REQUIREMENTS_PATH}/constraints.txt",
+        *requirement_files,
+    )
 
 
 @nox.session(python=False, venv_backend=VENV_BACKEND)
@@ -157,3 +184,9 @@ def print_standard_logs(session: nox.Session) -> None:
     version = session.run("python", "--version", silent=True)
     session.log(f"Running from: {session.bin}")
     session.log(f"Running with: {version}")
+
+
+def get_requirement_files() -> list[pathlib.Path]:
+    """Get a list of requirement files matching "requirements*.txt"."""
+    glob = pathlib.Path(REQUIREMENTS_PATH).glob("requirements*.txt")
+    return [path for path in glob]

pyproject.toml

@@ -19,7 +19,7 @@ classifiers = [
     "Programming Language :: Python :: 3 :: Only",
     "Programming Language :: Python :: Implementation :: CPython",
 ]
-dynamic = ["dependencies", "optional-dependencies", "version"]
+dynamic = ["version"]
 
 [project.urls]
 homepage = "https://github.com/[ORG NAME]/[REPO NAME]"
@@ -30,13 +30,6 @@ homepage = "https://github.com/[ORG NAME]/[REPO NAME]"
 [tool.setuptools_scm]
 # Purposely left empty
 
-[tool.setuptools.dynamic.dependencies]
-file = ["requirements/requirements.txt"]
-
-[tool.setuptools.dynamic.optional-dependencies]
-dev = { file = ["requirements/requirements-dev.txt"] }
-test = { file = ["requirements/requirements-test.txt"] }
-
 [tool.black]
 line-length = 100
 target-version = ['py39']

requirements/constraints.txt

@@ -0,0 +1,31 @@
+#
+# This file is autogenerated by pip-compile with Python 3.12
+# by the following command:
+#
+#    pip-compile --no-annotate --no-emit-index-url --output-file=./requirements/constraints.txt requirements/requirements-dev.txt requirements/requirements-test.txt requirements/requirements.txt
+#
+black==25.1.0
+certifi==2025.4.26
+charset-normalizer==3.4.2
+click==8.2.1
+coverage==7.8.2
+flake8==7.2.0
+flake8-builtins==2.5.0
+flake8-pep585==0.1.7
+idna==3.10
+iniconfig==2.1.0
+isort==6.0.1
+mccabe==0.7.0
+mypy==1.15.0
+mypy-extensions==1.1.0
+packaging==25.0
+pathspec==0.12.1
+platformdirs==4.3.8
+pluggy==1.6.0
+pycodestyle==2.13.0
+pyflakes==3.3.2
+pytest==8.3.5
+pytest-randomly==3.16.0
+requests==2.32.3
+typing-extensions==4.13.2
+urllib3==2.4.0