Skip to content

build: make the database drivers optional extras - #1628

Open
dimitri-yatsenko wants to merge 1 commit into
masterfrom
build/drivers-optional-extras
Open

dimitri-yatsenko wants to merge 1 commit into
masterfrom
build/drivers-optional-extras

Conversation

@dimitri-yatsenko

Copy link
Copy Markdown
Member

Closes #1626.

Summary

pymysql is no longer a core dependency. Each database driver is an optional extra, and import datajoint loads no driver:

Extra Installs
datajoint[postgres] psycopg2-binary>=2.9.0 (unchanged)
datajoint[mysql] pymysql>=0.7.2 (new; same floor as the former core dependency)

A bare pip install datajoint installs no database driver.

Changes

  • pyproject.toml. pymysql moves from [project].dependencies to the new mysql extra. The test dependency group and the test and dev extras list both drivers.
  • Lazy driver import. A new helper, import_driver in adapters/base.py, imports a driver when its adapter is created, which happens on connect. A missing driver raises a DataJointError that names the extra:
    The MySQL backend requires the pymysql driver, which is not installed. Install it with: pip install "datajoint[mysql]"
    
    adapters/mysql.py no longer imports pymysql at module load. adapters/postgres.py drops its module-level try: import psycopg2 (and the unused psycopg2.sql import). Its adapter raised ImportError for a missing driver and now raises the same DataJointError.
  • psycopg2 vs psycopg2-binary. The adapter imports the psycopg2 module, which both distributions provide, so either one satisfies it.
  • Other places that assumed pymysql is installed.
    • README install line: now shows datajoint[postgres] and datajoint[mysql].
    • CONTRIBUTING.md: the backend section lists both extras.
    • Development Dockerfile: installs /main[postgres,mysql].
    • RELEASE_MEMO.md: conda has no extras, so the feedstock recipe lists both drivers under run.
    • The lazy-import comment in __init__.py.

Migration

  • MySQL: pip install "datajoint[mysql]", or datajoint[mysql] in a requirements file.
  • PostgreSQL: pip install "datajoint[postgres]", as before.
  • An upgrade in place keeps a pymysql that is already installed. A fresh environment built from a requirements file that lists only datajoint has no driver, and the first connection names the extra to install.
  • Code that caught ImportError from PostgreSQLAdapter() should catch DataJointError.

Tests

  • tests/unit/test_lazy_imports.py: in a fresh interpreter, import datajoint loads neither pymysql nor psycopg2.
  • tests/unit/test_adapters.py::TestMissingDriver: blocks each driver in sys.modules, then checks two things:
    • get_adapter and Connection(...) raise a DataJointError naming the right extra, before any server is contacted.
    • Each backend needs only its own driver.
  • Full suite (testcontainers, local): 1166 passed, 14 skipped.
  • Clean virtual environments, each with an editable install of this branch:
    • Bare install: no driver installed. import datajoint succeeds with no driver loaded. dj.conn() raises ... pip install "datajoint[mysql]" (or [postgres] with DJ_BACKEND=postgresql).
    • [mysql]: installs PyMySQL only. dj.conn() reaches the MySQL driver, and psycopg2 is never imported.
    • [postgres]: installs psycopg2-binary only. dj.conn() reaches psycopg2, and pymysql is never imported.

Notes for review

  • pixi.lock is not regenerated in this PR. Re-solving it changes no package versions in the existing default, dev, and test environments. The only changes are that default drops pymysql and gains packaging, which the committed lock was already missing. The rest of the re-solve adds the test-py310/test-py314 environments, which the open Python 3.15 build PR (build: support Python 3.15 #1596) also adds. pixi.lock merges as binary, so regenerating it in both PRs would conflict. CI runs with locked: false.
  • datajoint[mysql] installs plain pymysql. Without cryptography, pymysql cannot complete caching_sha2_password authentication over a non-TLS connection. This was equally true when pymysql was a core dependency. pymysql[rsa] would add cryptography.

pymysql was a core dependency, so every install of datajoint pulled in the
MySQL driver whether or not it was used, while the PostgreSQL driver was
already the optional postgres extra. Both drivers are now extras:
pip install "datajoint[postgres]" installs psycopg2-binary, and
pip install "datajoint[mysql]" installs pymysql, with the same >=0.7.2 floor
as before. A bare pip install datajoint installs no database driver.

import datajoint no longer imports a driver. Each adapter imports its driver
when it is created, which happens when a connection is made, through a new
helper, import_driver, in adapters/base.py. When the driver is missing,
connecting raises a DataJointError that names the extra, for example: The
MySQL backend requires the pymysql driver, which is not installed. Install it
with: pip install "datajoint[mysql]". The PostgreSQL adapter raised
ImportError for its missing driver; it now raises the same DataJointError.
The adapter imports the psycopg2 module, which both the psycopg2 and the
psycopg2-binary distributions provide, so either one satisfies it.

The test dependency group and the test and dev extras install both drivers.
The README install line, CONTRIBUTING.md, the development Docker image (it
now installs both extras), and the conda-forge notes in RELEASE_MEMO.md
(conda has no extras, so the recipe lists both drivers) are updated to match.

Migration: an environment that connects to MySQL needs
pip install "datajoint[mysql]", or pymysql installed some other way; one
that connects to PostgreSQL needs pip install "datajoint[postgres]". An
upgrade in place keeps a pymysql that is already installed, but a fresh
install from a requirements file that lists only datajoint gets no driver.
Code that caught ImportError from PostgreSQLAdapter() should catch
DataJointError instead.

Tests: test_lazy_imports.py checks, in a fresh interpreter, that
import datajoint loads neither pymysql nor psycopg2. test_adapters.py blocks
each driver in sys.modules and checks that get_adapter and Connection raise
a DataJointError naming the right extra before any server is contacted, and
that each backend needs only its own driver.
@dimitri-yatsenko dimitri-yatsenko added this to the v2.4 milestone Oct 17, 2026
@dimitri-yatsenko dimitri-yatsenko added the breaking Not backward compatible changes label Oct 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking Not backward compatible changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Database drivers become optional extras

1 participant