Repository navigation
build: make the database drivers optional extras - #1628
Open
dimitri-yatsenko wants to merge 1 commit into
Open
dimitri-yatsenko wants to merge 1 commit into
dimitri-yatsenko wants to merge 1 commit into
Conversation
pymysql was a core dependency, so every install of datajoint pulled in the MySQL driver whether or not it was used, while the PostgreSQL driver was already the optional postgres extra. Both drivers are now extras: pip install "datajoint[postgres]" installs psycopg2-binary, and pip install "datajoint[mysql]" installs pymysql, with the same >=0.7.2 floor as before. A bare pip install datajoint installs no database driver. import datajoint no longer imports a driver. Each adapter imports its driver when it is created, which happens when a connection is made, through a new helper, import_driver, in adapters/base.py. When the driver is missing, connecting raises a DataJointError that names the extra, for example: The MySQL backend requires the pymysql driver, which is not installed. Install it with: pip install "datajoint[mysql]". The PostgreSQL adapter raised ImportError for its missing driver; it now raises the same DataJointError. The adapter imports the psycopg2 module, which both the psycopg2 and the psycopg2-binary distributions provide, so either one satisfies it. The test dependency group and the test and dev extras install both drivers. The README install line, CONTRIBUTING.md, the development Docker image (it now installs both extras), and the conda-forge notes in RELEASE_MEMO.md (conda has no extras, so the recipe lists both drivers) are updated to match. Migration: an environment that connects to MySQL needs pip install "datajoint[mysql]", or pymysql installed some other way; one that connects to PostgreSQL needs pip install "datajoint[postgres]". An upgrade in place keeps a pymysql that is already installed, but a fresh install from a requirements file that lists only datajoint gets no driver. Code that caught ImportError from PostgreSQLAdapter() should catch DataJointError instead. Tests: test_lazy_imports.py checks, in a fresh interpreter, that import datajoint loads neither pymysql nor psycopg2. test_adapters.py blocks each driver in sys.modules and checks that get_adapter and Connection raise a DataJointError naming the right extra before any server is contacted, and that each backend needs only its own driver.
dimitri-yatsenko
requested review from
MilagrosMarin,
lum-agilyti and
ttngu207
October 17, 2026 23:49
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1626.
Summary
pymysqlis no longer a core dependency. Each database driver is an optional extra, andimport datajointloads no driver:datajoint[postgres]psycopg2-binary>=2.9.0(unchanged)datajoint[mysql]pymysql>=0.7.2(new; same floor as the former core dependency)A bare
pip install datajointinstalls no database driver.Changes
pymysqlmoves from[project].dependenciesto the newmysqlextra. Thetestdependency group and thetestanddevextras list both drivers.import_driverinadapters/base.py, imports a driver when its adapter is created, which happens on connect. A missing driver raises aDataJointErrorthat names the extra:adapters/mysql.pyno longer importspymysqlat module load.adapters/postgres.pydrops its module-leveltry: import psycopg2(and the unusedpsycopg2.sqlimport). Its adapter raisedImportErrorfor a missing driver and now raises the sameDataJointError.psycopg2module, which both distributions provide, so either one satisfies it.datajoint[postgres]anddatajoint[mysql]./main[postgres,mysql].run.__init__.py.Migration
pip install "datajoint[mysql]", ordatajoint[mysql]in a requirements file.pip install "datajoint[postgres]", as before.pymysqlthat is already installed. A fresh environment built from a requirements file that lists onlydatajointhas no driver, and the first connection names the extra to install.ImportErrorfromPostgreSQLAdapter()should catchDataJointError.Tests
tests/unit/test_lazy_imports.py: in a fresh interpreter,import datajointloads neitherpymysqlnorpsycopg2.tests/unit/test_adapters.py::TestMissingDriver: blocks each driver insys.modules, then checks two things:get_adapterandConnection(...)raise aDataJointErrornaming the right extra, before any server is contacted.import datajointsucceeds with no driver loaded.dj.conn()raises... pip install "datajoint[mysql]"(or[postgres]withDJ_BACKEND=postgresql).[mysql]: installs PyMySQL only.dj.conn()reaches the MySQL driver, and psycopg2 is never imported.[postgres]: installs psycopg2-binary only.dj.conn()reaches psycopg2, and pymysql is never imported.Notes for review
pixi.lockis not regenerated in this PR. Re-solving it changes no package versions in the existingdefault,dev, andtestenvironments. The only changes are thatdefaultdropspymysqland gainspackaging, which the committed lock was already missing. The rest of the re-solve adds thetest-py310/test-py314environments, which the open Python 3.15 build PR (build: support Python 3.15 #1596) also adds.pixi.lockmerges as binary, so regenerating it in both PRs would conflict. CI runs withlocked: false.datajoint[mysql]installs plainpymysql. Withoutcryptography, pymysql cannot completecaching_sha2_passwordauthentication over a non-TLS connection. This was equally true when pymysql was a core dependency.pymysql[rsa]would addcryptography.