Skip to content

fix(settings): environment variables beat datajoint.json for every setting - #1603

Open
dimitri-yatsenko wants to merge 1 commit into
masterfrom
fix/env-var-precedence
Open

dimitri-yatsenko wants to merge 1 commit into
masterfrom
fix/env-var-precedence

Conversation

@dimitri-yatsenko

Copy link
Copy Markdown
Member

The module documents environment variables as taking precedence over datajoint.json, but _update_from_flat_dict enforced that only for keys in the hand-kept ENV_VAR_MAPPING. For every other setting the file replaced the value read from the environment: jobs.* (DJ_JOBS_*), provenance.* (DJ_PROVENANCE_*), database.use_tls, database.reconnect, and the top-level safemode, query_cache, download_path and ignore_config_file. A deployment could not, for example, turn off safemode or provenance capture for one worker when the project's file set it.

ENV_VAR_MAPPING is now derived from the settings models: a field's variable is its validation_alias, otherwise its model's env_prefix plus the field name. A setting added later is covered as soon as it is declared, including one that has only a validation_alias. Models without an env_prefix (connection, display) contribute only their aliased fields, matching the documented variables, and stores stays with _apply_stores_env, which already runs after the file. The eleven names in the old mapping are unchanged.

The tests list every setting that reads an environment variable, assert the derived mapping matches that list, and check for each one, in both flat and nested file layouts, that pydantic reads the variable and the file does not replace it.

Closes #1585.

…tting

The module documents environment variables as taking precedence over `datajoint.json`, but `_update_from_flat_dict` enforced that only for keys in the hand-kept `ENV_VAR_MAPPING`. For every other setting the file replaced the value read from the environment: `jobs.*` (`DJ_JOBS_*`), `provenance.*` (`DJ_PROVENANCE_*`), `database.use_tls`, `database.reconnect`, and the top-level `safemode`, `query_cache`, `download_path` and `ignore_config_file`. A deployment could not, for example, turn off `safemode` or provenance capture for one worker when the project's file set it.

`ENV_VAR_MAPPING` is now derived from the settings models: a field's variable is its `validation_alias`, otherwise its model's `env_prefix` plus the field name. A setting added later is covered as soon as it is declared, including one that has only a `validation_alias`. Models without an `env_prefix` (`connection`, `display`) contribute only their aliased fields, matching the documented variables, and `stores` stays with `_apply_stores_env`, which already runs after the file. The eleven names in the old mapping are unchanged.

The tests list every setting that reads an environment variable, assert the derived mapping matches that list, and check for each one, in both flat and nested file layouts, that pydantic reads the variable and the file does not replace it.

Closes #1585.
@dimitri-yatsenko dimitri-yatsenko added this to the v2.4 milestone Oct 17, 2026
@dimitri-yatsenko dimitri-yatsenko added the bug Indicates an unexpected problem or unintended behavior label Oct 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Indicates an unexpected problem or unintended behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Environment variables lose to datajoint.json for settings missing from ENV_VAR_MAPPING

1 participant