Databricks Python UDF library template

A working project layout for a Python library that powers SQL functions in Unity Catalog — plus the caveats that bite you in production.

The idea in one line: the UDF body is a shim, the wheel is the library.

A Unity Catalog Python UDF body isn't a module. For scalar UDFs, the $$...$$ block is the function body — you write a bare return with no def. Logic written there can't be unit tested, can't be versioned, and can't be reused. So push everything into a wheel on a UC volume and keep the SQL to two lines:

CREATE OR REPLACE FUNCTION main.text_fns.mask_email(email STRING)
RETURNS STRING
LANGUAGE PYTHON
DETERMINISTIC
COMMENT 'acme_udf 1.0.0 - masks the local part of an email, preserves domain.'
ENVIRONMENT (
  dependencies = '["/Volumes/main/udf_libs/wheels/acme_udf/1.0.0/acme_udf-1.0.0-py3-none-any.whl"]',
  environment_version = '3'
)
AS $$
from acme_udf.text import mask_email
return mask_email(email)
$$;

Important

Unity Catalog UDFs and custom dependencies are both Public Preview. Fine for internal use — flag it before putting a customer's critical path on it.

Layout

├── pyproject.toml                  # name, version, deps — the single source of version truth
├── src/acme_udf/
│   ├── __init__.py                 # re-export entrypoints ONLY; keep imports cheap
│   ├── text.py                     # pure scalar functions
│   ├── tokenize.py                 # scalar + pandas Series form for batch UDFs
│   └── _data/                      # package data, read via importlib.resources
├── tests/                          # plain pytest — no Spark, no Databricks
├── udfs/                           # one .sql template per function = deployable contract
├── scripts/
│   ├── render_ddl.py               # injects version + volume path into the templates
│   └── deploy_udfs.py              # runs CREATE OR REPLACE via Statement Execution API
├── databricks.yml                  # builds the wheel, uploads it to the volume
└── .github/workflows/              # ci: test+build+lint | release: deploy+register+smoke

Volume layout — version in the path, never overwrite:

/Volumes/<catalog>/udf_libs/wheels/acme_udf/1.0.0/acme_udf-1.0.0-py3-none-any.whl

Put UDFs in their own catalog/schema with their own grants. Databricks recommends this explicitly, and it keeps EXECUTE grants separate from data grants.

Quickstart

pip install -e '.[dev]'
pytest

# Render DDL from pyproject version + your volume path
python scripts/render_ddl.py \
    --catalog main --schema text_fns \
    --volume /Volumes/main/udf_libs/wheels \
    --out build/ddl

python scripts/deploy_udfs.py --dry-run --ddl-dir build/ddl   # inspect
python scripts/deploy_udfs.py --profile my-ws \
    --warehouse-id <id> --ddl-dir build/ddl                   # register

Then grant access — both of these, see the gotcha below:

GRANT EXECUTE ON FUNCTION main.text_fns.mask_email TO `data_users`;
GRANT READ VOLUME ON VOLUME main.udf_libs.wheels TO `data_users`;

Caveats

Packaging

• Architecture is not stable. A serverless SQL warehouse runs on aarch64 or x86_64, and it can change between restarts. A wheel with native C extensions built for one arch fails on the other with ISOLATION_ENVIRONMENT_USER_ERROR.GENERIC. Ship both variants gated by platform_machine markers — render_ddl.py --native emits exactly that. Strongly prefer pure-Python (py3-none-any) and make it a design constraint; CI asserts it.
• environment_version = '3' pins Python 3.12.3 and a fixed preinstalled package set, independent of the DBR. Build against that interpreter. Values '3'+ work only on serverless compute and serverless SQL warehouses — on classic compute pass --environment-version None.
• Never list pyspark or databricks-connect as dependencies; they collide with the sandbox runtime. CI asserts this too.
• Keep the dependency closure small. Deps resolve when the isolation environment is built, so every transitive package is cold-start latency on the first invocation.
• Immutable filenames. Never reupload a different build to the same wheel path — bump the version and re-run CREATE OR REPLACE. Overwriting in place leaves you no way to reason about what a warehouse actually loaded.

Library design, constrained by the sandbox

• No filesystem, no SparkContext, no internal services. The library can't read config files, write temp files, or call back into Spark. Pure functions only; bundle lookup data as package data and read it with importlib.resources (see tokenize.py).
• Module-level mutable state is a trap. Since DBR 18.0 (scalar) / 17.1 (batch), UDFs with the same owner + session share an isolation environment by default. If a module mutates globals, writes files, touches env vars, or evals input, you must add STRICT ISOLATION — otherwise you get cross-UDF interference. Better: design so you never need it. Module-level immutable state (compiled regexes, loaded tables) is correct and desirable — it's paid once per environment, not once per row.
• Import cost is per-environment. Do expensive one-time init at module scope, not inside the called function.

Runtime limits

• Max 5 UDFs per query. A hard planner limit that shapes how you factor functions: one enrich_record returning a STRUCT beats five single-field UDFs. Heavy plans can also hit UDF_MAX_COUNT_EXCEEDED.
• NULLs are yours to handle. No implicit null propagation — guard every input. Every function here returns None for None.
• Scalar returns only for standard UC Python UDFs.
• Views containing a UC Python UDF fail on classic SQL warehouses. Serverless or pro only.
• Row-at-a-time is slow. Past toy volume, use Batch UC Python UDFs (PARAMETER STYLE PANDAS + HANDLER, DBR 16.3+). The handler must yield exactly as many rows as it received — design the library with a Series-in/Series-out form from day one rather than retrofitting it. See tokenize_pii.sql.
• Declare DETERMINISTIC where true; the default assumption is non-deterministic and you lose optimizations.
• Lineage goes opaque through UDF bodies — don't hide join or filter logic in there.

Governance and ops

• EXECUTE on the function is not sufficient. The invoking user needs READ VOLUME on the wheel's volume. This is the caveat that bites teams: you ship a governed function and callers still get permission errors. Either grant READ VOLUME broadly on a dedicated read-only udf_libs volume, or accept the coupling. (Contrast service credentials, which do run as the creator.) The release workflow smoke-tests a real invocation to catch this.
• Service credentials are unavailable in standard UC Python UDFs — only Batch UC Python UDFs and scalar Python UDFs. Inside a UDF the API is databricks.service_credentials.getServiceCredentialsProvider(), not dbutils.credentials.*.
• Network egress: UDFs get TCP/UDP on ports 80/443/53. Installing deps from public URLs on a serverless SQL warehouse needs the Enable networking for isolated workloads in Serverless SQL Warehouses preview enabled; serverless notebook/job compute needs egress policies configured. Volume-hosted wheels avoid this entirely — another reason to prefer them over PyPI.
• No DABs resource for functions. Bundles build and upload the wheel (artifacts: type: whl, workspace.artifact_path: /Volumes/...), but there's no resources.functions — registration must be a post-deploy job or script step. Also: artifact_path can't be a DBFS path, and the volume must exist before you can point at it.

Requirements

Capability	Minimum
UC Python UDFs	Serverless/pro SQL warehouse, or DBR 13.3 LTS+
Custom dependencies (ENVIRONMENT)	Serverless, pro SQL warehouse, or DBR 16.2+ classic
Batch UC Python UDFs	DBR 16.3+
Shared isolation environments	DBR 18.0 (scalar) / 17.1 (batch)

Creating functions needs USAGE + CREATE on the schema and USAGE on the catalog; running them needs EXECUTE plus USAGE on schema and catalog.

References

• SQL and Python UDFs in Unity Catalog
• Batch Python UDFs in Unity Catalog
• Serverless environment versions
• CREATE FUNCTION reference
• pip requirements file format · environment markers

License

Apache 2.0