labs install: stale virtualenv with a dangling interpreter symlink is never repaired, so install fails permanently

[vladsagot]

Describe the issue

databricks labs install <project> reuses the existing virtualenv directory at ~/.databricks/labs/<project>/state/venv by running python -m venv <dir> over it.

Python's venv module will not replace files or symlinks that already exist in the target directory unless --clear is passed. If the venv's bin/python3 symlink has become dangling — which happens routinely on macOS whenever Homebrew retires the Python minor version the venv was originally built against — then:

1. python -m venv <dir> reports success and rewrites pyvenv.cfg to advertise the newly detected interpreter version.
2. bin/python and bin/python3 still point at the now-deleted old interpreter.
3. The installer then invokes <dir>/bin/python3 -m pip install ..., which fails instantly because that path does not exist.
4. The command exits with Error: python: failed to install dependencies of .

Because nothing in the flow ever repairs or discards the broken venv, every subsequent retry fails identically. The install is unrecoverable without manually deleting the venv, and nothing in the output hints that this is the problem.

Steps to reproduce the behavior

# 1. Install a labs project while python@3.13 is the newest Homebrew Python
databricks labs install lakebridge

# 2. Let Homebrew retire that interpreter (any of these will do it over time)
brew upgrade
brew cleanup
# ~/.databricks/labs/lakebridge/state/venv/bin/python3 is now a dangling symlink

# 3. Re-run the install
databricks labs install lakebridge
# Error: python: failed to install dependencies of .
# ...and it will fail this way forever

Confirming the broken state:

$ ls -la ~/.databricks/labs/lakebridge/state/venv/bin/python3
python3 -> python3.13 -> /opt/homebrew/opt/python@3.13/bin/python3.13   # does not exist

$ cat ~/.databricks/labs/lakebridge/state/venv/pyvenv.cfg
home = /opt/homebrew/opt/python@3.10/bin
include-system-site-packages = false
version = 3.10.20          # <-- claims 3.10.20, but bin/python3 still points at 3.13

Expected Behavior

labs install should end up with a working virtualenv, or fail with an error that says what is actually wrong. Any of the following would resolve it:

• Create the venv with --clear (or EnvBuilder(clear=True)), so a reused directory is rebuilt from scratch.
• os.RemoveAll the venv directory before creating it — this is an install command, a clean environment is the reasonable expectation.
• Validate that <venv>/bin/python3 resolves to an existing, executable file after creation, and rebuild if it does not.

Actual Behavior

Install fails with Error: python: failed to install dependencies of . and every retry fails identically until the stale venv is deleted manually.

Second, smaller issue: pip's stderr is swallowed

The failure surfaces only as:

Error: python: failed to install dependencies of .

The underlying error was no such file or directory: .../state/venv/bin/python3. Propagating pip's actual stderr (or even just the exec error) would turn this from a debug-log archaeology exercise into an immediately obvious fix. Happy to split this into a separate issue if preferred.

Why this is worth fixing

This is not project-specific — it affects every labs project (ucx, lakebridge, and others), since they all share the same install path. On macOS with Homebrew it is essentially a matter of time: any brew upgrade or brew cleanup that removes an old python@3.X formula silently breaks every venv built against it, and the labs installer has no way to recover.

OS and CLI version

• Databricks CLI: v1.3.0
• Labs project: lakebridge v0.14.2 (requires-python = ">=3.10.1,<3.15")
• OS: macOS 26.5.2 (arm64)
• Pythons present: Homebrew 3.10.20 and 3.14.6, system 3.9.6
• Previously present (now removed by Homebrew): 3.13

Is this a regression?

Unknown — observed on CLI v1.3.0 after a Homebrew Python upgrade removed the interpreter the existing labs venv was linked to.

Debug Logs

15:39:24 Info: start pid=NNNNN version=1.3.0 args="databricks, labs, install, lakebridge, --profile=REDACTED, --log-level=DEBUG"
15:39:25 Debug: Latest lakebridge version is: v0.14.2 pid=NNNNN
15:39:37 Debug: Unpacking zipball to: ~/.databricks/labs/lakebridge/lib pid=NNNNN
15:39:38 Debug: found 8 potential alternative Python versions in $PATH pid=NNNNN
15:39:38 Debug: /opt/homebrew/.../python3.14 --version: Python 3.14.6 pid=NNNNN
15:39:38 Debug: /opt/homebrew/.../python3.10 --version: Python 3.10.20 pid=NNNNN
15:39:38 Debug: /usr/bin/python3 --version: Python 3.9.6 pid=NNNNN
15:39:39 Debug: Detected Python v3.10.20 at: /opt/homebrew/Cellar/python@3.10/3.10.20_4/Frameworks/Python.framework/Versions/3.10/bin/python3.10 pid=NNNNN
15:39:39 Debug: Creating Python Virtual Environment at: ~/.databricks/labs/lakebridge/state/venv pid=NNNNN
15:39:39 Debug: running: /opt/homebrew/.../python3.10 -m venv ~/.databricks/labs/lakebridge/state/venv pid=NNNNN
15:39:41 Debug: Installing Python dependencies for: ~/.databricks/labs/lakebridge/lib pid=NNNNN
15:39:41 Debug: running: ~/.databricks/labs/lakebridge/state/venv/bin/python3 -m pip install --upgrade --upgrade-strategy eager . pid=NNNNN

Error: python: failed to install dependencies of .
15:39:41 Info: failed execution pid=NNNNN exit_code=1 error="python: failed to install dependencies of ." pid=NNNNN

Workaround

rm -rf ~/.databricks/labs/<project>/state/venv
databricks labs install <project>

[vladsagot]

Related desktop-app UX report (same underlying failure, different surface): databrickslabs/lakebridge#2593