Reject credentials_provider with oauth_client_secret; document M2M options

- credentials_provider together with oauth_client_secret now raises
  ValueError instead of silently using the provider, matching the kernel
  auth bridge. azure-sp-m2m keeps ignoring oauth_* values.
- The missing-oauth_client_id check no longer fires for azure-sp-m2m.
- oauth_redirect_port is documented as U2M-only; oauth_client_id and
  oauth_client_secret docs describe the M2M shape and its exclusions.
- CONNECTION_PARAMETERS.md: oauth_client_secret is supported on Thrift.
- Add changelog entries.

Signed-off-by: Amin Ghadersohi <amin.ghadersohi@gmail.com>

Author: aminghadersohi

CHANGELOG.md

@@ -1,5 +1,10 @@
 # Release History
 
+# Unreleased
+- Support OAuth M2M (client credentials) for Databricks service principals on the default Thrift backend: `oauth_client_id` + `oauth_client_secret` now authenticate with tokens from the workspace `/oidc/v1/token` endpoint (scope `all-apis`), refreshed as they expire. Previously the secret was ignored and the connection started an interactive browser login.
+- Reject an unsupported `auth_type` with `ValueError` instead of falling back to the interactive browser login, and reject `oauth_client_secret` together with `credentials_provider` or a U2M `auth_type`, as the kernel backend does.
+- Fix: token responses with extra fields (such as `scope`) no longer fail in `ClientCredentialsTokenSource`.
+
 # 4.6.0 (2026-09-24)
 - Upgrade Databricks SQL Kernel to 1.1.0; the kernel dependency is now stable and no longer experimental.
 - Transparently auto-recover Thrift connections to Reyden / Real-Time warehouses: when a warehouse rejects the default Thrift protocol (SQLSTATE `KP001`), the session is re-opened on the kernel backend and the warehouse is remembered so later connections skip Thrift. Applies only when no backend was chosen explicitly.

CONNECTION_PARAMETERS.md

@@ -83,10 +83,10 @@ to change without notice.
 | Option                                              | Type                 | Thrift | Kernel | Default Value             | Note                                                                                                                                                            |
 | --------------------------------------------------- | -------------------- | :----: | :----: | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `access_token` (PAT)                                | `str`                |   ✅   |   ✅   | `None`                    | Personal Access Token / bearer token. When no auth signal is supplied, Thrift falls back to Databricks OAuth U2M; Kernel requires an explicit U2M `auth_type` or another supported credential flow. |
-| `auth_type`                                         | `str`                |   ✅   |   ✅   | `None`                    | `databricks-oauth` (U2M), `azure-oauth` (Azure AD U2M), or `azure-sp-m2m` (Azure service-principal M2M). All three work on both backends. Thrift treats an otherwise credential-less `None` as Databricks OAuth; Kernel does not implicitly select U2M. On Kernel, `azure-oauth` uses the same workspace-federated browser flow as `databricks-oauth`. |
-| `oauth_client_id` (OAuth)                           | `str`                |   ✅   |   ✅   | built-in id for U2M       | Custom U2M client id on both backends. Kernel also uses it with `oauth_client_secret` or the JWT options for M2M; those M2M flows have no built-in client-id default. |
-| `oauth_redirect_port` (U2M)                         | `int`                |   ✅   |   ✅   | `None`                    | Localhost redirect port for the browser flow. On **both** backends it is only honored when a custom `oauth_client_id` is also supplied — then that single port becomes the redirect URI. With the built-in client id (or when omitted) the connector uses the full registered range 8020–8024 and binds the first free port, so a bare `oauth_redirect_port` has no effect. (Thrift: `auth.py` `oauth_redirect_port_range`; Kernel: same logic, forwarded as `redirect_ports`.) |
-| `oauth_client_secret` (OAuth M2M)                   | `str`                |   ❌   |   ✅   | `None`                    | **Kernel-only in practice.** The Thrift auth path never reads `oauth_client_secret`; use `credentials_provider` or an Azure service principal for M2M on Thrift. |
+| `auth_type`                                         | `str`                |   ✅   |   ✅   | `None`                    | `databricks-oauth` (U2M), `azure-oauth` (Azure AD U2M), or `azure-sp-m2m` (Azure service-principal M2M). All three work on both backends. Thrift rejects any other value with `ValueError`. Thrift treats an otherwise credential-less `None` as Databricks OAuth; Kernel does not implicitly select U2M. On Kernel, `azure-oauth` uses the same workspace-federated browser flow as `databricks-oauth`. |
+| `oauth_client_id` (OAuth)                           | `str`                |   ✅   |   ✅   | built-in id for U2M       | Custom U2M client id on both backends. Both backends also use it with `oauth_client_secret` for M2M, and Kernel with the JWT options; those M2M flows have no built-in client-id default. |
+| `oauth_redirect_port` (U2M)                         | `int`                |   ✅   |   ✅   | `None`                    | Localhost redirect port for the browser (U2M) flow; not used for M2M. On **both** backends it is only honored when a custom `oauth_client_id` is also supplied — then that single port becomes the redirect URI. With the built-in client id (or when omitted) the connector uses the full registered range 8020–8024 and binds the first free port, so a bare `oauth_redirect_port` has no effect. (Thrift: `auth.py` `oauth_redirect_port_range`; Kernel: same logic, forwarded as `redirect_ports`.) |
+| `oauth_client_secret` (OAuth M2M)                   | `str`                |   ✅   |   ✅   | `None`                    | With `oauth_client_id`, selects OAuth M2M (client credentials) for a Databricks service principal on both backends. On Thrift the token comes from `https://<host>/oidc/v1/token` with scope `all-apis` and is refreshed as it expires. Rejected together with `credentials_provider` or `auth_type` `databricks-oauth`/`azure-oauth`; ignored with `azure-sp-m2m`. |
 | `oauth_jwt_key_file` (OAuth M2M, JWT private key)   | `str`                |   ❌   |   ✅   | `None`                    | **Kernel-only.** Path to the PEM private key for JWT private-key M2M (RFC 7523 client assertion). Supplying it selects the JWT flow: the kernel signs a short-lived assertion with the key instead of sending a client secret. Requires `oauth_client_id` + `oauth_jwt_kid`; mutually exclusive with `oauth_client_secret` / `credentials_provider`. |
 | `oauth_jwt_kid` (OAuth M2M, JWT private key)        | `str`                |   ❌   |   ✅   | `None`                    | **Kernel-only.** Key id written into the JWT header so the IdP can select the registered public key. Required with `oauth_jwt_key_file`. (For Entra ID this is the certificate's `x5t` thumbprint.) |
 | `oauth_jwt_passphrase` (OAuth M2M, JWT private key) | `str`                |   ❌   |   ✅   | `None`                    | **Kernel-only.** Passphrase for an encrypted PKCS#8 private key; omit for an unencrypted key.                                                                    |
@@ -215,9 +215,9 @@ installed kernel binding. `force_enable_telemetry` remains Python-driver-only.
 
 ### Supported on Kernel, missing / ignored on Thrift
 
-1. Kernel-managed OAuth M2M: `oauth_client_secret`, `oauth_jwt_key_file`,
-   `oauth_jwt_kid`, `oauth_jwt_passphrase`, `oauth_jwt_algorithm`, and
-   `token_url`.
+1. Kernel-managed OAuth M2M options: `oauth_jwt_key_file`, `oauth_jwt_kid`,
+   `oauth_jwt_passphrase`, `oauth_jwt_algorithm`, and `token_url`
+   (`oauth_client_secret` M2M works on both backends).
 2. Custom `oauth_scopes`.
 3. Kernel U2M encrypted token storage (`oauth_token_cache_enabled`).
 

src/databricks/sql/auth/auth.py

@@ -24,9 +24,22 @@ def get_auth_provider(cfg: ClientContext, http_client):
             + ", ".join(t.value for t in AuthType)
         )
 
+    # azure-sp-m2m is explicit and uses the azure_* credentials; oauth_* values
+    # are ignored for it, as on the kernel path.
+    oauth_m2m = (
+        bool(cfg.oauth_client_secret) and cfg.auth_type != AuthType.AZURE_SP_M2M.value
+    )
+    if oauth_m2m and cfg.credentials_provider:
+        # Rejected on the kernel path too; neither should silently win.
+        raise ValueError(
+            "Ambiguous auth: both a custom credentials_provider and "
+            "oauth_client_secret were provided. Pass oauth_client_id + "
+            "oauth_client_secret for OAuth M2M, or credentials_provider alone."
+        )
+
     if cfg.credentials_provider:
         base_provider = ExternalAuthProvider(cfg.credentials_provider)
-    elif cfg.oauth_client_secret and cfg.auth_type != AuthType.AZURE_SP_M2M.value:
+    elif oauth_m2m:
         if cfg.auth_type in [
             AuthType.DATABRICKS_OAUTH.value,
             AuthType.AZURE_OAUTH.value,
@@ -161,6 +174,11 @@ def get_python_sql_connector_auth_provider(hostname: str, http_client, **kwargs)
         identity_federation_client_id=kwargs.get("identity_federation_client_id"),
         oauth_client_secret=kwargs.get("oauth_client_secret"),
     )
-    if cfg.oauth_client_secret and not kwargs.get("oauth_client_id"):
+    if (
+        cfg.oauth_client_secret
+        and cfg.auth_type != AuthType.AZURE_SP_M2M.value
+        and not cfg.credentials_provider
+        and not kwargs.get("oauth_client_id")
+    ):
         raise ValueError("OAuth M2M needs oauth_client_id with oauth_client_secret")
     return get_auth_provider(cfg, http_client)

src/databricks/sql/client.py

@@ -229,16 +229,19 @@ def __init__(
 
             oauth_client_id: `str`, optional
                 custom oauth client_id. If not specified, it will use the built-in client_id of databricks-sql-python.
+                For OAuth M2M (with `oauth_client_secret`) this is the service principal's client ID and is required.
 
             oauth_client_secret: `str`, optional
                 OAuth secret of a service principal. Together with `oauth_client_id`
                 (the service principal's client ID) this selects OAuth
                 machine-to-machine authentication (client credentials, scope
-                `all-apis`), with tokens refreshed as they expire.
+                `all-apis`), with tokens refreshed as they expire. Cannot be
+                combined with `credentials_provider` or with `auth_type`
+                `databricks-oauth` / `azure-oauth`; ignored for `azure-sp-m2m`.
 
             oauth_redirect_port: `int`, optional
-                port of the oauth redirect uri (localhost). This is required when custom oauth client_id
-                `oauth_client_id` is set
+                port of the oauth redirect uri (localhost) for the interactive (U2M) flow. This is required when
+                a custom `oauth_client_id` is used for U2M. Not used for OAuth M2M (`oauth_client_secret`).
 
             identity_federation_client_id: `str`, optional
                 Service-principal client ID for mandatory SP-wide workload identity

tests/unit/test_auth.py

@@ -205,6 +205,24 @@ def test_get_python_sql_connector_auth_provider_oauth_m2m_errors(self):
                 oauth_client_secret="s",
             )
 
+    def test_get_python_sql_connector_auth_provider_oauth_m2m_ambiguous(self):
+        class MyProvider(CredentialsProvider):
+            def auth_type(self) -> str:
+                return "mine"
+
+            def __call__(self, *args, **kwargs) -> HeaderFactory:
+                return lambda: {"foo": "bar"}
+
+        for client_id in ("c", None):
+            with self.assertRaisesRegex(ValueError, "Ambiguous auth"):
+                get_python_sql_connector_auth_provider(
+                    "example.cloud.databricks.com",
+                    MagicMock(),
+                    credentials_provider=MyProvider(),
+                    oauth_client_id=client_id,
+                    oauth_client_secret="s",
+                )
+
     def test_get_python_sql_connector_auth_provider_unknown_auth_type(self):
         """An unsupported auth_type must not fall back to a browser login."""
         with patch(