pubsub: breaking changes to auth and tokens

UPDATING.md

@@ -124,6 +124,189 @@ await pubsub_realtime_client.channels.get('my-channel').detach()
 pubsub_realtime_client.channels.release('my-channel')
 ```
 
+### A key no longer overrides `auth_callback`, `auth_url` or a token
+
+A client given a `key` together with an `auth_callback`, `auth_url`, `token` or `token_details`
+now uses token auth, and keeps the key only as a means of renewing the token. 3.x used basic
+auth with the key whenever one was present, never called the `auth_callback` or `auth_url`,
+and discarded a `token` or `token_details` given alongside the key. So a client configured this
+way now makes requests to your auth server, or presents the token you gave it, where 3.x did
+neither.
+
+To keep using basic auth with the key, pass `use_token_auth=False`:
+
+Example 3.x code:
+```python
+rest = AblyRest(key='xxx', auth_callback=callback)
+```
+
+Example 4.0.0 code:
+```python
+pubsub_http_client = create_http_client(key='xxx', auth_callback=callback, use_token_auth=False)
+```
+
+Or drop the `auth_callback`, `auth_url` or token, which 3.x never used.
+
+### Token params reach an `auth_url` under their wire names
+
+A client with an `auth_url` now sends the token params to it under the names and in the forms
+the other Ably SDKs use, in the query string for `GET` and in the form body for `POST`:
+
+- the client ID as `clientId`, where 3.x sent `client_id`
+- a `capability` given as a dict or `Capability` as JSON, where 3.x sent its Python repr
+- a `ttl` given as a `timedelta` as an integer count of milliseconds, where 3.x sent `1:00:00`
+
+Unset params are left out rather than sent empty. This changes what your auth server receives.
+An auth server written for 3.x clients that reads `client_id` must read `clientId` instead, as it
+already does for clients built with the other SDKs. There is no transition period in which both
+are sent, so update the auth server before, or together with, the clients. The token params
+passed to `create_token_request`, `request_token` and `authorize` keep their snake_case names.
+
+Example 3.x auth server code:
+```python
+client_id = request.args.get('client_id')
+```
+
+Example 4.0.0 auth server code:
+```python
+client_id = request.args.get('clientId')
+```
+
+### A realtime client knows its configured `client_id` before it connects
+
+A realtime client built with a `client_id` now reports it from `auth.client_id` as soon as it is
+created, as an HTTP client always has. 3.x held `auth.client_id` at `None` until the connection
+reached `CONNECTED`. A realtime client built with a `token_details` that carries a client ID
+likewise reports that ID straight away.
+
+Because the client ID is known when the first token is requested, a realtime client now passes
+it to its `auth_callback` as `client_id` in the token params, and to its `auth_url` as
+`clientId`, which 3.x did not do. An auth server or callback that issues tokens from those params
+now issues them for that client ID. Check that it does not reject or override a client ID it was
+not expecting.
+
+Code that read `auth.client_id` before connecting to tell whether the client was connected
+should check `connection.state` instead.
+
+### `authorize()` keeps an explicit `client_id`
+
+A `client_id` passed in the token params to `auth.authorize()` now reaches the token request, the
+`auth_callback` or the `auth_url` as given. The client's own client ID is only filled in when the
+params name none. 3.x replaced an explicit `client_id` with the client's own whenever the client
+had one, and wrote it into the dict the caller passed. The caller's dict is no longer modified.
+
+So a client configured with one client ID that authorizes for another now receives a token for
+the one it asked for, and then fails with an `IncompatibleClientIdException` (code 40102), as the
+client ID of a client cannot change once configured. 3.x silently issued a token for the
+configured client ID instead. Drop the explicit `client_id`, or pass the configured one:
+
+Example 3.x code:
+```python
+await rest.auth.authorize({'client_id': 'someone-else', 'ttl': 3600000})
+```
+
+Example 4.0.0 code:
+```python
+await pubsub_http_client.auth.authorize({'ttl': 3600000})
+```
+
+### A `token_details` whose client ID conflicts with `client_id` is rejected
+
+Creating a client with both a `client_id` and a `token_details` whose own client ID is a different
+one, and is not the wildcard `*`, now raises an `IncompatibleClientIdException` (code 40102) from
+the factory. 3.x built the client: an HTTP client then sent requests under a token for the other
+client ID, and a realtime client failed only once it connected. Make the two agree, or leave
+`client_id` out and let the client take the token's.
+
+Example 3.x code:
+```python
+rest = AblyRest(token_details=token_details_for_bob, client_id='alice')
+```
+
+Example 4.0.0 code:
+```python
+pubsub_http_client = create_http_client(token_details=token_details_for_bob)
+```
+
+A bare `token` string carries no client ID the client can read, so it cannot be checked when the
+client is created. A realtime client still checks it against the client ID the server reports
+when the connection opens, and fails the connection with 40102 if they conflict.
+
+### `TokenDetails.capability` is a JSON string
+
+`TokenDetails.capability` now holds the capability as the JSON string the Ably service returned,
+exactly as it was sent, or `None` where none is known. 3.x parsed it into a `Capability`. Wrap
+the string in `Capability` where you indexed or compared it:
+
+Example 3.x code:
+```python
+token_details = await rest.auth.request_token()
+operations = token_details.capability['my-channel']
+```
+
+Example 4.0.0 code:
+```python
+from ably.pubsub.server import Capability
+
+token_details = await pubsub_http_client.auth.request_token()
+operations = Capability(token_details.capability)['my-channel']
+```
+
+`Capability` is unchanged as the way to build a `capability` token param, and its
+`add_operation_to_resource` method now works where 3.x raised `AttributeError`. A `TokenDetails`
+you construct yourself with a dict or `Capability` stores it as canonical JSON, and
+`TokenDetails.to_dict()` now gives the capability as that string rather than as a dict. So an
+auth server that returns `to_dict()` to its clients from an `auth_url` now sends the capability
+as a JSON string, the form the Ably service itself returns, rather than as a JSON object.
+
+### A `TokenDetails` known only by its token string no longer invents its details
+
+A `TokenDetails` built from a token string alone, whether you construct it yourself, pass a
+`token`, or have an `auth_callback` or `auth_url` return a bare token string or JWT, now has
+`expires`, `issued` and `capability` set to `None`. 3.x invented an expiry an hour after the
+`TokenDetails` was created, an `issued` of `0` and an empty capability. Code that read those
+values must handle `None`; the client itself does, and leaves it to the Ably service to report
+when such a token has expired.
+
+Example 3.x code:
+```python
+seconds_left = (rest.auth.token_details.expires - now_ms) / 1000
+```
+
+Example 4.0.0 code:
+```python
+expires = pubsub_http_client.auth.token_details.expires
+seconds_left = None if expires is None else (expires - now_ms) / 1000
+```
+
+### `default_token_params` apply to `create_token_request()`, and given params replace them
+
+`auth.create_token_request()` called without token params now applies the client's
+`default_token_params`, as `request_token()` and `authorize()` already did. 3.x ignored them. So
+an auth server whose client is configured with `default_token_params` and which signs token
+requests with `create_token_request()` now issues tokens with those defaults, typically a shorter
+`ttl` or a narrower `capability` than before. Check that the defaults are what you want your
+clients' tokens to carry, or remove them.
+
+Token params passed to `create_token_request()` or `request_token()` now replace the defaults
+entirely, as they already did for `authorize()` and as every other Ably SDK does. 3.x merged them
+over the defaults key by key in `request_token()`. Take care where a default `capability` or
+`ttl` restricted the tokens: a `request_token()` call that names other params no longer carries
+them, so its token gets the key's full capability and the service's default `ttl`. Pass every
+param you want, including the defaults you still want:
+
+Example 3.x code:
+```python
+rest = AblyRest(key='xxx', default_token_params={'capability': capability})
+await rest.auth.request_token({'ttl': 600000})  # carried the default capability
+```
+
+Example 4.0.0 code:
+```python
+pubsub_http_client = create_http_client(key='xxx', default_token_params={'capability': capability})
+await pubsub_http_client.auth.request_token({'ttl': 600000, 'capability': capability})
+```
+
 ## Version 2.x to 3.0.0
 
 The 3.0.0 version of ably-python introduces several breaking changes to improve the realtime experience and align the API with the Ably specification. These include:

ably/pubsub/http/auth.py

@@ -34,20 +34,23 @@ def __init__(self, ably: DefaultPubSubHttpClient | DefaultPubSubRealtimeClient,
         self.__auth_options = options
         self.__clock = select_clock(options)
 
-        if not self.ably._is_realtime:
-            self.__client_id = options.client_id
-            if not self.__client_id and options.token_details:
-                self.__client_id = options.token_details.client_id
-        else:
-            self.__client_id = None
+        # RSA7b1, RTC17: a configured clientId holds from the start, on realtime clients too
+        self.__client_id = options.client_id
+        if not self.__client_id and options.token_details:
+            self.__client_id = options.token_details.client_id
         self.__client_id_validated: bool = False
 
         self.__basic_credentials: str | None = None
         self.__auth_params: dict | None = None
         self.__token_details: TokenDetails | None = None
         self.__time_offset: int | None = None
 
-        must_use_token_auth = options.use_token_auth is True
+        # RSA4: token auth unless use_token_auth says otherwise, or nothing but a key is given
+        if options.use_token_auth is None:
+            must_use_token_auth = bool(options.auth_callback or options.auth_url
+                                       or options.auth_token or options.token_details)
+        else:
+            must_use_token_auth = options.use_token_auth
         must_not_use_token_auth = options.use_token_auth is False
         can_use_basic_auth = options.key_secret is not None
         if not must_use_token_auth and can_use_basic_auth:
@@ -67,6 +70,12 @@ def __init__(self, ably: DefaultPubSubHttpClient | DefaultPubSubRealtimeClient,
 
         if options.token_details:
             self.__token_details = options.token_details
+            # RSA15a, RSA15c: the token's clientId is known before any request, so check it now
+            token_client_id = options.token_details.client_id
+            if options.client_id is not None and token_client_id not in (None, '*', options.client_id):
+                raise IncompatibleClientIdException(
+                    f"clientId '{options.client_id}' in the client options does not match the "
+                    f"token's clientId '{token_client_id}'", 400, 40102)
         elif options.auth_token:
             self.__token_details = TokenDetails(token=options.auth_token)
         else:
@@ -109,21 +118,26 @@ async def __authorize_when_necessary(self, token_params=None, auth_options=None,
 
     async def _ensure_valid_auth_credentials(self, token_params=None, auth_options=None, force=False):
         self.__auth_mechanism = Auth.Method.TOKEN
+        explicit_client_id = None
         if token_params is None:
             token_params = dict(self.auth_options.default_token_params)
         else:
+            token_params = dict(token_params)
+            explicit_client_id = token_params.get('client_id')
             self.auth_options.default_token_params = dict(token_params)
             self.auth_options.default_token_params.pop('timestamp', None)
 
         if auth_options is not None:
             self.auth_options.replace(auth_options)
         auth_options = dict(self.auth_options.auth_options)
-        if self.client_id is not None:
+        # RSA10h: Auth#clientId is only the default for an explicit clientId (RSA10b, RSA10j),
+        # and takes precedence over one from default_token_params (RSA7a4)
+        if self.client_id is not None and explicit_client_id is None:
             token_params['client_id'] = self.client_id
 
         token_details = self.__token_details
         if not force and not self.token_details_has_expired():
-            log.debug("using cached token; expires = %d",
+            log.debug("using cached token; expires = %s",
                       token_details.expires)
             return token_details
 
@@ -159,9 +173,10 @@ async def request_token(self, token_params: dict | None = None,
                             auth_url: str | None = None, auth_method: str | None = None,
                             auth_headers: dict | None = None, auth_params: dict | None = None,
                             query_time=None):
-        token_params = token_params or {}
-        token_params = dict(self.auth_options.default_token_params,
-                            **token_params)
+        # RSA8e: given params replace the defaults rather than merging with them
+        if token_params is None:
+            token_params = self.auth_options.default_token_params
+        token_params = dict(token_params)
         key_name = key_name or self.auth_options.key_name
         key_secret = key_secret or self.auth_options.key_secret
 
@@ -232,7 +247,9 @@ async def request_token(self, token_params: dict | None = None,
 
     async def create_token_request(self, token_params: dict | str | None = None, key_name: str | None = None,
                                    key_secret: str | None = None, query_time=None):
-        token_params = token_params or {}
+        # RSA5c, RSA6c, RSA9h: the defaults apply when no params are given, and given params replace them
+        if token_params is None:
+            token_params = dict(self.auth_options.default_token_params)
         token_request = {}
 
         key_name = key_name or self.auth_options.key_name
@@ -336,9 +353,14 @@ def _configure_client_id(self, new_client_id):
         log.debug("Auth._configure_client_id(): new client_id = %s", new_client_id)
         original_client_id = self.client_id or self.auth_options.client_id
 
+        # RSA12a, RSA15a: a token that names no clientId, such as a bare token string or a JWT
+        # the library does not parse, leaves a configured clientId as it is
+        if new_client_id is None and self.auth_options.client_id is not None:
+            return
+
         # If new client ID from Ably is a wildcard, but preconfigured clientId is set,
-        # then keep the existing clientId
-        if original_client_id != '*' and new_client_id == '*':
+        # then keep the existing clientId. With none set, the wildcard is the clientId (RSA7b4)
+        if original_client_id is not None and original_client_id != '*' and new_client_id == '*':
             self.__client_id_validated = True
             self.__client_id = original_client_id
             return
@@ -364,11 +386,12 @@ def can_assume_client_id(self, assumed_client_id):
 
     async def _get_auth_headers(self):
         if self.__auth_mechanism == Auth.Method.BASIC:
-            # RSA7e2
-            if self.client_id:
+            # RSA7e2: the configured clientId, not a wildcard the server reported
+            client_id = self.auth_options.client_id
+            if client_id:
                 return {
                     'Authorization': f'Basic {self.basic_credentials}',
-                    'X-Ably-ClientId': base64.b64encode(self.client_id.encode('utf-8'))
+                    'X-Ably-ClientId': base64.b64encode(client_id.encode('utf-8'))
                 }
             return {
                 'Authorization': f'Basic {self.basic_credentials}',
@@ -386,11 +409,29 @@ def _timestamp(self):
     def _random_nonce(self):
         return uuid.uuid4().hex[:16]
 
+    @staticmethod
+    def _auth_url_token_params(token_params):
+        """Returns the token params under their wire names and in their wire forms (RSA8c1a)"""
+        params = {}
+        for name, value in token_params.items():
+            if value is None:
+                continue
+            if name == 'client_id':
+                name = 'clientId'
+            elif name == 'ttl' and isinstance(value, timedelta):
+                value = int(value.total_seconds() * 1000)
+            elif name == 'capability' and not isinstance(value, str):
+                value = str(Capability(value))
+            params[name] = value
+        return params
+
     async def token_request_from_auth_url(self, method: str, url: str, token_params,
                                           headers, auth_params):
         # Extract URL parameters using utility function
         clean_url, url_params = extract_url_params(url)
 
+        token_params = self._auth_url_token_params(token_params)
+
         body = None
         params = None
         if method == 'GET':

ably/pubsub/http/http.py

@@ -69,14 +69,15 @@ def __init__(self, key: Optional[str] = None, token: Optional[str] = None,
         if key is not None and ('key_name' in kwargs or 'key_secret' in kwargs):
             raise ValueError("key and key_name or key_secret are mutually exclusive. "
                              "Provider either a key or key_name & key_secret")
-        if key is not None:
-            options = Options(key=key, **kwargs)
-        elif token is not None:
-            options = Options(auth_token=token, **kwargs)
+        # RSA4: a key given alongside a token is kept, to renew the token with
+        if token is not None:
+            options = Options(key=key, auth_token=token, **kwargs)
         elif token_details is not None:
             if not isinstance(token_details, TokenDetails):
                 raise ValueError("token_details must be an instance of TokenDetails")
-            options = Options(token_details=token_details, **kwargs)
+            options = Options(key=key, token_details=token_details, **kwargs)
+        elif key is not None:
+            options = Options(key=key, **kwargs)
         elif not ('auth_callback' in kwargs or 'auth_url' in kwargs or
                   # and don't have both key_name and key_secret
                   ('key_name' in kwargs and 'key_secret' in kwargs)):

ably/pubsub/realtime/presence.py

@@ -27,7 +27,7 @@
 
 def _get_client_id(presence: RealtimePresence) -> str | None:
     """Get the clientId for the current connection."""
-    # Use auth.client_id if available (set after CONNECTED),
+    # Use auth.client_id if available,
     # otherwise fall back to auth_options.client_id
     return presence.channel.ably.auth.client_id or presence.channel.ably.auth.auth_options.client_id