pubsub: breaking changes to HTTP and client options

UPDATING.md

@@ -196,11 +196,9 @@ async def auth_callback(token_params):
 The factories raised a bare `ValueError` in 3.x when given no means of authenticating, a
 `key` together with `key_name` or `key_secret`, a `token_details` that is not a
 `TokenDetails`, or `use_token_auth=False` without a key. In 4.0.0 each raises an
-`AblyException` with code 40106 and status 401. Combining `endpoint` with `environment`,
-`rest_host` or `realtime_host` still raises an `AblyException` with code 40106 and status
-400. `push.admin.publish` raised `TypeError` for a `recipient` or `data` that is not a dict
-and `ValueError` for an empty one, and now raises an `AblyException` with code 40000 and
-status 400 for all four.
+`AblyException` with code 40106 and status 401. `push.admin.publish` raised `TypeError`
+for a `recipient` or `data` that is not a dict and `ValueError` for an empty one, and now
+raises an `AblyException` with code 40000 and status 400 for all four.
 
 These exceptions are also a `ValueError`, so `except ValueError` still catches them, but
 `except TypeError` no longer catches the `push.admin.publish` ones:
@@ -405,6 +403,156 @@ pubsub_http_client = create_http_client(key='xxx', default_token_params={'capabi
 await pubsub_http_client.auth.request_token({'ttl': 600000, 'capability': capability})
 ```
 
+### The HTTP options read back their defaults
+
+`http_open_timeout`, `http_request_timeout`, `http_max_retry_count` and
+`http_max_retry_duration` on `client.options` used to read `None` unless you set them, while
+the client applied defaults of its own. They now read back the values in effect: 4, 10, 3 and
+15 by default. The timeouts and the retry duration are still in seconds. Code that tested an
+option for `None`, to tell whether it had been set, will no longer find it so.
+
+### Requests no longer take a `timeout` argument
+
+`time()`, `stats()`, a REST channel's `publish()`, `publish_message()`, `publish_messages()`
+and `publish_name_data()`, `get_message()` on REST and realtime channels, and
+`push.admin.publish()` accepted a `timeout` keyword argument that was never applied: every
+request used the client's `http_open_timeout` and `http_request_timeout` whatever was passed.
+The argument is gone, and passing it raises `TypeError`. `stats()` also loses its unused
+`paginated` argument. Set the timeouts on the client instead:
+
+Example 3.x code:
+```python
+await rest.time(timeout=5)
+```
+
+Example 4.0.0 code:
+```python
+pubsub_http_client = create_http_client(key='xxx', http_request_timeout=5)
+await pubsub_http_client.time()
+```
+
+### `http_request_timeout` bounds writes and connection-pool waits
+
+In 3.x, `http_open_timeout` bounded connecting and `http_request_timeout` bounded each read
+of a response, while writing a request body and waiting for a free pooled connection had no
+timeout at all. `http_request_timeout` now bounds those too, so a request stuck writing to,
+or waiting for, a wedged connection raises `httpx.TimeoutException` and is retried against a
+fallback host instead of hanging. If you publish large bodies over a slow link, raise
+`http_request_timeout` to fit:
+
+```python
+pubsub_http_client = create_http_client(key='xxx', http_request_timeout=30)
+```
+
+### `http_max_retry_count` counts fallback hosts only
+
+In 3.x, `http_max_retry_count` capped the total number of hosts a request was tried
+against, the primary endpoint included, so the default of 3 meant the primary and two
+fallbacks. It now counts fallback hosts only, as in the Ably specification and the other
+Ably SDKs: a request goes to the primary endpoint and then to at most
+`http_max_retry_count` fallback hosts. The default of 3 therefore tries one more host than
+before, and `http_max_retry_count=0`, which used to fail every request, now tries the
+primary endpoint alone. To keep the 3.x number of attempts, subtract one:
+
+Example 3.x code:
+```python
+rest = AblyRest(key='xxx', http_max_retry_count=3)
+```
+
+Example 4.0.0 code:
+```python
+pubsub_http_client = create_http_client(key='xxx', http_max_retry_count=2)
+```
+
+### The `log_level` option is removed
+
+`log_level` was stored on the client options and never read, so it changed nothing. It is
+gone, and passing it raises `TypeError`. The library logs through Python's standard
+`logging` module, to the `ably.pubsub` logger and the loggers beneath it, so configure that
+logger instead to set the level or to choose where records go:
+
+Example 3.x code:
+```python
+rest = AblyRest(key='xxx', log_level=logging.DEBUG)
+```
+
+Example 4.0.0 code:
+```python
+import logging
+
+logger = logging.getLogger('ably.pubsub')
+logger.setLevel(logging.DEBUG)
+logger.addHandler(logging.StreamHandler())
+
+pubsub_http_client = create_http_client(key='xxx')
+```
+
+### Deprecated and duplicate options are removed
+
+These options were deprecated in 3.x, or duplicated another option, and are gone in 4.0.0.
+Passing any of them raises `TypeError`:
+
+- `environment`: pass the same value as `endpoint`, so `environment='sandbox'` becomes
+  `endpoint='sandbox'`.
+- `rest_host` and `realtime_host`: pass the host as `endpoint`, which one host now serves
+  for both REST and realtime.
+- `auth_token`: pass the token as `token`, the name the factories document. The token is now
+  read back as `client.options.token` rather than `client.options.auth_token`.
+- `connection_state_ttl`: this was never accepted as an option, and the
+  `client.options.connection_state_ttl` property, which nothing read, is removed. The value
+  the server sends is on `connection.connection_details.connection_state_ttl`.
+
+Example 3.x code:
+```python
+rest = AblyRest(key='xxx', environment='sandbox')
+realtime = AblyRealtime(key='xxx', realtime_host='custom.ably.example.com')
+rest = AblyRest(auth_token='xxx')
+```
+
+Example 4.0.0 code:
+```python
+pubsub_http_client = create_http_client(key='xxx', endpoint='sandbox')
+pubsub_realtime_client = create_realtime_client(key='xxx', endpoint='custom.ably.example.com')
+pubsub_http_client = create_http_client(token='xxx')
+```
+
+### REST publishes are checked against `max_message_size`
+
+`max_message_size` is now a client option, defaulting to 65536 bytes (64 KiB). A publish
+whose messages exceed it between them raises an `AblyException` with code 40009 and is not
+sent. In 3.x the realtime client checked a fixed 64 KiB limit and REST publishes were not
+checked at all, so an oversized REST publish now fails before it reaches Ably rather than
+being rejected by the server.
+
+Both clients now measure a message the way the Ably specification does: the length of its
+name, client ID and string data, the byte length of binary data, and the length of the JSON
+of object data and of extras. 3.x measured the realtime publish as the whole encoded
+message, so some realtime publishes that 3.x refused are now sent. Raise the limit only if
+your Ably account allows larger messages:
+
+```python
+pubsub_http_client = create_http_client(key='xxx', max_message_size=131072)
+```
+
+### The `recover` option is removed
+
+The realtime client accepted a `recover` option, documented as resuming an earlier
+connection from its recovery key, but connection recovery is not implemented: the option was
+stored and never sent, so a client given one always started a fresh connection. It is gone,
+and passing it raises `TypeError`. Remove it, and rely on the client's own resumption of a
+dropped connection, which is implemented. Connection recovery may return in a minor release,
+along with the `recover` option.
+
+Example 3.x code:
+```python
+realtime = AblyRealtime(key='xxx', recover=recovery_key)
+```
+
+Example 4.0.0 code:
+```python
+pubsub_realtime_client = create_realtime_client(key='xxx')
+```
+
 ## Version 2.x to 3.0.0
 
 The 3.0.0 version of ably-python introduces several breaking changes to improve the realtime experience and align the API with the Ably specification. These include:

ably/pubsub/http/auth.py

@@ -55,7 +55,7 @@ def __init__(self, ably: DefaultPubSubHttpClient | DefaultPubSubRealtimeClient,
         # RSA4: token auth unless use_token_auth says otherwise, or nothing but a key is given
         if options.use_token_auth is None:
             must_use_token_auth = bool(options.auth_callback or options.auth_url
-                                       or options.auth_token or options.token_details)
+                                       or options.token or options.token_details)
         else:
             must_use_token_auth = options.use_token_auth
         must_not_use_token_auth = options.use_token_auth is False
@@ -83,8 +83,8 @@ def __init__(self, ably: DefaultPubSubHttpClient | DefaultPubSubRealtimeClient,
                 raise IncompatibleClientIdException(
                     f"clientId '{options.client_id}' in the client options does not match the "
                     f"token's clientId '{token_client_id}'", 400, 40102)
-        elif options.auth_token:
-            self.__token_details = TokenDetails(token=options.auth_token)
+        elif options.token:
+            self.__token_details = TokenDetails(token=options.token)
         else:
             self.__token_details = None
 
@@ -94,7 +94,7 @@ def __init__(self, ably: DefaultPubSubHttpClient | DefaultPubSubRealtimeClient,
             log.debug("using token auth with auth_url")
         elif options.key_secret:
             log.debug("using token auth with client-side signing")
-        elif options.auth_token:
+        elif options.token:
             log.debug("using token auth with supplied token only")
         elif options.token_details:
             log.debug("using token auth with supplied token_details")

ably/pubsub/http/channel.py

@@ -16,6 +16,7 @@
     assign_idempotent_ids,
     make_message_response_handler,
     make_single_message_response_handler,
+    validate_message_size,
 )
 from ably.pubsub.types.operations import MessageOperation, PublishResult, UpdateDeleteResult
 from ably.pubsub.types.presence import Presence
@@ -76,6 +77,9 @@ def __publish_request_body(self, messages):
 
             request_body_list.append(m)
 
+        # RSL1i
+        validate_message_size(request_body_list, self.ably.options.max_message_size)
+
         request_body = [
             message.as_dict(binary=self.ably.options.use_binary_protocol)
             for message in request_body_list]
@@ -95,10 +99,10 @@ async def _publish(self, arg, *args, **kwargs):
         else:
             raise TypeError(f'Unexpected type {type(arg)}')
 
-    async def publish_message(self, message, params=None, timeout=None):
-        return await self.publish_messages([message], params, timeout=timeout)
+    async def publish_message(self, message, params=None):
+        return await self.publish_messages([message], params)
 
-    async def publish_messages(self, messages, params=None, timeout=None):
+    async def publish_messages(self, messages, params=None):
         request_body = self.__publish_request_body(messages)
         if not self.ably.options.use_binary_protocol:
             request_body = json.dumps(request_body, separators=(',', ':'))
@@ -109,17 +113,17 @@ async def publish_messages(self, messages, params=None, timeout=None):
         if params:
             params = {k: str(v).lower() if isinstance(v, bool) else v for k, v in params.items()}
             path += '?' + parse.urlencode(params)
-        response = await self.ably.http.post(path, body=request_body, timeout=timeout)
+        response = await self.ably.http.post(path, body=request_body)
 
         # Parse response to extract serials
         result_data = response.to_native()
         if result_data and isinstance(result_data, dict):
             return PublishResult.from_dict(result_data)
         return PublishResult()
 
-    async def publish_name_data(self, name, data, timeout=None):
+    async def publish_name_data(self, name, data):
         messages = [Message(name, data)]
-        return await self.publish_messages(messages, timeout=timeout)
+        return await self.publish_messages(messages)
 
     async def publish(self, *args, **kwargs):
         """Publishes a message on this channel.
@@ -259,7 +263,7 @@ async def append_message(self, message: Message, operation: MessageOperation = N
         """
         return await self._send_update(message, MessageAction.MESSAGE_APPEND, operation, params)
 
-    async def get_message(self, serial_or_message, timeout=None):
+    async def get_message(self, serial_or_message):
         """Retrieves a single message by its serial.
 
         Parameters:
@@ -291,7 +295,7 @@ async def get_message(self, serial_or_message, timeout=None):
         path = self.__base_path + 'messages/' + parse.quote_plus(serial, safe=':')
 
         # Make the request
-        response = await self.ably.http.get(path, timeout=timeout)
+        response = await self.ably.http.get(path)
 
         # Create Message from the response
         message_handler = make_single_message_response_handler(self.__cipher)

ably/pubsub/http/http.py

@@ -50,17 +50,10 @@ def __init__(self, key: Optional[str] = None, token: Optional[str] = None,
           - `client_id`: Undocumented
           - `endpoint`: Endpoint specifies either a routing policy name or
             fully qualified domain name to connect to Ably.
-          - `rest_host`: Deprecated: this property is deprecated and will
-            be removed in a future version. The host to connect to.
-            Defaults to rest.ably.io
-          - `environment`: Deprecated: this property is deprecated and
-            will be removed in a future version. The environment to use.
-            Defaults to 'production'
           - `port`: The port to connect to. Defaults to 80
           - `tls_port`: The tls_port to connect to. Defaults to 443
           - `tls`: Specifies whether the client should use TLS. Defaults
             to True
-          - `auth_token`: Undocumented
           - `auth_callback`: Undocumented
           - `auth_url`: Undocumented
         """
@@ -71,7 +64,7 @@ def __init__(self, key: Optional[str] = None, token: Optional[str] = None,
                                            "Provider either a key or key_name & key_secret", 401, 40106)
         # RSA4: a key given alongside a token is kept, to renew the token with
         if token is not None:
-            options = Options(key=key, auth_token=token, **kwargs)
+            options = Options(key=key, token=token, **kwargs)
         elif token_details is not None:
             if not isinstance(token_details, TokenDetails):
                 raise InvalidArgumentException("token_details must be an instance of TokenDetails", 401, 40106)
@@ -103,17 +96,17 @@ async def __aenter__(self):
 
     @catch_all
     async def stats(self, direction: Optional[str] = None, start=None, end=None, params: Optional[dict] = None,
-                    limit: Optional[int] = None, paginated=None, unit=None, timeout=None):
+                    limit: Optional[int] = None, unit=None):
         """Returns the stats for this application"""
         formatted_params = format_params(params, direction=direction, start=start, end=end, limit=limit, unit=unit)
         url = '/stats' + formatted_params
         return await PaginatedResult.paginated_query(
             self.http, url=url, response_processor=stats_response_processor)
 
     @catch_all
-    async def time(self, timeout: Optional[float] = None) -> float:
+    async def time(self) -> float:
         """Returns the current server time in ms since the unix epoch"""
-        r = await self.http.get('/time', skip_auth=True, timeout=timeout)
+        r = await self.http.get('/time', skip_auth=True)
         AblyException.raise_for_response(r)
         return r.to_native()[0]
 

ably/pubsub/http/push.py

@@ -1,5 +1,3 @@
-from typing import Optional
-
 from ably.pubsub.request.paginatedresult import PaginatedResult, format_params
 from ably.pubsub.types.channelsubscription import (
     PushChannelSubscription,
@@ -40,7 +38,7 @@ def device_registrations(self):
     def channel_subscriptions(self):
         return self.__channel_subscriptions
 
-    async def publish(self, recipient: dict, data: dict, timeout: Optional[float] = None):
+    async def publish(self, recipient: dict, data: dict):
         """Publish a push notification to a single device.
 
         :Parameters:
@@ -61,7 +59,7 @@ async def publish(self, recipient: dict, data: dict, timeout: Optional[float] =
 
         body = data.copy()
         body.update({'recipient': recipient})
-        await self.ably.http.post('/push/publish', body=body, timeout=timeout)
+        await self.ably.http.post('/push/publish', body=body)
 
 
 class PushDeviceRegistrations:

ably/pubsub/prototypes.py

@@ -65,11 +65,11 @@ def push(self) -> Push:
         ...
 
     async def stats(self, direction: str | None = None, start=None, end=None, params: dict | None = None,
-                    limit: int | None = None, paginated=None, unit=None, timeout=None) -> PaginatedResult:
+                    limit: int | None = None, unit=None) -> PaginatedResult:
         """Return the stats for this application."""
         ...
 
-    async def time(self, timeout: float | None = None) -> float:
+    async def time(self) -> float:
         """Return the current server time in ms since the unix epoch."""
         ...