4.0.0 findings: presence type exports, ignored recover, auth_url param names #736

These came up while validating the Pub/Sub 2.0 website docs against integration/v4 (the 4.0.0 release) at 7032b4cd2f8c0e55557a864e3d5995a84b67c568. They're grouped here so they can be triaged together. If you'd rather track any of them on its own, split it into a separate issue.

The missing ably-pubsub-server token in Ably-Agent is already tracked in #735, so it isn't repeated here.

Summary

P1 = should be fixed before the 2.0 release · P2 = should be fixed soon · P3 = nice to have.

1. Export PresenceMessage and PresenceAction, and let from_encoded decrypt with public types (TM3/TP4) (P2)

What's wrong

• The presence types aren't exported. ably.pubsub.server is the whole public API in 4.0, but it doesn't export PresenceMessage or PresenceAction, and neither does ably.pubsub.server.sync.
• from_encoded can't decrypt with public types. Decrypting an encrypted payload needs an internal cipher object that no public function returns.

Evidence

Missing exports. The __all__ list (server/__init__.py#L256-L299) has no PresenceMessage or PresenceAction. They live only in ably.pubsub.types.presence (#L22, #L30), which the upgrade guide calls "internal and free to move" (UPDATING.md#L47-L50).

So there is no supported way to:

• call PresenceMessage.from_encoded_array(...) on presence payloads received outside a channel (a queue, a webhook);
• compare against PresenceAction.ENTER and the other actions;
• annotate against the presence types.

Decrypting.

• Message.from_encoded(obj, cipher=None, ...) (message.py#L291) hands cipher to decode, which calls cipher.decrypt(data) (mixins.py#L119).
• Only the internal CbcChannelCipher has decrypt (crypto.py#L45).
• The function that builds one, get_cipher (crypto.py#L163), isn't exported.

With an encrypted payload and only public types:

from ably.pubsub.server import Message, ChannelOptions, get_default_params, generate_random_key
params = get_default_params({'key': key})
Message.from_encoded(wire, cipher=params)
# AttributeError: 'CipherParams' object has no attribute 'decrypt'
Message.from_encoded(wire, ChannelOptions(cipher=params))
# AttributeError: 'ChannelOptions' object has no attribute 'decrypt'

PresenceMessage.from_encoded and from_encoded_array behave the same way.

# PresenceMessage, PresenceAction, Message present in dir(...)?
async False False True
sync  False False True

public params type: CipherParams has decrypt: False
wire encoding: utf-8/cipher+aes-256-cbc/base64
internal cipher -> secret text
public CipherParams -> AttributeError: 'CipherParams' object has no attribute 'decrypt'
cipher dict -> AttributeError: 'dict' object has no attribute 'decrypt'
ChannelOptions -> AttributeError: 'ChannelOptions' object has no attribute 'decrypt'
no cipher -> data type bytearray encoding utf-8/cipher+aes-256-cbc
PresenceMessage + CipherParams -> AttributeError: 'CipherParams' object has no attribute 'decrypt'

History

This is an oversight, not a decision:

• The presence types were never exported. The first server/__init__.py (db091b5) carried the 3.x export list over, and the 3.x list didn't have them either.
• The same gap has already been fixed for other types:
  • 4a869cf exported Message, because "the migration guide tells users to import types like it from ably.pubsub.server";
  • Export generate_random_key and get_default_params from ably.pubsub.server #733 exported the crypto helpers.
• The decrypt gap is new in 4.0 in practice. from_encoded(obj, cipher=None) dates from 2017. In 3.x it worked through ably.util.crypto.get_cipher, which 4.0 makes internal.

Impact

TM3 and TP4 define fromEncoded / fromEncodedArray as:

• taking an optional channelOptions;
• decrypting "using the cipher in the channelOptions";
• for "users receiving messages other than from a REST or Realtime channel (for example, from a queue)".

Python 4.0 has no public way to do that. The Ably docs hide the presence fromEncodedArray() paragraph for Python, because there is nothing public to import.

Recommendation

• Export PresenceMessage and PresenceAction from ably.pubsub.server and ably.pubsub.server.sync.
• Have from_encoded / from_encoded_array on both types accept ChannelOptions (per TM3/TP4), and ideally CipherParams or a cipher dict too. Build the channel cipher internally with get_cipher.

2. recover client option is documented but ignored: no recover param on connect, no create_recovery_key() (P2)

What's wrong

create_realtime_client documents recover as a working option (server/__init__.py#L194-L196):

A recovery key from a previous connection, used to resume that connection's state and message continuity rather than starting fresh.

Options accepts and stores it (options.py#L30, #L112, #L198-L204), but nothing reads it.

Evidence

• No recover on connect. The connect parameters never include recover (connectionmanager.py#L205-L220). With the UTS mock websocket, a client created with recover='...' connects with {'key': ..., 'v': '6', 'format': 'msgpack', 'echo': 'true'}, and nothing is logged about it (checked at WARNING).
• No way to get a key. Connection has no create_recovery_key() (connection.py).
• The proxy test agrees. The UTS deviations record the same thing against the proxy: a client given a valid recover= was handed a fresh connectionId (test/uts/deviations.md#L1047).

options.recover = {"connectionKey":"abc!def","msgSerial":5,"channelSerials":{}}
connect query params = {'key': 'app.key:secret', 'v': '6', 'format': 'msgpack', 'echo': 'true'}
'recover' in params: False
Connection has create_recovery_key: False | createRecoveryKey: False
recovery-ish attrs on Connection: []

History

• Recovery was never built. The option has been stored since 353100b (2014), but recovery was never implemented (and never removed).
• The docstring is new in 4.0. It came in with the expanded factory docstrings in 85f579d; the 3.x AblyRealtime docstring never mentioned recover. So 4.0 would be the first release to advertise the option, and a docstring fix before release is possible.
• Known elsewhere, not tracked here. The open README PR docs: correct the supported Python versions and the feature support claims #724 (on main) already lists "connection recovery via recover" as absent.

Impact

Someone porting recovery code gets no error. That code uses RTN16 (Connection#createRecoveryKey()) and the recover option, which should be sent as a recover query param on the first connect (RTN16k). Instead, the connection silently starts fresh, and the message continuity they asked for is lost. The 2.0 docs already leave Python out of connection recovery.

Recommendation

• Implement RTN16:
  • Connection.create_recovery_key();
  • send recover on the first connect;
  • restore msgSerial and the channel serials.
• Until then, before 4.0.0 ships:
  • drop recover from the factory docstring;
  • reject it at construction, or at least log a warning, so it doesn't fail silently.

3. Stale ably[vcdiff] install hint, README Python floor and upgrade-guide gaps (P3)

What's wrong

• The vcdiff hint points at the 3.x package. Without vcdiff-decoder, AblyVCDiffDecoder() fails with an ImportError that says to install ably[vcdiff].
• The README still says Python 3.7+.
• UPDATING.md misses the Python 3.8 floor and the extras rename.

The pyproject.toml / version.py bump and the 4.0.0 CHANGELOG entry are already release steps (CONTRIBUTING.md), and the version strings were flagged on #693, so they aren't repeated here.

Evidence

Stale vcdiff hint

ImportError: VCDiff plugin requires vcdiff library. Install with: pip install ably[vcdiff]

• The message is at defaultvcdiffdecoder.py#L47 and #L50.
• Following it installs the 3.x ably distribution next to 4.0, just to get the decoder. That makes the 4.0 decoder work, but it also drags in 3.x and turns ably into a regular package.
• The crypto message already says ably-pubsub-server[crypto] (nocrypto.py#L5).

README Python floor. README.md#L36 and #L39 say 3.7+. But #L77 and requires-python (pyproject.toml#L6) say 3.8. #724 fixes this on main only.

Upgrade guide gaps (UPDATING.md#L3-L125):

• 3.8 floor: Python 3.8 is now the minimum (requires-python = ">=3.8"; 3.x allowed 3.7). On 3.7, pip install ably-pubsub-server finds no installable version. That follows from requires-python and from the distribution having no earlier releases; I didn't run it on a 3.7 interpreter.
• Extras rename: the extras move with the distribution. ably[crypto] becomes ably-pubsub-server[crypto], and likewise vcdiff and oldcrypto (pyproject.toml#L39-L41). A requirements file that renames only the base package drops those dependencies.

History

• vcdiff hint: missed by accident. It came from 2d487dd ([ECO-5456] feat: add VCDiff support for delta message decoding #620). The 4.0 docstring pass in 85f579d moved the file's usage docstring but left the message lines. Export generate_random_key and get_default_params from ably.pubsub.server #733 then fixed only the crypto twin.
• 3.8 floor: deliberate. 85f579d raised it on purpose ("typing.Protocol makes 3.8 the minimum supported Python"). Only the guide line is missing.

Impact

The 4.0 docs install ably-pubsub-server. Users who follow the vcdiff error end up with the 3.x package installed as well. Users on 3.7, or with un-renamed extras, get no hint from the guide.

Recommendation

• Change the vcdiff message (both lines) to pip install 'ably-pubsub-server[vcdiff]'.
• Make sure the README's 3.8 floor reaches integration/v4.
• Add the Python 3.8 minimum and the extras rename to the 3.x → 4.0.0 section of UPDATING.md.

4. Token params: auth_url receives client_id instead of clientId, and camelCase token params are silently ignored (P3)

What's wrong

Two halves of the same problem:

• auth_url gets the SDK's internal parameter names. An auth server receives client_id, not clientId.
• camelCase keys are dropped without a warning. {'clientId': 'alice'} produces a TokenRequest with no client ID.

Together, a Python client and a Python auth server agree with each other, but neither interoperates with the other SDKs.

Evidence

auth_url gets the SDK's internal parameter names.

• _ensure_valid_auth_credentials puts the client ID into the token params as client_id (auth.py#L121-L122).
• token_request_from_auth_url sends the dict as-is: as query params for GET (auth.py#L401), or as the form body for POST (auth.py#L407).

Against a local auth endpoint, create_http_client(auth_url=..., client_id='alice') sends:

GET  ?ttl=60000&client_id=alice
POST (application/x-www-form-urlencoded) ttl=60000&client_id=alice

The UTS spec for RSA12b (uts/rest/unit/auth/client_id.md, clientid-sent-to-authurl-0) asserts query_params["clientId"] (or clientId in the POST body). An auth server written against Ably's docs or another SDK reads clientId, for example by passing req.query to createTokenRequest. It never sees the client ID, and it issues a token without one. This is already listed in test/uts/deviations.md#L1931.

camelCase keys are dropped without a warning. create_token_request reads only token_params.get('client_id') (auth.py#L275), so:

await client.auth.create_token_request({'clientId': 'alice'})   # -> TokenRequest with clientId None

• The same goes for request_token(token_params={'clientId': 'alice'}).
• The result is a signed TokenRequest with no client ID (TK2c: the token then can't assume one), and nothing is logged.
• client_id is the right key in Python, and the SDK maps it to clientId when it serialises the TokenRequest (tokenrequest.py#L43).
• But every other SDK's examples use the camelCase spelling, and other SDKs send it to an authUrl, so this is an easy mistake to make and a hard one to spot.

('GET',  {'ttl': ['60000'], 'client_id': ['alice']})
('POST', 'application/x-www-form-urlencoded', {'ttl': ['60000'], 'client_id': ['alice']})
('GET',  {'client_id': ['bob'], 'ttl': ['60000']})          # caller passed token_params={'client_id': 'bob'}

{'clientId': 'alice'} -> {'clientId': None, 'ttl': None}
{'client_id': 'alice'} -> {'clientId': 'alice', 'ttl': None}
{'clientId': 'alice', 'ttl': 60000} -> {'clientId': None, 'ttl': 60000}

History

• Long-standing behaviour that nobody documented as a design choice. The client_id insertion dates from f96ff72 (2016), and the GET/POST split from 3447b62 (2015).
• Known, but never filed as an issue. owenpearson recorded it as a known deviation when deriving the auth unit specs (95e85bd), and test: derive the auth unit specs #702 lists it among the items "for a maintainer's eye".
• The reverse direction was fixed in 2019. Authentication with auth_url doesn't accept camel case #136 covered a camelCase auth_url response, and Fix authentication with auth_url #137 fixed it.

Impact

Mixed-SDK setups break silently. A Python client talking to a Node or Java auth server, or a Python server handling another SDK's params, ends up with tokens that have no client ID. Nothing in the logs explains why.

Recommendation

• In token_request_from_auth_url, map TokenParams to their wire names (clientId, ttl, capability, timestamp, nonce) before sending.
• In create_token_request / request_token, accept the wire names as aliases (or at least warn on unrecognised keys).

5. batch_publish() needs overloads: a single spec returns one BatchResult, but the type says it might be a list (P3)

What's wrong

batch_publish() takes one spec or a list of specs, and returns one BatchResult or a list to match. The return type doesn't express that, so a type checker treats every call as possibly returning a list, and rejects attribute access on the single-spec result.

Evidence

• The prototype the factories return declares a plain union (prototypes.py#L76-L77):

  async def batch_publish(self, specs: BatchPublishSpec | dict | list[BatchPublishSpec | dict]
                          ) -> BatchResult | list[BatchResult]:

  The implementation has the same union (http.py#L117).

• At runtime a single spec returns one BatchResult (http.py#L156):

  return results[0] if single else results

• So this correct, single-spec call fails pyright:

  batch_result = await pubsub_http_client.batch_publish(BatchPublishSpec(
      channels=['test1', 'test2'], messages=[Message(data='myData')]))
  print('Success count:', batch_result.success_count)

  error: Cannot access attribute "success_count" for class "list[BatchResult[Unknown]]" (reportAttributeAccessIssue)
  

  The same error follows for failure_count and results. At runtime the call works and prints the counts.

• With the two overloads below added to a local copy of prototypes.py, pyright reports no errors on the unchanged call.

History

Added with batch publish in 1059edc ("feat: add batch publish and batch presence to the HTTP client", #727). The commit message describes the single-or-list behaviour, so the runtime is intended; only the typing doesn't follow it.

Impact

Anyone type-checking code that publishes one spec has to add an isinstance assert or a cast, or ignore the error. The 2.0 website docs' Python batch-publish sample is a single-spec call, so it fails pyright as written.

Recommendation

Add overloads to the prototype; the sync door is generated from it by unasync, so it picks them up too:

@overload
async def batch_publish(self, specs: BatchPublishSpec | dict) -> BatchResult: ...
@overload
async def batch_publish(self, specs: list[BatchPublishSpec | dict]) -> list[BatchResult]: ...

Checklist

• PresenceMessage and PresenceAction exported; from_encoded accepts ChannelOptions
• recover implemented (RTN16), or removed from the docstring and rejected or warned on before 4.0.0
• vcdiff install hint, README 3.8 floor, and UPDATING lines for the 3.8 minimum and the extras rename
• auth_url sends wire-named TokenParams; camelCase keys accepted or warned on
• batch_publish() overloads for a single spec and a list

Related

• 4.0.0 release PR: Integration branch for ably-pubsub-server v4 package #693
• Ably-Agent no longer includes ably-pubsub-server on integration/v4 #735: Ably-Agent no longer includes ably-pubsub-server. That's a related item, tracked there and not duplicated here.
• Add types + Make library PEP-561 compatible #528: py.typed (I'm adding a comment there about where the marker needs to go for 4.0)
• docs: correct the supported Python versions and the feature support claims #724 (README floor and missing features, on main), Export generate_random_key and get_default_params from ably.pubsub.server #733 (crypto exports), [ECO-5456] feat: add VCDiff support for delta message decoding #620 (vcdiff support), feat: add batch publish and batch presence to the HTTP client #727 (batch publish)
• test: derive the auth unit specs #702 (auth unit specs recording the auth_url deviation), Authentication with auth_url doesn't accept camel case #136 (the reverse camelCase case)

┆Issue is synchronized with this Jira Task by Unito