Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,40 @@ if __name__ == "__main__":

See `examples/async_collection_operations.py` for a fuller async walkthrough.

## Streaming document exports

`documents.export()` loads the whole export into memory. `documents.export_stream()` reads it as it arrives:

```python
with client.collections["books"].documents.export_stream({"batch_size": 500}) as stream:
for line in stream: # str, without the line feed
document = json.loads(line)

with client.collections["books"].documents.export_stream() as stream, open("books.jsonl", "wb") as file:
for chunk in stream.iter_bytes(): # decoded bytes
file.write(chunk)
```

With `AsyncClient`, await the call and use `async with` and `async for`, with `stream.aiter_bytes()` for the bytes:

```python
async with await client.collections["books"].documents.export_stream() as stream:
async for line in stream:
document = json.loads(line)
```

A stream can be read once, either as lines or as bytes. Keep these in mind:

- **Always close the stream**, ideally with a context manager. With `max_concurrent_requests` set, a stream that is abandoned before its end permanently uses up one of the open export places below.
- **A failed export has to be restarted.** Nodes are only retried until the response headers arrive. After that, errors such as `httpx.RemoteProtocolError` or `httpx.ReadTimeout` are raised as they are, and exports cannot resume from where they stopped.
- **Keep reading.** If your code stops reading for about 60 seconds, the server drops the connection. `stream_read_timeout_seconds` (default 60) separately limits how long the client waits for the server's next chunk.
- **Lines are raw.** They are not parsed, and the server could write an error message into the body after a successful status, so a line is not guaranteed to be a document.
- **Exports are uncompressed by default.** The server compresses on the thread that handles all other requests, which delays them, especially with a large `batch_size`. Pass `compress=True` if bandwidth matters more. An `Accept-Encoding` in `additional_headers` is always respected.

With `max_concurrent_requests`, an export only takes a slot while it opens, so requests made while reading an export do not wait for it. To keep connections free for those requests, at most `max_connections - max_concurrent_requests` exports can be open at once. If that is not positive, one export is still allowed, and it can leave a request short of a connection, so keep `max_concurrent_requests` below `max_connections`. Further exports wait for a place, so opening an export inside another one can wait forever when no place is free. The limit does not apply to an `http_client` you pass in, because its pool size is unknown.

If your application cancels many requests, prefer [httpx2](#using-httpx2), which fixes a connection pool leak that cancellation can trigger in httpx.

## Using httpx2

The client sends requests with [httpx](https://www.python-httpx.org/) by default. On Python 3.10+ you can pass an [httpx2](https://github.com/pydantic/httpx2) client instead. httpx2 is Pydantic's maintained continuation of httpx, and it fixes a connection pool leak in httpcore ([encode/httpcore#1093](https://github.com/encode/httpcore/issues/1093)) that can leave an `AsyncClient` failing every request with `PoolTimeout` under load.
Expand Down
61 changes: 61 additions & 0 deletions examples/async_export_streaming.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
import asyncio
import json
import os
import sys

curr_dir = os.path.dirname(os.path.realpath(__file__))
repo_root = os.path.abspath(os.path.join(curr_dir, os.pardir))
sys.path.insert(1, os.path.join(repo_root, "src"))

import typesense

COLLECTION = "async_export_streaming_books"


async def main() -> None:
client = typesense.AsyncClient(
{
"api_key": os.environ.get("TYPESENSE_API_KEY", "xyz"),
"nodes": [{"host": "localhost", "port": "8108", "protocol": "http"}],
"connection_timeout_seconds": 10,
}
)

await client.collections.create(
{
"name": COLLECTION,
"fields": [
{"name": "title", "type": "string"},
{"name": "year", "type": "int32"},
],
}
)

try:
documents = client.collections[COLLECTION].documents
await documents.import_(
[{"id": str(year), "title": f"Book {year}", "year": year} for year in range(2000, 2010)],
{"action": "create"},
)

# Read one document at a time. The stream is closed when the block ends,
# even if the loop stops early.
async with await documents.export_stream(
{"filter_by": "year:>=2005", "batch_size": 100},
) as stream:
async for line in stream:
print(json.loads(line)["title"])

# Count the bytes without keeping the export in memory.
size = 0
async with await documents.export_stream() as stream:
async for chunk in stream.aiter_bytes():
size += len(chunk)
print(f"Read {size} bytes")
finally:
await client.collections[COLLECTION].delete()
await client.api_call.aclose()


if __name__ == "__main__":
asyncio.run(main())
52 changes: 52 additions & 0 deletions examples/export_streaming.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
import json
import os
import sys

curr_dir = os.path.dirname(os.path.realpath(__file__))
repo_root = os.path.abspath(os.path.join(curr_dir, os.pardir))
sys.path.insert(1, os.path.join(repo_root, "src"))

import typesense

COLLECTION = "export_streaming_books"

client = typesense.Client(
{
"api_key": os.environ.get("TYPESENSE_API_KEY", "xyz"),
"nodes": [{"host": "localhost", "port": "8108", "protocol": "http"}],
"connection_timeout_seconds": 10,
}
)

client.collections.create(
{
"name": COLLECTION,
"fields": [
{"name": "title", "type": "string"},
{"name": "year", "type": "int32"},
],
}
)

try:
documents = client.collections[COLLECTION].documents
documents.import_(
[{"id": str(year), "title": f"Book {year}", "year": year} for year in range(2000, 2010)],
{"action": "create"},
)

# Read one document at a time. The stream is closed when the block ends,
# even if the loop stops early.
with documents.export_stream({"filter_by": "year:>=2005", "batch_size": 100}) as stream:
for line in stream:
print(json.loads(line)["title"])

# Write the export to a file without keeping it in memory.
path = f"{COLLECTION}.jsonl"
with documents.export_stream() as stream, open(path, "wb") as file:
for chunk in stream.iter_bytes():
file.write(chunk)
print(f"Wrote {os.path.getsize(path)} bytes to {path}")
os.remove(path)
finally:
client.collections[COLLECTION].delete()
Loading
Loading