Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .genignore
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,19 @@ src/unstructured_client/general.py
# Custom min_attempts / absolute_max_elapsed_time_ms fields on BackoffStrategy.
# Push upstream to Speakeasy templates to remove this entry.
src/unstructured_client/utils/retries.py

# Custom elements_file field on PartitionResponse, for the NDJSON elements-file mode.
# The field is client-side only - the server never returns it - so it cannot come from
# the OpenAPI spec, and regenerating would drop it. If /general/v0/general gains a new
# response field, follow the same procedure as general.py above.
# See test_regeneration_guards.py::test_partition_response_keeps_elements_file.
src/unstructured_client/models/operations/partition.py

# Docs for that same custom elements_file field. Generated from the spec, so a
# regeneration would drop the row.
#
# Note: SDK generation is currently blocked at the Speakeasy account level
# ("generation access blocked"), so nothing can regenerate today and gen.lock has not
# moved since 2026-01. These entries are insurance for when that is restored, not a
# defence against an imminent run.
docs/models/operations/partitionresponse.md
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
## 0.46.0

### Features
* Add an NDJSON elements-file mode to `partition()`. Pass `accept_header_override=PartitionAcceptEnum.APPLICATION_X_NDJSON` to get `PartitionResponse.elements_file` — a path to an NDJSON file with one element per line — instead of `PartitionResponse.elements`. On the split-PDF path the per-chunk temp files are concatenated on disk rather than parsed, flattened, re-serialized with `json.dumps` and re-parsed by the SDK, which held four copies of the document in memory at once; peak memory becomes roughly one chunk instead of the whole document. `elements_file` is set for every input, including ones that are not split and responses from a server that ignores the `Accept` header, so callers need only one code path — but the memory saving itself applies only to split PDFs (a PDF of more than two pages, with `split_pdf_page=True`). **The caller owns the returned file and is responsible for deleting it.** Requesting `application/json` (the default) is unchanged.

## 0.45.0

### Features
Expand Down
31 changes: 31 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -427,6 +427,37 @@ req = operations.PartitionRequest(
)
```

### Streaming elements to a file instead of memory

For very large documents, the parsed element list can dominate the client's memory: the split-PDF path holds a list per chunk, a flattened list, a serialized blob and the SDK's re-parse of that blob. Request `application/x-ndjson` to skip all of it. The chunk responses are concatenated on disk and you get back a path in `elements_file` instead of a list in `elements`, which keeps peak memory at roughly one chunk.
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.

**You own the returned file and are responsible for deleting it.**

> [!NOTE]
> `elements_file` is always set when you pass this header, but the **memory saving** only applies to the split-PDF path. An input is only split when it is a PDF, `split_pdf_page=True` (the default), and it has more than two pages — `split_size` is floored at 2, so one- and two-page PDFs are sent whole. For those, and for non-PDFs, the response body is read fully into memory before being written to disk, so peak memory can reach roughly twice the body size.

Example:
```python
import json
from pathlib import Path

from unstructured_client.general import PartitionAcceptEnum

res = client.general.partition(
request=req,
accept_header_override=PartitionAcceptEnum.APPLICATION_X_NDJSON,
)

try:
with open(res.elements_file, encoding="utf-8") as f:
for line in f:
element = json.loads(line)
...
finally:
# missing_ok so a failure to open the file isn't masked by the cleanup.
Path(res.elements_file).unlink(missing_ok=True)
```

<!-- Start File uploads [file-upload] -->
## File uploads

Expand Down
10 changes: 10 additions & 0 deletions RELEASES.md
Original file line number Diff line number Diff line change
Expand Up @@ -1241,3 +1241,13 @@ Based on:
- [python v0.45.0] .
### Releases
- [PyPI v0.45.0] https://pypi.org/project/unstructured-client/0.45.0 - .

## 2026-08-01 00:00:00
### Changes
Based on:
- OpenAPI Doc
- Speakeasy CLI 1.601.0 (2.680.0) https://github.com/speakeasy-api/speakeasy
### Generated
- [python v0.46.0] .
### Releases
- [PyPI v0.46.0] https://pypi.org/project/unstructured-client/0.46.0 - .
Loading