Commit 5cd086c
Implement batch API with changeset, upsert, and DataFrame integration (#129)
## Summary
- Adds `client.batch` namespace -- a deferred-execution batch API that
packs multiple
Dataverse Web API operations into a single `POST $batch` HTTP request
- Adds `client.batch.dataframe` namespace -- pandas DataFrame wrappers
for batch operations
- Adds `client.records.upsert()` and `client.batch.records.upsert()`
backed by the
`UpsertMultiple` bound action with alternate-key support
- Fixes a bug where alternate key fields were merged into the
UpsertMultiple request
body, causing `400 Bad Request` on the create path
## Batch API Design
Implements the [Batch API
Design](#129 (comment))
spec from @sagebree:
| Capability | How to use | Status |
|---|---|---|
| Record CRUD (create / update / delete / get) | `batch.records.*` |
Done |
| Upsert by alternate key | `batch.records.upsert(...)` | Done |
| Table metadata (create / delete / columns / relationships) |
`batch.tables.*` | Done |
| SQL queries | `batch.query.sql(...)` | Done |
| Atomic write groups | `batch.changeset()` | Done |
| Continue past failures | `batch.execute(continue_on_error=True)` |
Done |
| DataFrame integration | `batch.dataframe.create/update/delete` | Done
(new) |
**Design constraints enforced:**
- Maximum 1000 operations per batch (validated before sending)
- `records.get` paginated overload not supported -- single-record only
- GET operations cannot be placed inside a changeset (enforced by API
design)
- Content-ID references are only valid within the same changeset
- File upload operations not batchable
- `tables.create` returns no table metadata on success (HTTP 204)
- `tables.add_columns` / `tables.remove_columns` do not flush the
picklist cache
- `client.flush_cache()` not supported in batch (client-side operation)
## What's included
### New: `client.batch` API
- `batch.records.create / get / update / delete / upsert`
- `batch.tables.create / get / list / add_columns / remove_columns /
delete`
- `batch.tables.list(filter=..., select=...)` -- parity with
`client.tables.list()` from #112
- `batch.tables.create_one_to_many_relationship /
create_many_to_many_relationship / delete_relationship /
get_relationship / create_lookup_field`
- `batch.query.sql`
- `batch.changeset()` context manager for transactional (all-or-nothing)
operations
- Content-ID reference chaining inside changesets (globally unique
across all changesets via shared counter)
- `execute(continue_on_error=True)` for mixed success/failure batches
- `BatchResult` with `.responses`, `.succeeded`, `.failed`,
`.created_ids`, `.has_errors`
### New: `client.batch.dataframe` API
- `batch.dataframe.create(table, df)` -- DataFrame rows to
CreateMultiple batch item
- `batch.dataframe.update(table, df, id_column)` -- DataFrame rows to
update batch items
- `batch.dataframe.delete(table, ids_series)` -- pandas Series to delete
batch items
### Existing: Refactored existing APIs
- Payload generation shared between batch and direct API via `_build_*`
/ `_RawRequest` pattern
- Execution of batch operations deferred to `execute()`
### OData $batch spec compliance
- Audited against [Microsoft Learn
docs](https://learn.microsoft.com/en-us/power-apps/developer/data-platform/webapi/execute-batch-operations-using-web-api)
- `Content-Transfer-Encoding: binary` per part
- `Content-Type: application/http` per part
- `Content-Type: application/json; type=entry` for POST/PATCH bodies
- CRLF line endings throughout
- Absolute URLs in batch parts
- Empty changesets silently skipped (prevents invalid multipart)
- Top-level batch error handling (non-multipart 4xx/5xx raises
`HttpError` with parsed Dataverse error details)
- Accepts `200`, `202 Accepted`, `207 Multi-Status`, and `400` batch
response codes
### Review comment fixes
- Fixed `expected` status codes to include `202`/`207` for all Dataverse
environments
- Fixed `_split_multipart` / `_parse_mime_part` return type annotations:
`List[Tuple[Dict[str, str], str]]`
- Fixed OptionSet string check regression: now uses dict key lookup
instead of JSON string search
- Fixed `_build_get` to lowercase select column names (consistency with
`_get_multiple`)
- Added RFC 3986 `%20` encoding documentation in `_build_sql` docstring
- Fixed content-id response parsing for non-changeset parts
- Fixed test assertions after merge: `data` bytes instead of `json`
kwarg
- Exception type parity: `batch.records.upsert()` raises `TypeError`
(matching `client.records.upsert()`)
### Testing
**Unit tests -- 579 tests passing:**
- `test_batch_operations.py` -- BatchRequest, BatchRecordOperations,
BatchTableOperations, BatchQueryOperations, ChangeSet,
BatchItemResponse, BatchResult
- `test_batch_serialization.py` -- multipart serialization, response
parsing, intent resolution, upsert dispatch, batch size limit,
content-ID uniqueness, top-level error handling
- `test_batch_edge_cases.py` -- 40 edge case tests: empty changeset,
changeset rollback, content-ID in standalone parts, mixed batch,
multiple changesets, batch size limits, top-level errors,
continue-on-error, serialization compliance, multipart parsing,
content-ID references, intent validation
- `test_batch_dataframe.py` -- 18 tests: DataFrame create/update/delete,
validation, NaN handling, empty series, bulk delete
- `test_odata_internal.py` -- `_build_upsert_multiple` body exclusion,
conflict detection, URL/method correctness
**E2E tests -- 14 tests passing against live Dataverse
(`crm10.dynamics.com`):**
1. Basic batch CRUD (single create + CreateMultiple, update, get,
delete)
2. Changeset happy path (create + update via `$ref` content-ID)
3. Changeset rollback (failing op rolls back entire changeset)
4. Multiple changesets (globally unique content-IDs)
5. Continue-on-error (mixed success/failure)
6. Batch SQL query
7. Batch tables.get + tables.list
8. DataFrame batch create
9. DataFrame batch update
10. DataFrame batch delete
11. Mixed batch (changeset + standalone GET)
12. Empty changeset (silently skipped)
13. Content-ID chaining (2 creates + 2 updates via `$ref`)
14. Table setup/teardown
### Examples & docs
- `examples/advanced/batch.py` -- reference examples for all batch
operation types
- `examples/advanced/walkthrough.py` -- batch section added (section 11)
- `examples/basic/functional_testing.py` --
`test_batch_all_operations()` covering all operation categories against
a live environment
---------
Co-authored-by: Samson Gebre <sagebree@microsoft.com>
Co-authored-by: Saurabh Badenkal <sbadenkal@microsoft.com>1 parent 5a395ec commit 5cd086c
24 files changed
Lines changed: 6381 additions & 217 deletions
File tree
- .claude/skills
- dataverse-sdk-dev
- dataverse-sdk-use
- examples
- advanced
- basic
- src/PowerPlatform/Dataverse
- claude_skill
- dataverse-sdk-dev
- dataverse-sdk-use
- data
- models
- operations
- tests/unit
- data
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
16 | | - | |
| 16 | + | |
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
25 | 26 | | |
26 | 27 | | |
27 | 28 | | |
| |||
369 | 370 | | |
370 | 371 | | |
371 | 372 | | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
372 | 417 | | |
373 | 418 | | |
374 | 419 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
| |||
43 | 44 | | |
44 | 45 | | |
45 | 46 | | |
| 47 | + | |
46 | 48 | | |
47 | 49 | | |
48 | 50 | | |
| |||
115 | 117 | | |
116 | 118 | | |
117 | 119 | | |
118 | | - | |
| 120 | + | |
119 | 121 | | |
120 | | - | |
| 122 | + | |
121 | 123 | | |
122 | 124 | | |
123 | 125 | | |
| |||
513 | 515 | | |
514 | 516 | | |
515 | 517 | | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
516 | 602 | | |
517 | 603 | | |
518 | 604 | | |
| |||
527 | 613 | | |
528 | 614 | | |
529 | 615 | | |
| 616 | + | |
530 | 617 | | |
531 | 618 | | |
532 | 619 | | |
| |||
0 commit comments