Merge origin/main into users/sagebree/issue157_fix_sql_truncation

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: Samson Gebre

.claude/skills/dataverse-sdk-dev/SKILL.md

@@ -20,6 +20,32 @@ This skill provides guidance for developers working on the PowerPlatform Dataver
 5. **Consider backwards compatibility** - Avoid breaking changes
 6. **Internal vs public naming** - Modules, files, and functions not meant to be part of the public API must use a `_` prefix (e.g., `_odata.py`, `_relationships.py`). Files without the prefix (e.g., `constants.py`, `metadata.py`) are public and importable by SDK consumers
 
+### Dataverse Property Naming Rules
+
+Dataverse uses two different naming conventions for properties. Getting this wrong causes 400 errors that are hard to debug.
+
+| Property type | Name convention | Example | When used |
+|---|---|---|---|
+| **Structural** (columns) | LogicalName (always lowercase) | `new_name`, `new_priority` | `$select`, `$filter`, `$orderby`, record payload keys |
+| **Navigation** (relationships / lookups) | Navigation Property Name (usually SchemaName, PascalCase, case-sensitive) | `new_CustomerId`, `new_AgentId` | `$expand`, `@odata.bind` annotation keys |
+
+Navigation property names are case-sensitive and must match the entity's `$metadata`. Using the logical name instead of the navigation property name results in 400 Bad Request errors.
+
+**Critical rule:** The OData parser validates `@odata.bind` property names **case-sensitively** against declared navigation properties. Lowercasing `new_CustomerId@odata.bind` to `new_customerid@odata.bind` causes: `ODataException: An undeclared property 'new_customerid' which only has property annotations...`
+
+**SDK implementation:**
+
+- `_lowercase_keys()` lowercases all keys EXCEPT those containing `@odata.` (preserves navigation property casing in `@odata.bind` keys)
+- `_lowercase_list()` lowercases `$select` and `$orderby` params (structural properties)
+- `$expand` params are passed as-is (navigation properties, PascalCase)
+- `_convert_labels_to_ints()` skips `@odata.` keys entirely (they are annotations, not attributes)
+
+**When adding new code that processes record dicts or builds query parameters:**
+
+- Always use `_lowercase_keys()` for record payloads. Never manually call `.lower()` on all keys
+- Never lowercase `$expand` values or `@odata.bind` key prefixes
+- If iterating record keys, skip keys containing `@odata.` when doing attribute-level operations
+
 ### Code Style
 
 6. **No emojis** - Do not use emoji in code, comments, or output
@@ -28,3 +54,35 @@ This skill provides guidance for developers working on the PowerPlatform Dataver
 9. **Document public APIs** - Add Sphinx-style docstrings with examples for public methods
 10. **Define __all__ in module files** - Each module declares its own exports via `__all__` (e.g., `errors.py` defines `__all__ = ["HttpError", ...]`). Package `__init__.py` files should not re-export or redefine another module's `__all__`; they use `__all__ = []` to indicate no star-import exports.
 11. **Run black before committing** - Always run `python -m black <changed files>` before committing. CI will reject unformatted code. Config is in `pyproject.toml` under `[tool.black]`.
+
+### Docstring Type Annotations (Microsoft Learn Compatibility)
+
+This SDK's API reference is published on Microsoft Learn. The Learn doc pipeline parses `:type:` and `:rtype:` directives differently from standard Sphinx -- every word between `:class:` references is treated as a separate cross-reference (`<xref:word>`). Using Sphinx-style `:class:\`list\` of :class:\`str\`` produces broken `<xref:of>` links on Learn.
+
+**Rules for `:type:` and `:rtype:` directives:**
+
+- Use Python bracket notation for generic types: `list[str]`, `dict[str, typing.Any]`, `list[dict]`
+- Use `or` (without `:class:`) for union types: `str or None`, `dict or list[dict]`
+- Use bracket nesting for complex types: `collections.abc.Iterable[list[dict]]`
+- Use `~` prefix for SDK types to show short name: `list[~PowerPlatform.Dataverse.models.record.Record]`
+- `:class:` is fine for single standalone types: `:class:\`str\``, `:class:\`bool\``
+
+**Never** use `:class:\`X\` of :class:\`Y\`` or `:class:\`X\` mapping :class:\`Y\` to :class:\`Z\`` -- the words `of`, `mapping`, `to` become broken `<xref:>` links.
+
+**Correct examples:**
+
+```rst
+:type data: dict or list[dict]
+:rtype: list[str]
+:rtype: collections.abc.Iterable[list[~PowerPlatform.Dataverse.models.record.Record]]
+:type select: list[str] or None
+:type columns: dict[str, typing.Any]
+```
+
+**Wrong examples (NEVER use):**
+
+```rst
+:type data: :class:`dict` or :class:`list` of :class:`dict`
+:rtype: :class:`list` of :class:`str`
+:type columns: :class:`dict` mapping :class:`str` to :class:`typing.Any`
+```

.claude/skills/dataverse-sdk-use/SKILL.md

@@ -31,6 +31,9 @@ The SDK supports Dataverse's native bulk operations: Pass lists to `create()`, `
 - Control page size with `page_size` parameter
 - Use `top` parameter to limit total records returned
 
+### DataFrame Support
+- DataFrame operations are accessed via the `client.dataframe` namespace: `client.dataframe.get()`, `client.dataframe.create()`, `client.dataframe.update()`, `client.dataframe.delete()`
+
 ## Common Operations
 
 ### Import
@@ -54,7 +57,13 @@ credential = AzureCliCredential()
 credential = ClientSecretCredential(tenant_id, client_id, client_secret)
 credential = CertificateCredential(tenant_id, client_id, cert_path)
 
-# Create client (no trailing slash on URL!)
+# Create client with context manager (recommended -- enables HTTP connection pooling)
+# No trailing slash on URL!
+with DataverseClient("https://yourorg.crm.dynamics.com", credential) as client:
+    ...  # all operations here
+# Session closed, caches cleared automatically
+
+# Or without context manager:
 client = DataverseClient("https://yourorg.crm.dynamics.com", credential)
 ```
 
@@ -100,6 +109,20 @@ for page in client.records.get(
         print(f"{account['name']} - {contact.get('fullname', 'N/A')}")
 ```
 
+#### Create Records with Lookup Bindings (@odata.bind)
+```python
+# Set lookup fields using @odata.bind with PascalCase navigation property names
+# CORRECT: use the navigation property name (case-sensitive, must match $metadata)
+guid = client.records.create("new_ticket", {
+    "new_name": "TKT-001",
+    "new_CustomerId@odata.bind": f"/new_customers({customer_id})",
+    "new_AgentId@odata.bind": f"/new_agents({agent_id})",
+})
+
+# WRONG: lowercase navigation property causes 400 error
+# "new_customerid@odata.bind" -> ODataException: undeclared property 'new_customerid'
+```
+
 #### Update Records
 ```python
 # Single update
@@ -110,7 +133,7 @@ client.records.update("account", [id1, id2, id3], {"industry": "Technology"})
 ```
 
 #### Upsert Records
-Creates or updates records identified by alternate keys. Single item → PATCH; multiple items → `UpsertMultiple` bulk action.
+Creates or updates records identified by alternate keys. Single item -> PATCH; multiple items -> `UpsertMultiple` bulk action.
 > **Prerequisite**: The table must have an alternate key configured in Dataverse for the columns used in `alternate_key`. Without it, Dataverse will reject the request with a 400 error.
 ```python
 from PowerPlatform.Dataverse.models.upsert import UpsertItem
@@ -152,6 +175,42 @@ client.records.delete("account", account_id)
 client.records.delete("account", [id1, id2, id3], use_bulk_delete=True)
 ```
 
+### DataFrame Operations
+
+The SDK provides DataFrame wrappers for all CRUD operations via the `client.dataframe` namespace, using pandas DataFrames and Series as input/output.
+
+```python
+import pandas as pd
+
+# Query records -- returns a single DataFrame
+df = client.dataframe.get("account", filter="statecode eq 0", select=["name"])
+print(f"Got {len(df)} rows")
+
+# Limit results with top for large tables
+df = client.dataframe.get("account", select=["name"], top=100)
+
+# Fetch single record as one-row DataFrame
+df = client.dataframe.get("account", record_id=account_id, select=["name"])
+
+# Create records from a DataFrame (returns a Series of GUIDs)
+new_accounts = pd.DataFrame([
+    {"name": "Contoso", "telephone1": "555-0100"},
+    {"name": "Fabrikam", "telephone1": "555-0200"},
+])
+new_accounts["accountid"] = client.dataframe.create("account", new_accounts)
+
+# Update records from a DataFrame (id_column identifies the GUID column)
+new_accounts["telephone1"] = ["555-0199", "555-0299"]
+client.dataframe.update("account", new_accounts, id_column="accountid")
+
+# Clear a field by setting clear_nulls=True (by default, NaN/None fields are skipped)
+df = pd.DataFrame([{"accountid": "guid-1", "websiteurl": None}])
+client.dataframe.update("account", df, id_column="accountid", clear_nulls=True)
+
+# Delete records by passing a Series of GUIDs
+client.dataframe.delete("account", new_accounts["accountid"])
+```
+
 ### SQL Queries
 
 SQL queries are **read-only** and support limited SQL syntax. A single SELECT statement with optional WHERE, TOP (integer literal), ORDER BY (column names only), and a simple table alias after FROM is supported. But JOIN and subqueries may not be. Refer to the Dataverse documentation for the current feature set.
@@ -348,7 +407,7 @@ print(f"Succeeded: {len(result.succeeded)}, Failed: {len(result.failed)}")
 - `result.succeeded` -- responses with 2xx status codes
 - `result.failed` -- responses with non-2xx status codes
 - `result.has_errors` -- True if any response failed
-- `result.created_ids` -- GUIDs from successful create operations
+- `result.entity_ids` -- GUIDs from OData-EntityId headers (creates and updates)
 
 **Batch limitations:**
 - Maximum 1000 operations per batch
@@ -398,6 +457,7 @@ except ValidationError as e:
 - Check filter/expand parameters use correct case
 - Verify column names exist and are spelled correctly
 - Ensure custom columns include customization prefix
+- For `@odata.bind` errors ("undeclared property"): the navigation property name before `@odata.bind` is case-sensitive and must match the entity's `$metadata` exactly (e.g., `new_CustomerId@odata.bind` for custom lookups, `parentaccountid@odata.bind` for system lookups). The SDK preserves `@odata.bind` key casing.
 
 ## Best Practices
 
@@ -410,7 +470,7 @@ except ValidationError as e:
 5. **Use production credentials** - ClientSecretCredential or CertificateCredential for unattended operations
 6. **Error handling** - Implement retry logic for transient errors (`e.is_transient`)
 7. **Always include customization prefix** for custom tables/columns
-8. **Use lowercase** - Generally using lowercase input won't go wrong, except for custom table/column naming
+8. **Use lowercase for column names, match `$metadata` for navigation properties** - Column names in `$select`/`$filter`/record payloads use lowercase LogicalNames. Navigation properties in `$expand` and `@odata.bind` keys are case-sensitive and must match the entity's `$metadata` (PascalCase for custom lookups like `new_CustomerId`, lowercase for system lookups like `parentaccountid`)
 9. **Test in non-production environments** first
 10. **Use named constants** - Import cascade behavior constants from `PowerPlatform.Dataverse.common.constants`
 

.gitignore

@@ -25,3 +25,4 @@ Thumbs.db
 
 # Claude local settings
 .claude/*.local.json
+.claude/*.local.md

CHANGELOG.md

@@ -10,6 +10,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Fixed
 - `client.query.sql()` silently truncated results at 5,000 rows. The method now follows `@odata.nextLink` pagination and returns all matching rows (#157).
 
+## [0.1.0b7] - 2026-03-17
+
+### Added
+- DataFrame namespace: `client.dataframe.get()`, `.create()`, `.update()`, `.delete()` for working with Dataverse records as pandas DataFrames and Series — no manual dict conversion required (#98)
+- Table metadata now includes `primary_name_attribute` and `primary_id_attribute` from `tables.create()` and `tables.get_info()` (#148)
+
+### Changed
+- `pandas>=2.0.0` is now a required dependency (#98)
+
+## [0.1.0b6] - 2026-03-12
+
+### Added
+- Context manager support: `with DataverseClient(...) as client:` for automatic resource cleanup, HTTP connection pooling, and `close()` for explicit lifecycle management (#117)
+- Typed return models `Record`, `TableInfo`, and `ColumnInfo` for record and table metadata operations, replacing raw `Dict[str, Any]` returns with full backward compatibility (`result["key"]` still works) (#115)
+- Alternate key management: `client.tables.create_alternate_key()`, `client.tables.get_alternate_keys()`, `client.tables.delete_alternate_key()` with typed `AlternateKeyInfo` model (#126)
+
+### Fixed
+- `@odata.bind` lookup bindings now preserve navigation property casing (e.g., `new_CustomerId@odata.bind`), fixing `400 Bad Request` errors on create/update/upsert with lookup fields (#137)
+- Reduced unnecessary HTTP round-trips on create/update/upsert when records contain `@odata.bind` keys (#137)
+- Single-record `get()` now lowercases `$select` column names consistently with multi-record queries (#137)
+
 ## [0.1.0b5] - 2026-02-27
 
 ### Fixed
@@ -75,6 +96,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Comprehensive error handling with specific exception types (`DataverseError`, `AuthenticationError`, etc.) (#22, #24)
 - HTTP retry logic with exponential backoff for resilient operations (#72)
 
+[0.1.0b7]: https://github.com/microsoft/PowerPlatform-DataverseClient-Python/compare/v0.1.0b6...v0.1.0b7
+[0.1.0b6]: https://github.com/microsoft/PowerPlatform-DataverseClient-Python/compare/v0.1.0b5...v0.1.0b6
 [0.1.0b5]: https://github.com/microsoft/PowerPlatform-DataverseClient-Python/compare/v0.1.0b4...v0.1.0b5
 [0.1.0b4]: https://github.com/microsoft/PowerPlatform-DataverseClient-Python/compare/v0.1.0b3...v0.1.0b4
 [0.1.0b3]: https://github.com/microsoft/PowerPlatform-DataverseClient-Python/compare/v0.1.0b2...v0.1.0b3

CONTRIBUTING.md

@@ -119,6 +119,43 @@ published release:
 
 ```bash
 # After publishing v0.1.0b4, bump to v0.1.0b5 on main
-# Update both pyproject.toml and src/PowerPlatform/Dataverse/__version__.py
+# Update version in pyproject.toml
 # Commit directly to main: "Bump version to 0.1.0b5 for next development cycle"
+```
+
+### Docstring Type Annotations (Microsoft Learn Compatibility)
+
+This SDK's API reference is published on [Microsoft Learn](https://learn.microsoft.com). The Learn doc pipeline processes `:type:` and `:rtype:` Sphinx directives differently from standard Sphinx -- every word between `:class:` back-tick references is treated as a separate cross-reference (`<xref:word>`). For example:
+
+```
+:rtype: :class:`list` of :class:`str`
+```
+
+This produces a broken `<xref:of>` link because `of` is not a valid type.
+
+**Rules for `:type:` and `:rtype:` directives:**
+
+- Use **Python bracket notation** for generic types: `list[str]`, `dict[str, typing.Any]`, `list[dict]`
+- Use **`or`** (without `:class:`) for union types: `str or None`, `dict or list[dict]`
+- Use **bracket nesting** for complex types: `collections.abc.Iterable[list[dict]]`
+- `:class:` is fine for **single standalone types**: `` :class:`str` ``, `` :class:`bool` ``
+
+**NEVER** use the following patterns -- the connector words (`of`, `mapping`, `to`) become broken `<xref:>` links on Learn:
+
+```
+:class:`X` of :class:`Y`
+:class:`X` mapping :class:`Y` to :class:`Z`
+```
+
+Correct:
+```
+:type data: dict or list[dict]
+:rtype: list[str]
+:type select: list[str] or None
+```
+
+Wrong:
+```
+:type data: :class:`dict` or :class:`list` of :class:`dict`
+:rtype: :class:`list` of :class:`str`
 ```