Skip to content

Commit 52735bd

Browse files
Merge pull request #7 from microsoft/user/tpellissier/bulkops-followups
Prefix and logicalname fixes; readme/docstring updates
2 parents c7b16a6 + 5be7b22 commit 52735bd

4 files changed

Lines changed: 176 additions & 55 deletions

File tree

README.md

Lines changed: 22 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ A minimal Python SDK to use Microsoft Dataverse as a database for Azure AI Found
44

55
- Read (SQL) — Execute read-only T‑SQL via the McpExecuteSqlQuery Custom API. Returns `list[dict]`.
66
- OData CRUD — Thin wrappers over Dataverse Web API (create/get/update/delete).
7-
- Bulk create — Pass a list of records to `create(...)` to invoke the bound `CreateMultiple` action; returns `list[str]` of GUIDs.
7+
- Bulk create — Pass a list of records to `create(...)` to invoke the bound `CreateMultiple` action; returns `list[str]` of GUIDs. If `@odata.type` is absent the SDK resolves the logical name from metadata (cached).
88
- Retrieve multiple (paging) — Generator-based `get_multiple(...)` that yields pages, supports `$top` and Prefer: `odata.maxpagesize` (`page_size`).
99
- Metadata helpers — Create/inspect/delete simple custom tables (EntityDefinitions + Attributes).
1010
- Pandas helpers — Convenience DataFrame oriented wrappers for quick prototyping/notebooks.
@@ -127,6 +127,9 @@ print({"created_ids": ids})
127127
Notes:
128128
- The bulk create response typically includes IDs only; the SDK returns the list of GUID strings.
129129
- Single-record `create` still returns the full entity representation.
130+
- `@odata.type` handling: If any payload in the list omits `@odata.type`, the SDK performs a one-time metadata query (`EntityDefinitions?$filter=EntitySetName eq '<entity_set>'`) to resolve the logical name, caches it, and stamps each missing item with `Microsoft.Dynamics.CRM.<logical>`. If **all** payloads already include `@odata.type`, no metadata call is made.
131+
- The metadata lookup is per entity set and reused across subsequent multi-create calls in the same client instance (in-memory cache only).
132+
130133

131134
## Retrieve multiple with paging
132135

@@ -149,16 +152,23 @@ for page in pages: # each page is a list[dict]
149152
print({"total_rows": total})
150153
```
151154

152-
Parameters
153-
- `entity`: str — Entity set name (plural logical name), e.g., `"accounts"`.
154-
- `select`: list[str] | None — Columns to include; joined into `$select`.
155-
- `filter`: str | None — OData `$filter` expression (e.g., `"contains(name,'Acme') and statecode eq 0"`).
156-
- `orderby`: list[str] | None — Sort expressions (e.g., `["name asc", "createdon desc"]`).
157-
- `top`: int | None — Cap on total rows across all pages (sent as `$top` on first request).
158-
- `expand`: list[str] | None — Navigation expansions as raw OData strings. Example:
159-
- `"primarycontactid($select=fullname,emailaddress1)"`
160-
- `"parentaccountid($select=name)"`
161-
- `page_size`: int | None — Per-page hint via Prefer: `odata.maxpagesize=N`.
155+
Parameters (all optional except `entity_set`)
156+
- `entity_set`: str — Entity set (plural logical name), e.g., `"accounts"`.
157+
- `select`: list[str] | None — Columns -> `$select` (comma joined).
158+
- `filter`: str | None — OData `$filter` expression (e.g., `contains(name,'Acme') and statecode eq 0`).
159+
- `orderby`: list[str] | None — Sort expressions -> `$orderby` (comma joined).
160+
- `top`: int | None — Global cap via `$top` (applied on first request; service enforces across pages).
161+
- `expand`: list[str] | None — Navigation expansions -> `$expand`; pass raw clauses (e.g., `primarycontactid($select=fullname,emailaddress1)`).
162+
- `page_size`: int | None — Per-page hint using Prefer: `odata.maxpagesize=<N>` (not guaranteed; last page may be smaller).
163+
164+
Return value & semantics
165+
- `$select`, `$filter`, `$orderby`, `$expand`, `$top` map directly to corresponding OData query options on the first request.
166+
- `$top` caps total rows; the service may partition those rows across multiple pages.
167+
- `page_size` (Prefer: `odata.maxpagesize`) is a hint; the server decides actual page boundaries.
168+
- Returns a generator yielding non-empty pages (`list[dict]`). Empty pages are skipped.
169+
- Each yielded list corresponds to a `value` page from the Web API.
170+
- Iteration stops when no `@odata.nextLink` remains (or when `$top` satisfied server-side).
171+
- The generator does not materialize all results; pages are fetched lazily.
162172

163173
Example (all parameters + expected response)
164174

@@ -192,11 +202,6 @@ for page in pages: # page is list[dict]
192202
print({"page_size": len(page)})
193203
```
194204

195-
Semantics:
196-
- `$top`: caps the total number of rows returned across all pages.
197-
- `page_size`: per-page size hint via Prefer: `odata.maxpagesize`; the service may return fewer/more.
198-
- The generator follows `@odata.nextLink` until exhausted or `$top` is satisfied.
199-
```
200205

201206
### Custom table (metadata) example
202207

@@ -248,7 +253,7 @@ VS Code Tasks
248253
- No general-purpose OData batching, upsert, or association operations yet.
249254
- `DeleteMultiple`/`UpdateMultiple` are not exposed; quickstart may demonstrate faster deletes using client-side concurrency only.
250255
- Minimal retry policy in library (network-error only); examples include additional backoff for transient Dataverse consistency.
251-
- Entity naming conventions in Dataverse (schema/logical/entity set plural & publisher prefix) using the SDK is currently not well-defined
256+
- Entity naming conventions in Dataverse: for multi-create the SDK resolves logical names from entity set metadata.
252257

253258
## Contributing
254259

examples/quickstart.py

Lines changed: 13 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@
2121
sys.exit(1)
2222

2323
base_url = entered.rstrip('/')
24-
delete_choice = input("Delete the SampleItem table at end? (Y/n): ").strip() or "y"
24+
delete_choice = input("Delete the new_SampleItem table at end? (Y/n): ").strip() or "y"
2525
delete_table_at_end = (str(delete_choice).lower() in ("y", "yes", "true", "1"))
2626
# Ask once whether to pause between steps during this run
2727
pause_choice = input("Pause between test steps? (y/N): ").strip() or "n"
@@ -68,11 +68,12 @@ def backoff_retry(op, *, delays=(0, 2, 5, 10, 20), retry_http_statuses=(400, 403
6868
table_info = None
6969
created_this_run = False
7070

71-
# First check for existing table
72-
log_call("client.get_table_info('SampleItem')")
73-
existing = client.get_table_info("SampleItem")
74-
if existing:
75-
table_info = existing
71+
# Check for existing table using list_tables
72+
log_call("client.list_tables()")
73+
tables = client.list_tables()
74+
existing_table = next((t for t in tables if t.get("SchemaName") == "new_SampleItem"), None)
75+
if existing_table:
76+
table_info = client.get_table_info("new_SampleItem")
7677
created_this_run = False
7778
print({
7879
"table": table_info.get("entity_schema"),
@@ -85,9 +86,9 @@ def backoff_retry(op, *, delays=(0, 2, 5, 10, 20), retry_http_statuses=(400, 403
8586
else:
8687
# Create it since it doesn't exist
8788
try:
88-
log_call("client.create_table('SampleItem', schema={code,count,amount,when,active})")
89+
log_call("client.create_table('new_SampleItem', schema={code,count,amount,when,active})")
8990
table_info = client.create_table(
90-
"SampleItem",
91+
"new_SampleItem",
9192
{
9293
"code": "string",
9394
"count": "int",
@@ -415,11 +416,11 @@ def _del_one(rid: str) -> tuple[str, bool, str | None]:
415416
print("Cleanup (Metadata):")
416417
if delete_table_at_end:
417418
try:
418-
log_call("client.get_table_info('SampleItem')")
419-
info = client.get_table_info("SampleItem")
419+
log_call("client.get_table_info('new_SampleItem')")
420+
info = client.get_table_info("new_SampleItem")
420421
if info:
421-
log_call("client.delete_table('SampleItem')")
422-
client.delete_table("SampleItem")
422+
log_call("client.delete_table('new_SampleItem')")
423+
client.delete_table("new_SampleItem")
423424
print({"table_deleted": True})
424425
else:
425426
print({"table_deleted": False, "reason": "not found"})

src/dataverse_sdk/client.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -227,6 +227,16 @@ def delete_table(self, tablename: str) -> None:
227227
"""
228228
self._get_odata().delete_table(tablename)
229229

230+
def list_tables(self) -> list[str]:
231+
"""List all custom tables in the Dataverse environment.
232+
233+
Returns
234+
-------
235+
list[str]
236+
A list of table names.
237+
"""
238+
return self._get_odata().list_tables()
239+
230240

231241
__all__ = ["DataverseClient"]
232242

src/dataverse_sdk/odata.py

Lines changed: 131 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@ def __init__(self, auth, base_url: str, config=None) -> None:
2222
backoff=self.config.http_backoff,
2323
timeout=self.config.http_timeout,
2424
)
25+
# Cache: entity set name -> logical name (resolved via metadata lookup)
26+
self._entityset_logical_cache = {}
2527

2628
def _headers(self) -> Dict[str, str]:
2729
"""Build standard OData headers with bearer auth."""
@@ -42,13 +44,28 @@ def _request(self, method: str, url: str, **kwargs):
4244
def create(self, entity_set: str, data: Union[Dict[str, Any], List[Dict[str, Any]]]) -> Union[Dict[str, Any], List[str]]:
4345
"""Create one or many records.
4446
45-
Behaviour:
46-
- Single (dict): POST /{entity_set} with Prefer: return=representation and return the created record (dict).
47-
- Multiple (list[dict]): POST bound action /{entity_set}/Microsoft.Dynamics.CRM.CreateMultiple
48-
and return a list[str] of created record GUIDs (server only returns Ids for this action).
49-
50-
@odata.type is auto-inferred (Microsoft.Dynamics.CRM.<logical>) for each item when doing multi-create
51-
if not already supplied.
47+
Parameters
48+
----------
49+
entity_set : str
50+
Entity set (plural logical name), e.g. "accounts".
51+
data : dict | list[dict]
52+
Single entity payload or list of payloads for batch create.
53+
54+
Behaviour
55+
---------
56+
- Single (dict): POST /{entity_set} with Prefer: return=representation. Returns created record (dict).
57+
- Multiple (list[dict]): POST /{entity_set}/Microsoft.Dynamics.CRM.CreateMultiple. Returns list[str] of created GUIDs.
58+
59+
Multi-create logical name resolution
60+
------------------------------------
61+
- If any payload omits ``@odata.type`` the client performs a metadata lookup (once per entity set, cached)
62+
to resolve the logical name and stamps ``Microsoft.Dynamics.CRM.<logical>`` into missing payloads.
63+
- If all payloads already include ``@odata.type`` no lookup or modification occurs.
64+
65+
Returns
66+
-------
67+
dict | list[str]
68+
Created entity (single) or list of created IDs (multi).
5269
"""
5370
if isinstance(data, dict):
5471
return self._create_single(entity_set, data)
@@ -71,17 +88,44 @@ def _create_single(self, entity_set: str, record: Dict[str, Any]) -> Dict[str, A
7188
except ValueError:
7289
return {}
7390

74-
def _infer_logical(self, entity_set: str) -> str:
75-
# Basic heuristic: drop trailing 's'. (Metadata lookup could be added later.)
76-
return entity_set[:-1] if entity_set.endswith("s") else entity_set
91+
def _logical_from_entity_set(self, entity_set: str) -> str:
92+
"""Resolve logical name from an entity set using metadata (cached)."""
93+
es = (entity_set or "").strip()
94+
if not es:
95+
raise ValueError("entity_set is required")
96+
cached = self._entityset_logical_cache.get(es)
97+
if cached:
98+
return cached
99+
url = f"{self.api}/EntityDefinitions"
100+
params = {
101+
"$select": "LogicalName,EntitySetName",
102+
"$filter": f"EntitySetName eq '{es}'",
103+
}
104+
r = self._request("get", url, headers=self._headers(), params=params)
105+
r.raise_for_status()
106+
try:
107+
body = r.json()
108+
items = body.get("value", []) if isinstance(body, dict) else []
109+
except ValueError:
110+
items = []
111+
if not items:
112+
raise RuntimeError(f"Unable to resolve logical name for entity set '{es}'. Provide @odata.type explicitly.")
113+
logical = items[0].get("LogicalName")
114+
if not logical:
115+
raise RuntimeError(f"Metadata response missing LogicalName for entity set '{es}'.")
116+
self._entityset_logical_cache[es] = logical
117+
return logical
77118

78119
def _create_multiple(self, entity_set: str, records: List[Dict[str, Any]]) -> List[str]:
79120
if not all(isinstance(r, dict) for r in records):
80121
raise TypeError("All items for multi-create must be dicts")
81-
logical = self._infer_logical(entity_set)
122+
need_logical = any("@odata.type" not in r for r in records)
123+
logical: Optional[str] = None
124+
if need_logical:
125+
logical = self._logical_from_entity_set(entity_set)
82126
enriched: List[Dict[str, Any]] = []
83127
for r in records:
84-
if "@odata.type" in r:
128+
if "@odata.type" in r or not logical:
85129
enriched.append(r)
86130
else:
87131
nr = r.copy()
@@ -104,7 +148,7 @@ def _create_multiple(self, entity_set: str, records: List[Dict[str, Any]]) -> Li
104148
ids = body.get("Ids")
105149
if isinstance(ids, list):
106150
return [i for i in ids if isinstance(i, str)]
107-
# Future-proof: some environments might eventually return value/list of entities.
151+
108152
value = body.get("value")
109153
if isinstance(value, list):
110154
# Extract IDs if possible
@@ -128,6 +172,22 @@ def _format_key(self, key: str) -> str:
128172
return f"({k})"
129173

130174
def update(self, entity_set: str, key: str, data: Dict[str, Any]) -> Dict[str, Any]:
175+
"""Update an existing record and return the updated representation.
176+
177+
Parameters
178+
----------
179+
entity_set : str
180+
Entity set name (plural logical name).
181+
key : str
182+
Record GUID (with or without parentheses) or alternate key.
183+
data : dict
184+
Partial entity payload.
185+
186+
Returns
187+
-------
188+
dict
189+
Updated record representation.
190+
"""
131191
url = f"{self.api}/{entity_set}{self._format_key(key)}"
132192
headers = self._headers().copy()
133193
headers["If-Match"] = "*"
@@ -137,13 +197,25 @@ def update(self, entity_set: str, key: str, data: Dict[str, Any]) -> Dict[str, A
137197
return r.json()
138198

139199
def delete(self, entity_set: str, key: str) -> None:
200+
"""Delete a record by GUID or alternate key."""
140201
url = f"{self.api}/{entity_set}{self._format_key(key)}"
141202
headers = self._headers().copy()
142203
headers["If-Match"] = "*"
143204
r = self._request("delete", url, headers=headers)
144205
r.raise_for_status()
145206

146207
def get(self, entity_set: str, key: str, select: Optional[str] = None) -> Dict[str, Any]:
208+
"""Retrieve a single record.
209+
210+
Parameters
211+
----------
212+
entity_set : str
213+
Entity set name.
214+
key : str
215+
Record GUID (with or without parentheses) or alternate key syntax.
216+
select : str | None
217+
Comma separated columns for $select.
218+
"""
147219
params = {}
148220
if select:
149221
params["$select"] = select
@@ -178,22 +250,20 @@ def get_multiple(
178250
Max number of records across all pages. Passed as $top on the first request; the server will paginate via nextLink as needed.
179251
expand : list[str] | None
180252
Navigation properties to expand; joined with commas into $expand.
253+
page_size : int | None
254+
Hint for per-page size using Prefer: ``odata.maxpagesize``.
181255
182256
Yields
183257
------
184258
list[dict]
185259
A page of records from the Web API (the "value" array for each page).
186260
"""
187261

188-
# Build headers once; include odata.maxpagesize to force smaller pages for demos/testing
189262
headers = self._headers().copy()
190263
if page_size is not None:
191-
try:
192-
ps = int(page_size)
193-
if ps > 0:
194-
headers["Prefer"] = f"odata.maxpagesize={ps}"
195-
except Exception:
196-
pass
264+
ps = int(page_size)
265+
if ps > 0:
266+
headers["Prefer"] = f"odata.maxpagesize={ps}"
197267

198268
def _do_request(url: str, *, params: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
199269
r = self._request("get", url, headers=headers, params=params)
@@ -234,6 +304,23 @@ def _do_request(url: str, *, params: Optional[Dict[str, Any]] = None) -> Dict[st
234304

235305
# --------------------------- SQL Custom API -------------------------
236306
def query_sql(self, tsql: str) -> list[dict[str, Any]]:
307+
"""Execute a read-only T-SQL query via the configured Custom API.
308+
309+
Parameters
310+
----------
311+
tsql : str
312+
SELECT-style Dataverse-supported T-SQL (read-only).
313+
314+
Returns
315+
-------
316+
list[dict]
317+
Rows materialised as list of dictionaries (empty list if no rows).
318+
319+
Raises
320+
------
321+
RuntimeError
322+
If the Custom API response is missing the expected ``queryresult`` property or type is unexpected.
323+
"""
237324
payload = {"querytext": tsql}
238325
headers = self._headers()
239326
api_name = self.config.sql_api_name
@@ -392,20 +479,38 @@ def _attribute_payload(self, schema_name: str, dtype: str, *, is_primary_name: b
392479
return None
393480

394481
def get_table_info(self, tablename: str) -> Optional[Dict[str, Any]]:
395-
# Accept tablename as a display/logical root; infer a default schema using 'new_' if not provided.
396-
# If caller passes a full SchemaName, use it as-is.
397-
schema_name = tablename if "_" in tablename else f"new_{self._to_pascal(tablename)}"
398-
entity_schema = schema_name
399-
ent = self._get_entity_by_schema(entity_schema)
482+
"""Return basic metadata for a custom table if it exists.
483+
484+
Parameters
485+
----------
486+
tablename : str
487+
Friendly name or full schema name (with publisher prefix and underscore).
488+
489+
Returns
490+
-------
491+
dict | None
492+
Metadata summary or ``None`` if not found.
493+
"""
494+
ent = self._get_entity_by_schema(tablename)
400495
if not ent:
401496
return None
402497
return {
403-
"entity_schema": ent.get("SchemaName") or entity_schema,
498+
"entity_schema": ent.get("SchemaName") or tablename,
404499
"entity_logical_name": ent.get("LogicalName"),
405500
"entity_set_name": ent.get("EntitySetName"),
406501
"metadata_id": ent.get("MetadataId"),
407502
"columns_created": [],
408503
}
504+
505+
def list_tables(self) -> List[Dict[str, Any]]:
506+
"""List all tables in the Dataverse, excluding private tables (IsPrivate=true)."""
507+
url = f"{self.api}/EntityDefinitions"
508+
params = {
509+
"$filter": "IsPrivate eq false"
510+
}
511+
r = self._request("get", url, headers=self._headers(), params=params)
512+
r.raise_for_status()
513+
return r.json().get("value", [])
409514

410515
def delete_table(self, tablename: str) -> None:
411516
schema_name = tablename if "_" in tablename else f"new_{self._to_pascal(tablename)}"

0 commit comments

Comments
 (0)