diff --git a/.changes/unreleased/Added-20260721-140000.yaml b/.changes/unreleased/Added-20260721-140000.yaml new file mode 100644 index 00000000..28d91fd4 --- /dev/null +++ b/.changes/unreleased/Added-20260721-140000.yaml @@ -0,0 +1,6 @@ +kind: Added +body: 'Documentation site at https://rayakame.github.io/sqlc-gen-better-python/, with a getting started page, a guide covering configuration, drivers, model types, writing queries, enums, type overrides, converters, working with JSON, docstrings, SQLite type conversion and naming, and a reference for every configuration option, the SQL to Python type mappings of both engines and per-driver feature support. Every generated code example is taken from the committed test fixtures. The README now links to the site instead of documenting each feature inline.' +time: 2026-07-21T14:00:00.0000000Z +custom: + Author: Rayakame + PR: "208" diff --git a/README.md b/README.md index 9864d8b8..17b7b540 100644 --- a/README.md +++ b/README.md @@ -12,10 +12,15 @@ A WASM plugin for SQLC allowing the generation of Python code. The generated code requires **Python 3.12 or newer** (it uses PEP 695 type aliases and generics, and `enum.StrEnum`). -**Documentation: https://rayakame.github.io/sqlc-gen-better-python/** -(The docs site is new; some feature references still live in this README below -while they are ported over.) +## Documentation +**https://rayakame.github.io/sqlc-gen-better-python/** + +- [Getting Started](https://rayakame.github.io/sqlc-gen-better-python/docs/getting-started/) - install the plugin and generate your first models. +- [Guide](https://rayakame.github.io/sqlc-gen-better-python/docs/guide/) - configuration, drivers, model types, writing queries, and every feature, each with real generated output. +- [Reference](https://rayakame.github.io/sqlc-gen-better-python/docs/reference/) - all configuration options, SQL-to-Python type mappings, and per-driver feature support. + +Questions or feedback? Join the [Discord](https://discord.gg/hikari). > [!NOTE] > Every Release before `v1.0.0`, including this one is an beta release. @@ -25,7 +30,6 @@ while they are ported over.) > Since `v0.5.0` this includes full support for PostgreSQL enums and a fourth model type, `pydantic`. > Feel free to lmk any wanted features and I'm going to do my best on implementing them with the time I have rn. - ## Example Config ```yaml @@ -51,176 +55,36 @@ sql: ``` -More options at the [`sqlc` config reference](https://docs.sqlc.dev/en/stable/reference/config.html) - -## Configuration Options - -| Name | Type | Required | Description | -|----------------------------------|----------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `package` | string | yes | The name of the package where the generated files will be located | -| `emit_init_file` | bool | yes | If set to to `false` there will be no `__init__.py` file created in the package that you specified. Only set this to false if you know that you already have a `__init__.py` file otherwise the generated code wont work. | -| `sql_driver` | string | yes | The name of the sql driver you want to use. Valid options are listed [here](#feature-support) | -| `model_type` | string | no | The model type you want to use. This can be one of `dataclass`, `msgspec`, `attrs` or [`pydantic`](#pydantic-models). Defaults to `dataclass` | -| `initialisms` | list[string] | no | An array of [initialisms](https://google.github.io/styleguide/go/decisions.html#initialisms) to upper-case. For example, `app_id` becomes `AppID`. Defaults to `["id"]`. | -| `emit_exact_table_names` | bool | no | If `true`, model names will mirror table names. Otherwise sqlc attempts to singularize plural table names. | -| `emit_classes` | bool | no | If `true`, every query function will be put into a class called `Querier`. Otherwise every function will be a standalone function. | -| `inflection_exclude_table_names` | list[string] | no | An array of table names that should not be turned singular. Only applies if `emit_exact_table_names` is `false`. | -| `omit_unused_models` | bool | no | If set to `true` and there are models/tables that are not used in any query, they wont be turned into models. | -| `omit_typechecking_block` | bool | no | If set to `true`, will not wrap all non-builtin types behind a `typing.TYPE_CHECKING` block. Defaults to `false` | -| `docstrings` | string | no | If set, there will be docstrings generated in the selected format. This can be one of `google`, `numpy`, `pep257` and `none`. `none` will not generate any docstrings. | -| `docstrings_emit_sql` | bool | no | If set to `false` the SQL code for each query wont be included in the docstrings. This defaults to `true` but is not used when `docstrings` is not set or set to `none` | -| `query_parameter_limit` | integer | no | If set to a non-negative value, queries with more parameters than the limit get them bundled into a single `params: Params` argument instead of expanded arguments. If unset or negative, parameters are never bundled (`:copyfrom` always uses a Params class). | -| `omit_kwargs_limit` | integer | no | This can be used to set a limit where any query with less or equal amounts of parameters will not require kwargs for the parameters. This defaults to `0` which makes every query require kwargs for their parameters. | -| `speedups` | bool | no | If set to `true` the plugin will use other librarys for type conversion. Needs extra dependecys to be installed. This option currently only affects `sqlite3` & `aiosqlite` and uses the library `ciso8601` | -| `overrides` | list[Override] | no | A list of [type overrides](#type-overrides). | -| `converters` | list[Converter]| no | A list of [converters](#converters): named pairs of your own functions that serialize and deserialize a column value, referenced by an override. | -| `debug` | bool | no | If set to `true`, there will be debug logs generated into a `log.txt` file when executing `sqlc generate`. Defaults to `false` | - -### Type Overrides - -Similar to `sqlc-gen-go` this plugin supports overriding types with your own. You can either override the type of every -column that has a specific sql type, or you can overwrite the type of specific columns. - -```yaml -# filename: sqlc.yaml -# ... -options: - # ... - overrides: - - db_type: text - py_type: - import: collections - package: UserString - type: UserString - - column: table_name.text_column - py_type: - import: collections - type: collections.UserString - -``` - -### Converters - -An override replaces a column's type, but conversion is done by calling that type -(`Preferences(value)`), which does not work for things like JSON. A converter names -two of your own functions instead, used whenever the column is read or written: - -```yaml -# filename: sqlc.yaml -# ... -options: - # ... - converters: - - name: prefs - py_type: - import: myapp.models - package: Preferences - type: Preferences - to_db: myapp.converters.encode_preferences - from_db: myapp.converters.decode_preferences - overrides: - - db_type: jsonb - converter: prefs - - column: users.preferences - converter: prefs -``` - -```python -# generated -preferences=myapp.converters.decode_preferences(row[1]) # read -await conn.execute(CREATE_USER, id_, myapp.converters.encode_preferences(preferences)) # write -``` - -A converter is referenced by an override, so it applies to whichever columns that -override matches, and `converter` replaces `py_type` on the override itself. - -Both directions are required. `to_db` and `from_db` are dotted paths; their module is -imported and the function called fully qualified, so the names can never collide with -generated code. - -`to_db` must return the type the column would have had without the override (`jsonb` is -a `str`, `bytea` a `memoryview`), and `from_db` receives that same type. For a column -whose SQL type the plugin does not recognise there is no such type, so `to_db` must -return one the driver accepts. - -Your functions never see `None`: nullable columns are guarded, so a NULL stays `None` -without calling the converter. List columns convert element-wise. - -### Enums - -PostgreSQL enum types generate an `enums.py` module containing `enum.StrEnum` classes: - -```python -class Mood(enum.StrEnum): - SAD = "sad" - OK = "ok" - HAPPY = "happy" -``` - -Enum columns are typed with these classes in models and query functions, and values read -from the database are coerced into them. Enums in non-default schemas get schema-qualified -class names (e.g. `CustomMood` for `custom.mood`), so same-named enums never collide. - -### Identifier sanitization - -SQL identifiers that are not valid Python names are sanitized: invalid characters become -underscores, digit-leading columns/parameters get a `column_` prefix, table names that -would produce a digit-leading, keyword or empty class name get a `Model` prefix, digit- -or underscore-leading enum values get a `VALUE_` prefix, and colliding results are -deduplicated with numeric suffixes. Field names also prefix leading underscores, since -attrs and pydantic treat such fields as private. - -One caveat: sanitization checks characters with Go's Unicode tables, which differ from -Python's identifier rules in exotic cases (e.g. characters that may not START an -identifier in Python, or two names that Python normalizes to the same identifier via -NFKC). Such schemas are not detected; stick to ASCII identifiers if in doubt. - -### Pydantic models - -With `model_type: pydantic`, models are generated as `pydantic.BaseModel` subclasses -(requires `pydantic >= 2.9`). Two things to be aware of: - -- Every generated class sets `model_config = pydantic.ConfigDict(arbitrary_types_allowed=True)`. - This is required for field types pydantic has no core schema for (e.g. `memoryview` for - `bytea`/`blob` columns and custom override types); those fields are still validated with - an `isinstance` check, and all other fields get full pydantic validation on construction. -- Unlike the other model types, pydantic resolves field annotations at runtime, so the - generated files import the referenced modules at runtime instead of inside - `if typing.TYPE_CHECKING:` blocks. If you lint generated code with ruff, set - `lint.flake8-type-checking.runtime-evaluated-base-classes = ["pydantic.BaseModel"]`. - -### SQLite type conversion - -For the `sqlite3` and `aiosqlite` drivers, the generated code registers adapters and -converters (for `date`, `datetime`/`timestamp`, `decimal`, `bool` and `blob` columns) via -`register_adapter`/`register_converter`. For these to work, you must open your connection -with declared-type parsing enabled: - -```python -conn = sqlite3.connect(dsn, detect_types=sqlite3.PARSE_DECLTYPES) -``` - -## Feature Support - -Every [sqlc macro](https://docs.sqlc.dev/en/latest/reference/macros.html) is supported. -The supported [query commands](https://docs.sqlc.dev/en/latest/reference/query-annotations.html) depend on the SQL -driver you are using, supported commands are listed below. -> Every `:batch*` command is not supported by this plugin and probably will never be. - -> Prepared Queries are not planned for the near future, but will be implemented sooner or later - -| | `:exec` | `:execresult` | `:execrows` | `:execlastid` | `:many` | `:one` | `:copyfrom` | -|-----------|---------|---------------|-------------|---------------|---------|--------|-------------| -| aiosqlite | yes | yes | yes | yes | yes | yes | no | -| sqlite3 | yes | yes | yes | yes | yes | yes | no | -| asyncpg | yes | yes | yes | no | yes | yes | yes | -| psycopg2 | no | no | no | no | no | no | no | -| mysql | no | no | no | no | no | no | no | +More options at the [`sqlc` config reference](https://docs.sqlc.dev/en/stable/reference/config.html), +and the full plugin option list in the +[configuration reference](https://rayakame.github.io/sqlc-gen-better-python/docs/reference/configuration-options/). + +## Features + +- **Four model types** - `dataclass`, `attrs`, `msgspec`, or `pydantic` + ([docs](https://rayakame.github.io/sqlc-gen-better-python/docs/guide/model-types/)). +- **Three drivers** - `asyncpg` for PostgreSQL, `aiosqlite` and `sqlite3` for SQLite + ([docs](https://rayakame.github.io/sqlc-gen-better-python/docs/guide/drivers/)). +- **Typed query functions** - one module per query file, one function per query + ([docs](https://rayakame.github.io/sqlc-gen-better-python/docs/guide/writing-queries/)). +- **PostgreSQL enums** as `enum.StrEnum` classes + ([docs](https://rayakame.github.io/sqlc-gen-better-python/docs/guide/enums/)). +- **Type overrides and converters** - swap a column's Python type, or plug in your + own encode/decode functions + ([overrides](https://rayakame.github.io/sqlc-gen-better-python/docs/guide/type-overrides/), + [converters](https://rayakame.github.io/sqlc-gen-better-python/docs/guide/converters/)). +- **Typed JSON columns** via msgspec structs + ([docs](https://rayakame.github.io/sqlc-gen-better-python/docs/guide/working-with-json/)). +- **Optional docstrings** in `google`, `numpy`, or `pep257` convention + ([docs](https://rayakame.github.io/sqlc-gen-better-python/docs/guide/docstrings/)). +- Generated code passes **pyright strict** and **ruff**. + +Every [sqlc macro](https://docs.sqlc.dev/en/latest/reference/macros.html) is +supported. Which query commands are available depends on the driver - see the +[feature support matrix](https://rayakame.github.io/sqlc-gen-better-python/docs/reference/feature-support/). ## Development -A roadmap of what is planned & worked on can be found [here](https://github.com/users/rayakame/projects/1/). - Contributions are very welcome, for more information and help please read the [contribution guidelines](https://github.com/rayakame/sqlc-gen-better-python/blob/main/CONTRIBUTING.md). diff --git a/docs/assets/css/custom.css b/docs/assets/css/custom.css new file mode 100644 index 00000000..90ed1171 --- /dev/null +++ b/docs/assets/css/custom.css @@ -0,0 +1,11 @@ +/* Hextra bottom-aligns card content (margin-top: auto) so that text sits under + a card image. Text-only cards in a stretched grid row then get a gap above + the title when a taller sibling sets the row height, so top-align those and + keep the original behaviour for cards that do have an image. */ +.hextra-card > div { + margin-top: 0; +} + +.hextra-card-image ~ div { + margin-top: auto; +} diff --git a/docs/content/_index.md b/docs/content/_index.md index 93e20017..c4c5af7e 100644 --- a/docs/content/_index.md +++ b/docs/content/_index.md @@ -30,26 +30,167 @@ layout: hextra-home {{< hextra/feature-grid >}} {{< hextra/feature-card title="Four model types" + link="docs/guide/model-types" subtitle="Generate dataclass, attrs, msgspec, or pydantic models - pick per codegen block." >}} {{< hextra/feature-card title="Three drivers" + link="docs/guide/drivers" subtitle="asyncpg for PostgreSQL, plus aiosqlite and sqlite3 for SQLite." >}} {{< hextra/feature-card title="Strictly typed output" + link="docs/guide/writing-queries" subtitle="Generated code passes pyright strict and ruff, targeting Python 3.12+." >}} {{< hextra/feature-card title="Enums" + link="docs/guide/enums" subtitle="PostgreSQL enums become enum.StrEnum classes, wired through models and queries." >}} {{< hextra/feature-card title="Type overrides & converters" + link="docs/guide/type-overrides" subtitle="Swap a column's Python type, or plug in your own encode/decode functions." >}} {{< hextra/feature-card title="Docstrings" + link="docs/guide/docstrings" subtitle="Optional google, numpy, or pep257 docstrings on every generated function." >}} {{< /hextra/feature-grid >}} + +
+ +{{< hextra/hero-section >}} + From SQL to Python +{{< /hextra/hero-section >}} + +You write a query, annotated with the name and shape you want: + +```sql +-- name: GetUser :one +SELECT * FROM users WHERE id = $1; +``` + +and get a typed function back, with a model built from your schema: + +```python +async def get_user(conn: ConnectionLike, *, id_: int) -> models.User | None: + row = await conn.fetchrow(GET_USER, id_) + if row is None: + return None + return models.User(id_=row[0], name=row[1]) +``` + +No ORM, no hand-written row unpacking, and pyright checks every field access. + +
+ +{{< hextra/hero-section >}} + One schema, four model types +{{< /hextra/hero-section >}} + +Set `model_type` and the same table generates whichever flavour your project +already uses - the fields and annotations are identical: + +{{< tabs >}} + + {{< tab name="dataclass" >}} + +```python +@dataclasses.dataclass() +class User: + id_: int + name: str +``` + + {{< /tab >}} + + {{< tab name="attrs" >}} + +```python +@attrs.define() +class User: + id_: int + name: str +``` + + {{< /tab >}} + + {{< tab name="msgspec" >}} + +```python +class User(msgspec.Struct): + id_: int + name: str +``` + + {{< /tab >}} + + {{< tab name="pydantic" >}} + +```python +class User(pydantic.BaseModel): + model_config = pydantic.ConfigDict(arbitrary_types_allowed=True) + + id_: int + name: str +``` + + {{< /tab >}} + +{{< /tabs >}} + +
+ +{{< hextra/hero-section >}} + Built to be trusted +{{< /hextra/hero-section >}} + +Generated code is held to the same standard as hand-written code: + +- **Type-checked.** Every generated file passes pyright in strict mode and ruff. +- **Tested against real databases.** The suite runs the generated code against + live PostgreSQL and SQLite, across every driver and model type. +- **Deterministic.** Output is byte-identical between runs, and CI fails if a + change would silently alter what you get. + +
+ +{{< hextra/hero-section >}} + Set up in three steps +{{< /hextra/hero-section >}} + +Point `sqlc` at the plugin, pick a driver and a model type, and generate: + +```yaml +# sqlc.yaml +version: "2" +plugins: + - name: python + wasm: + url: https://github.com/rayakame/sqlc-gen-better-python/releases/download/v0.5.1/sqlc-gen-better-python.wasm + sha256: c7cc470df7625ae3232c2b042060b948180ae784ce3d81c32e8a2c040fe04fa7 +sql: + - engine: "postgresql" + queries: "query.sql" + schema: "schema.sql" + codegen: + - out: "app/db" + plugin: python + options: + package: "db" + emit_init_file: true + sql_driver: "asyncpg" +``` + +```bash +sqlc generate +``` + +
+ +{{< hextra/hero-button text="Read the Getting Started guide" link="docs/getting-started" >}} + +
diff --git a/docs/content/docs/_index.md b/docs/content/docs/_index.md index 5b28102d..40737bda 100644 --- a/docs/content/docs/_index.md +++ b/docs/content/docs/_index.md @@ -8,18 +8,46 @@ sidebar: --- `sqlc-gen-better-python` is a [sqlc](https://sqlc.dev) plugin that turns your SQL -schema and queries into modern, fully typed Python database code: models plus -async query functions, ready to run. +schema and queries into modern, fully typed Python database code: models, typed +query functions, and enums. You keep writing SQL; the Python stays in sync with +it. -Start with **[Getting Started](getting-started)** for installation and a first -generate, then explore the individual features. +{{< cards >}} + {{< card link="getting-started" title="Getting Started" icon="play" subtitle="Install the plugin and go from a schema to your first typed queries." >}} + {{< card link="guide" title="Guide" icon="book-open" subtitle="Every feature, explained with real schema, queries, and generated output." >}} + {{< card link="reference" title="Reference" icon="clipboard-list" subtitle="Configuration options, type mappings, and per-driver support." >}} +{{< /cards >}} + +## Explore by feature {{< cards >}} - {{< card link="getting-started" title="Getting Started" subtitle="Install the plugin and generate your first models." >}} - {{< card link="converters" title="Converters" subtitle="Serialize and deserialize columns with your own functions." >}} + {{< card link="guide/drivers" title="Drivers" icon="database" subtitle="asyncpg, aiosqlite, and sqlite3." >}} + {{< card link="guide/model-types" title="Model types" icon="cube" subtitle="dataclass, attrs, msgspec, or pydantic." >}} + {{< card link="guide/writing-queries" title="Writing queries" icon="code" subtitle="How query annotations become typed functions." >}} + {{< card link="guide/enums" title="Enums" icon="collection" subtitle="PostgreSQL enums as enum.StrEnum classes." >}} + {{< card link="guide/type-overrides" title="Type overrides" icon="adjustments" subtitle="Swap the Python type of a column." >}} + {{< card link="guide/converters" title="Converters" icon="puzzle" subtitle="Your own encode/decode functions for a column." >}} + {{< card link="guide/working-with-json" title="Working with JSON" icon="document-text" subtitle="Typed JSON columns with msgspec structs." >}} + {{< card link="guide/docstrings" title="Docstrings" icon="document" subtitle="google, numpy, or pep257 docstrings." >}} {{< /cards >}} -{{< callout type="info" >}} - These docs are new and growing. More feature pages (model types, drivers, - enums, type overrides, docstrings) are being ported from the README. -{{< /callout >}} +## At a glance + +| | | +|---|---| +| **Python** | 3.12 or newer | +| **Engines** | PostgreSQL, SQLite | +| **Drivers** | `asyncpg`, `aiosqlite`, `sqlite3` | +| **Model types** | `dataclass`, `attrs`, `msgspec`, `pydantic` | +| **Docstrings** | `google`, `numpy`, `pep257`, or none | +| **Checked with** | pyright (strict) and ruff | + +Full details in the [feature support reference](/docs/reference/feature-support). + +## Getting help + +{{< cards >}} + {{< card link="https://github.com/rayakame/sqlc-gen-better-python/issues" title="Issues" icon="github" subtitle="Report a bug or request a feature." >}} + {{< card link="https://discord.gg/hikari" title="Discord" icon="chat" subtitle="Ask a question or share what you built." >}} + {{< card link="https://github.com/rayakame/sqlc-gen-better-python/blob/main/CHANGELOG.md" title="Changelog" icon="clipboard-check" subtitle="What changed in each release." >}} +{{< /cards >}} diff --git a/docs/content/docs/converters.md b/docs/content/docs/converters.md deleted file mode 100644 index a4b53516..00000000 --- a/docs/content/docs/converters.md +++ /dev/null @@ -1,125 +0,0 @@ ---- -title: Converters -weight: 3 -prev: getting-started ---- - -A [type override](https://github.com/rayakame/sqlc-gen-better-python#type-overrides) -replaces a column's Python type, but it converts by *calling that type* -(`Preferences(value)`), which cannot express JSON or model (de)serialization. A -**converter** names two of your own functions instead, used whenever the column -is read from or written to the database. - -## Defining a converter - -Declare the converter under `converters`, then reference it from an override: - -```yaml -options: - # ... - converters: - - name: prefs - py_type: - import: myapp.models - package: Preferences - type: Preferences - to_db: myapp.converters.encode_preferences - from_db: myapp.converters.decode_preferences - overrides: - - db_type: jsonb # every jsonb column - converter: prefs - - column: users.preferences # or one specific column - converter: prefs -``` - -The generated code calls your functions on both sides: - -```python -# read -preferences = myapp.converters.decode_preferences(row[1]) -# write -await conn.execute(CREATE_USER, id_, myapp.converters.encode_preferences(preferences)) -``` - -## Rules - -{{< callout type="info" >}} - A converter is referenced *by an override*, so it applies to whichever columns - that override matches. `converter` replaces `py_type` on the override. -{{< /callout >}} - -- **Both directions are required.** `to_db` and `from_db` are dotted paths to - module-level functions; the module is imported and the function called fully - qualified, so the names can never collide with generated code. -- **Wire type.** `to_db` must return the type the column would have had *without* - the override (`jsonb` is a `str`, `bytea` a `memoryview`), and `from_db` - receives that same type. For an unrecognised SQL type there is no such type, so - `to_db` must return one the driver accepts. -- **Your functions never see `None`.** Nullable columns are guarded, so a SQL - `NULL` stays `None` without calling the converter. -- **List columns convert element-wise.** - -## Example: msgspec structs to and from JSON - -Because `msgspec.json.decode` needs a `type=` argument and returns `bytes`, you -wrap it in two small functions that match the wire type of a `jsonb` column -(`str`): - -{{< tabs >}} - - {{< tab name="converters.py" >}} - -```python -import msgspec - -from myapp.models import User - - -def encode_user(value: User) -> str: - # jsonb's wire type is str, so decode msgspec's bytes back to str. - return msgspec.json.encode(value).decode() - - -def decode_user(value: str) -> User: - return msgspec.json.decode(value, type=User) -``` - - {{< /tab >}} - - {{< tab name="models.py" >}} - -```python -import msgspec - - -class User(msgspec.Struct): - name: str - groups: set[str] - email: str | None = None -``` - - {{< /tab >}} - - {{< tab name="sqlc.yaml" >}} - -```yaml -converters: - - name: user - py_type: - import: myapp.models - package: User - type: User - to_db: myapp.converters.encode_user - from_db: myapp.converters.decode_user -overrides: - - db_type: jsonb - converter: user -``` - - {{< /tab >}} - -{{< /tabs >}} - -A `get_user` query then returns a fully typed `User` (set field and all), and a -`create_user` query accepts one directly - the plugin inserts the -`decode_user` / `encode_user` calls for you. diff --git a/docs/content/docs/getting-started.md b/docs/content/docs/getting-started.md index c33c46ff..19efddbe 100644 --- a/docs/content/docs/getting-started.md +++ b/docs/content/docs/getting-started.md @@ -1,20 +1,59 @@ --- title: Getting Started -weight: 2 +weight: 1 prev: /docs -next: converters +next: /docs/guide +tabs: + sync: true --- +From an empty project to your first typed queries. Pick your driver in the tabs +below and the whole page follows your choice. + ## Prerequisites You need [`sqlc`](https://docs.sqlc.dev/en/latest/overview/install.html) on your -`PATH`. The generated code targets **Python 3.12 or newer** (it uses PEP 695 type -aliases and generics, and `enum.StrEnum`). +`PATH` and **Python 3.12 or newer** (the generated code uses PEP 695 type aliases +and generics, and `enum.StrEnum`). + +Then install the database driver you want to use: + +{{< tabs >}} + + {{< tab name="asyncpg" >}} + +```bash +pip install asyncpg +``` + + {{< /tab >}} + + {{< tab name="aiosqlite" >}} + +```bash +pip install aiosqlite +``` + + {{< /tab >}} + + {{< tab name="sqlite3" >}} + +```text +Nothing to install - sqlite3 is in the standard library. +``` + + {{< /tab >}} -## Configure the plugin +{{< /tabs >}} -The plugin is a WASM binary that `sqlc generate` downloads and runs. Point your -`sqlc.yaml` at a release and select a driver and model type: +## 1. Configure the plugin + +The plugin is a WASM binary that `sqlc generate` downloads and runs. Create a +`sqlc.yaml`: + +{{< tabs >}} + + {{< tab name="asyncpg" >}} ```yaml # filename: sqlc.yaml @@ -35,15 +74,170 @@ sql: package: "db" emit_init_file: true sql_driver: "asyncpg" - model_type: "msgspec" + model_type: "dataclass" ``` + {{< /tab >}} + + {{< tab name="aiosqlite" >}} + +```yaml +# filename: sqlc.yaml +version: "2" +plugins: + - name: python + wasm: + url: https://github.com/rayakame/sqlc-gen-better-python/releases/download/v0.5.1/sqlc-gen-better-python.wasm + sha256: c7cc470df7625ae3232c2b042060b948180ae784ce3d81c32e8a2c040fe04fa7 +sql: + - engine: "sqlite" + queries: "query.sql" + schema: "schema.sql" + codegen: + - out: "app/db" + plugin: python + options: + package: "db" + emit_init_file: true + sql_driver: "aiosqlite" + model_type: "dataclass" +``` + + {{< /tab >}} + + {{< tab name="sqlite3" >}} + +```yaml +# filename: sqlc.yaml +version: "2" +plugins: + - name: python + wasm: + url: https://github.com/rayakame/sqlc-gen-better-python/releases/download/v0.5.1/sqlc-gen-better-python.wasm + sha256: c7cc470df7625ae3232c2b042060b948180ae784ce3d81c32e8a2c040fe04fa7 +sql: + - engine: "sqlite" + queries: "query.sql" + schema: "schema.sql" + codegen: + - out: "app/db" + plugin: python + options: + package: "db" + emit_init_file: true + sql_driver: "sqlite3" + model_type: "dataclass" +``` + + {{< /tab >}} + +{{< /tabs >}} + {{< callout type="warning" >}} - Always pin the `sha256` of the release you use - `sqlc` refuses to run a - plugin whose hash does not match. Each release lists its hash. + Always pin the `sha256` of the release you use - `sqlc` refuses to run a plugin + whose hash does not match. Each release lists its hash. {{< /callout >}} -## Generate +`model_type: "dataclass"` is the default and needs no extra dependency. See +[Model types](/docs/guide/model-types) for `attrs`, `msgspec`, and `pydantic`. + +## 2. Describe your schema + +{{< tabs >}} + + {{< tab name="asyncpg" >}} + +```sql +-- filename: schema.sql +CREATE TABLE users +( + id bigint PRIMARY KEY NOT NULL, + name text NOT NULL +); +``` + + {{< /tab >}} + + {{< tab name="aiosqlite" >}} + +```sql +-- filename: schema.sql +CREATE TABLE users +( + id INTEGER PRIMARY KEY NOT NULL, + name TEXT NOT NULL +); +``` + + {{< /tab >}} + + {{< tab name="sqlite3" >}} + +```sql +-- filename: schema.sql +CREATE TABLE users +( + id INTEGER PRIMARY KEY NOT NULL, + name TEXT NOT NULL +); +``` + + {{< /tab >}} + +{{< /tabs >}} + +## 3. Write your queries + +Each query is a `-- name: :` comment followed by SQL. The name +becomes the Python function, and the command decides what it returns - `:one` +returns a single row or `None`, `:many` returns all matching rows. + +{{< tabs >}} + + {{< tab name="asyncpg" >}} + +```sql +-- filename: query.sql +-- name: GetUser :one +SELECT * FROM users WHERE id = $1; + +-- name: ListUsers :many +SELECT * FROM users ORDER BY name; +``` + + {{< /tab >}} + + {{< tab name="aiosqlite" >}} + +```sql +-- filename: query.sql +-- name: GetUser :one +SELECT * FROM users WHERE id = ?; + +-- name: ListUsers :many +SELECT * FROM users ORDER BY name; +``` + + {{< /tab >}} + + {{< tab name="sqlite3" >}} + +```sql +-- filename: query.sql +-- name: GetUser :one +SELECT * FROM users WHERE id = ?; + +-- name: ListUsers :many +SELECT * FROM users ORDER BY name; +``` + + {{< /tab >}} + +{{< /tabs >}} + +PostgreSQL uses `$1` placeholders, SQLite uses `?`. Everything else is the same. + +## 4. Generate Run `sqlc` from the directory containing your `sqlc.yaml`: @@ -51,12 +245,197 @@ Run `sqlc` from the directory containing your `sqlc.yaml`: sqlc generate ``` -This writes a Python package to `out` containing `models.py`, one query module -per query file, and (when your schema has enums) an `enums.py`. +This writes a Python package to `out` (`app/db` above) containing `models.py`, +one query module per query file (`query.py`), and an `__init__.py`. + +## 5. What you got + +A model per table, and a typed function per query: + +{{< tabs >}} + + {{< tab name="asyncpg" >}} + +```python +# models.py +@dataclasses.dataclass() +class User: + id_: int + name: str + + +# query.py +async def get_user(conn: ConnectionLike, *, id_: int) -> models.User | None: + row = await conn.fetchrow(GET_USER, id_) + if row is None: + return None + return models.User(id_=row[0], name=row[1]) + + +def list_users(conn: ConnectionLike) -> QueryResults[models.User]: + ... +``` + + {{< /tab >}} + + {{< tab name="aiosqlite" >}} + +```python +# models.py +@dataclasses.dataclass() +class User: + id_: int + name: str + + +# query.py +async def get_user(conn: aiosqlite.Connection, *, id_: int) -> models.User | None: + row = await (await conn.execute(GET_USER, (id_,))).fetchone() + if row is None: + return None + return models.User(id_=row[0], name=row[1]) + + +def list_users(conn: aiosqlite.Connection) -> QueryResults[models.User]: + ... +``` + + {{< /tab >}} + + {{< tab name="sqlite3" >}} + +```python +# models.py +@dataclasses.dataclass() +class User: + id_: int + name: str + + +# query.py +def get_user(conn: sqlite3.Connection, *, id_: int) -> models.User | None: + row = conn.execute(GET_USER, (id_,)).fetchone() + if row is None: + return None + return models.User(id_=row[0], name=row[1]) + + +def list_users(conn: sqlite3.Connection) -> QueryResults[models.User]: + ... +``` + + {{< /tab >}} + +{{< /tabs >}} + +Two things to notice: + +- The `id` column became `id_`, because `id` is a Python builtin. See + [Naming and identifiers](/docs/guide/naming). +- Parameters are keyword-only (`*,`), so you call `get_user(conn, id_=1)`. + +## 6. Use it + +`:one` gives you a model or `None`. `:many` returns a `QueryResults`, which you +can consume all at once or stream row by row: + +{{< tabs >}} + + {{< tab name="asyncpg" >}} + +```python +import asyncio + +import asyncpg + +from app.db import query + + +async def main() -> None: + conn = await asyncpg.connect("postgresql://user:pass@localhost/mydb") + + user = await query.get_user(conn, id_=1) + if user is not None: + print(user.name) + + # every row at once + users = await query.list_users(conn) + + # or stream them + async for user in query.list_users(conn): + print(user.name) + + +asyncio.run(main()) +``` + + {{< /tab >}} + + {{< tab name="aiosqlite" >}} + +```python +import asyncio +import sqlite3 + +import aiosqlite + +from app.db import query + + +async def main() -> None: + async with aiosqlite.connect("app.db", detect_types=sqlite3.PARSE_DECLTYPES) as conn: + user = await query.get_user(conn, id_=1) + if user is not None: + print(user.name) + + # every row at once + users = await query.list_users(conn) + + # or stream them + async for user in query.list_users(conn): + print(user.name) + + +asyncio.run(main()) +``` + + {{< /tab >}} + + {{< tab name="sqlite3" >}} + +```python +import sqlite3 + +from app.db import query + +conn = sqlite3.connect("app.db", detect_types=sqlite3.PARSE_DECLTYPES) + +user = query.get_user(conn, id_=1) +if user is not None: + print(user.name) + +# every row at once - call the result +users = query.list_users(conn)() + +# or iterate +for user in query.list_users(conn): + print(user.name) +``` + + {{< /tab >}} + +{{< /tabs >}} + +{{< callout type="info" >}} + The `detect_types=sqlite3.PARSE_DECLTYPES` above is not needed for this schema, + but it is what makes generated converters work once your tables use dates, + decimals, booleans, or blobs. See + [SQLite type conversion](/docs/guide/sqlite-type-conversion). +{{< /callout >}} ## Next steps -- Full option reference and driver support: see the - [project README](https://github.com/rayakame/sqlc-gen-better-python#configuration-options) - while the remaining pages are ported here. -- Plug in your own serialization with [Converters](converters). +- Work through the [Guide](/docs/guide) - configuration, drivers, model types, + writing queries, and every feature, each with real generated output. +- Look up specifics in the [Reference](/docs/reference): the full option list, + SQL-to-Python type mappings, and per-driver support. diff --git a/docs/content/docs/guide/_index.md b/docs/content/docs/guide/_index.md new file mode 100644 index 00000000..098d88f3 --- /dev/null +++ b/docs/content/docs/guide/_index.md @@ -0,0 +1,48 @@ +--- +title: Guide +weight: 2 +prev: /docs/getting-started +next: /docs/guide/configuration +sidebar: + open: true +--- + +This guide walks through `sqlc-gen-better-python` one feature at a time. Each +page states what a feature does, the configuration it needs, and shows real +schema, queries, and the exact Python the plugin generates for them. + +## How the pieces fit + +You never call this plugin directly. You write a SQL schema and queries, and +`sqlc` runs the plugin for you: + +1. You write **schema** (`CREATE TABLE`, `CREATE TYPE`) and **queries** + (annotated `SELECT`/`INSERT`/...). +2. `sqlc` parses them into a typed **catalog** and hands it to the plugin. +3. The plugin walks the catalog and emits **Python**: a `models.py` (one class + per table), one query module per query `.sql` file (typed functions), and an + `enums.py` when your schema has enums. + +So everything in this guide is driven by two inputs - your SQL and your +`sqlc.yaml` options - and observed through one output: the generated Python. + +{{< callout type="info" >}} + New here? Start with [Getting Started](/docs/getting-started) for install and + a first `sqlc generate`, then come back to this guide. +{{< /callout >}} + +## How to read this guide + +Start with the essentials, then dip into feature pages as you need them: + +{{< cards >}} + {{< card link="configuration" title="Configuration" subtitle="The sqlc.yaml plugin block and the core options." >}} + {{< card link="drivers" title="Drivers" subtitle="asyncpg, aiosqlite, and sqlite3 - and which query commands each supports." >}} + {{< card link="model-types" title="Model types" subtitle="dataclass, attrs, msgspec, or pydantic models." >}} + {{< card link="writing-queries" title="Writing queries" subtitle="How query annotations become typed Python functions." >}} +{{< /cards >}} + +The remaining pages - enums, type overrides, converters, working with JSON, +docstrings, SQLite type conversion, and naming - cover specific features and can +be read in any order. The [Reference](/docs/reference) section holds the +canonical option, type-mapping, and driver-support tables. diff --git a/docs/content/docs/guide/configuration.md b/docs/content/docs/guide/configuration.md new file mode 100644 index 00000000..14215305 --- /dev/null +++ b/docs/content/docs/guide/configuration.md @@ -0,0 +1,95 @@ +--- +title: Configuration +weight: 10 +prev: /docs/guide +next: /docs/guide/drivers +--- + +Everything the plugin does is driven by your `sqlc.yaml`. This page covers how +the plugin is wired in and how its options are structured. For the exhaustive +option list, see the [configuration options reference](/docs/reference/configuration-options). + +## Anatomy of `sqlc.yaml` + +Two parts matter: a `plugins` entry that declares the WASM plugin, and a +`codegen` entry inside your `sql` block that runs it with a set of `options`. + +```yaml +# filename: sqlc.yaml +version: "2" +plugins: + - name: python + wasm: + url: https://github.com/rayakame/sqlc-gen-better-python/releases/download/v0.5.1/sqlc-gen-better-python.wasm + sha256: c7cc470df7625ae3232c2b042060b948180ae784ce3d81c32e8a2c040fe04fa7 +sql: + - engine: "postgresql" + queries: "query.sql" + schema: "schema.sql" + codegen: + - out: "app/db" + plugin: python + options: + package: "db" + emit_init_file: true + sql_driver: "asyncpg" + model_type: "msgspec" +``` + +- **`plugins`** declares the plugin once, pinned to a release `url` and its + `sha256`. `sqlc` downloads and runs the WASM binary. +- **`sql`** points `sqlc` at your `engine`, `queries`, and `schema`. +- **`codegen`** runs the plugin: `out` is the output directory, `plugin` refers + to the name declared above, and `options` configures this plugin. + +{{< callout type="warning" >}} + The `sha256` must match the release you pin - `sqlc` refuses to run a plugin + whose hash does not match. Each release lists its hash. +{{< /callout >}} + +## The options block + +`options` is where all plugin configuration lives. Three options are required: + +| Option | What it does | +|---|---| +| `package` | The name of the generated package. | +| `sql_driver` | `asyncpg`, `aiosqlite`, or `sqlite3` - must match the `engine`. See [Drivers](/docs/guide/drivers). | +| `emit_init_file` | Whether to emit `__init__.py`. Must be set explicitly. | + +Everything else is optional and has a sensible default. The most common ones to +reach for next are [`model_type`](/docs/guide/model-types) and +[`docstrings`](/docs/guide/docstrings). The full list, with types and defaults, +is in the [reference](/docs/reference/configuration-options). + +## Multiple outputs + +A single `sql` block can have several `codegen` entries, each writing to its own +`out`. This is handy to generate more than one flavor from the same schema and +queries - for example a `msgspec` package and a `dataclass` package: + +```yaml + codegen: + - out: "app/db_msgspec" + plugin: python + options: + package: "db_msgspec" + emit_init_file: true + sql_driver: "asyncpg" + model_type: "msgspec" + - out: "app/db_dataclass" + plugin: python + options: + package: "db_dataclass" + emit_init_file: true + sql_driver: "asyncpg" + model_type: "dataclass" +``` + +## Common pitfalls + +- **Driver/engine mismatch.** `sql_driver: asyncpg` requires `engine: "postgresql"`; + `aiosqlite`/`sqlite3` require `engine: "sqlite"`. A mismatch is an error. +- **Forgetting `emit_init_file`.** It has no default and generation fails if it + is omitted. Set it to `true` unless the package already has an `__init__.py`. +- **A stale `sha256`.** When you bump the plugin version, update the hash too. diff --git a/docs/content/docs/guide/converters.md b/docs/content/docs/guide/converters.md new file mode 100644 index 00000000..7c39803c --- /dev/null +++ b/docs/content/docs/guide/converters.md @@ -0,0 +1,97 @@ +--- +title: Converters +weight: 70 +prev: /docs/guide/type-overrides +next: /docs/guide/working-with-json +aliases: + - /docs/converters/ +--- + +A [type override](/docs/guide/type-overrides) replaces a column's Python type, +but it converts by *calling that type* (`Preferences(value)`), which cannot +express JSON or model (de)serialization. A **converter** names two of your own +functions instead, used whenever the column is read from or written to the +database. + +## Specification + +A converter is declared under `converters` and referenced *by an override*: + +| Field | Description | +|---|---| +| `name` | Unique name, referenced by an override's `converter`. | +| `py_type` | The Python type the converter produces and accepts (`import`/`package`/`type`). | +| `to_db` | Dotted path to a function turning the Python value into the column's wire type. | +| `from_db` | Dotted path to a function turning the wire value into the Python type. | + +```yaml +options: + converters: + - name: prefs + py_type: + import: myapp.models + package: Preferences + type: Preferences + to_db: myapp.converters.encode_preferences + from_db: myapp.converters.decode_preferences + overrides: + - db_type: jsonb # every jsonb column + converter: prefs + - column: users.preferences # or one specific column + converter: prefs +``` + +Because the converter is referenced by an override, it applies to whichever +columns that override matches, and `converter` replaces `py_type` on the +override. + +## Generated code + +The column is typed with your `py_type`, and the plugin inserts calls to your +functions on both sides: + +```python +# read - from_db on each column +return models.User(id_=row[0], preferences=myapp.converters.decode_preferences(row[1])) + +# write - to_db on each parameter +await conn.execute(CREATE_USER, id_, myapp.converters.encode_preferences(preferences)) +``` + +Nullable columns are guarded, so your functions never receive `None`: + +```python +maybe_prefs=myapp.converters.decode_preferences(row[2]) if row[2] is not None else None +``` + +and list columns convert element-wise: + +```python +[myapp.converters.encode_label(v) for v in labels] +``` + +## Rules + +- **Both directions are required.** `to_db` and `from_db` are dotted paths to + module-level functions; the module is imported and the function called fully + qualified, so the names can never collide with generated code. +- **Wire type.** `to_db` must return the type the column would have had *without* + the override (`jsonb` is a `str`, `bytea` a `memoryview` - see + [type mappings](/docs/reference/type-mappings)), and `from_db` receives that + same type. For an unrecognised SQL type there is no such type, so `to_db` must + return a type the driver accepts. +- **Your functions never see `None`.** A SQL `NULL` stays `None` without calling + the converter. +- **List columns convert element-wise.** + +{{< callout type="warning" >}} + To reach an `ANY($1::type[])` array parameter, match the override with + `db_type` rather than `column` - sqlc does not link column overrides to those + parameters. +{{< /callout >}} + +## Next + +Converters are most often reached for with JSON columns. See +[Working with JSON](/docs/guide/working-with-json) for a complete walkthrough +using `msgspec` structs. diff --git a/docs/content/docs/guide/docstrings.md b/docs/content/docs/guide/docstrings.md new file mode 100644 index 00000000..21fff0c1 --- /dev/null +++ b/docs/content/docs/guide/docstrings.md @@ -0,0 +1,143 @@ +--- +title: Docstrings +weight: 80 +prev: /docs/guide/working-with-json +next: /docs/guide/sqlite-type-conversion +--- + +The plugin can document everything it generates - models, enums, and query +functions - in the docstring convention of your choice. + +```yaml +options: + docstrings: "google" # none | google | numpy | pep257 + docstrings_emit_sql: true # embed each query's SQL (default true) +``` + +The default is `none`, which emits no docstrings at all. + +## The three conventions + +The same `:one` query, generated under each convention: + +{{< tabs >}} + + {{< tab name="google" >}} + +````python +async def get_field_naming(conn: ConnectionLike, *, id_: int) -> models.TestFieldNaming | None: + """Fetch one from the db using the SQL query with `name: GetFieldNaming :one`. + + ```sql + SELECT id, outputs + FROM test_field_namings + WHERE id = $1 LIMIT 1 + ``` + + Args: + conn: + Connection object of type `ConnectionLike` used to execute the query. + id_: int. + + Returns: + Result of type `models.TestFieldNaming` fetched from the db. Will be `None` if not found. + """ +```` + + {{< /tab >}} + + {{< tab name="numpy" >}} + +````python +async def get_field_naming(conn: ConnectionLike, *, id_: int) -> models.TestFieldNaming | None: + """Fetch one from the db using the SQL query with `name: GetFieldNaming :one`. + + ```sql + SELECT id, outputs + FROM test_field_namings + WHERE id = $1 LIMIT 1 + ``` + + Parameters + ---------- + conn : ConnectionLike + Connection object of type `ConnectionLike` used to execute the query. + id_ : int + + Returns + ------- + models.TestFieldNaming + Result fetched from the db. Will be `None` if not found. + + """ +```` + + {{< /tab >}} + + {{< tab name="pep257" >}} + +````python +async def get_field_naming(conn: ConnectionLike, *, id_: int) -> models.TestFieldNaming | None: + """Fetch one from the db using the SQL query with `name: GetFieldNaming :one`. + + ```sql + SELECT id, outputs + FROM test_field_namings + WHERE id = $1 LIMIT 1 + ``` + + Arguments: + conn -- Connection object of type `ConnectionLike` used to execute the query. + id_ -- int. + + Returns: + models.TestFieldNaming -- Result fetched from the db. Will be `None` if not found. + """ +```` + + {{< /tab >}} + +{{< /tabs >}} + +## Embedded SQL + +By default each query function's docstring includes the exact SQL it runs, in a +fenced `sql` block - handy when reading generated code or hovering a call in your +editor. Set `docstrings_emit_sql: false` to leave it out. + +{{< callout type="info" >}} + `:copyfrom` functions never embed SQL, since they do not execute a statement - + they stream rows via the driver's bulk copy API. +{{< /callout >}} + +## Models and enums + +Models and enums are documented too. Under `numpy`, the generated model carries an +`Attributes` section: + +```python +@attrs.define() +class TestFieldNaming: + """Model representing TestFieldNaming. + + Attributes + ---------- + id_ : int + outputs : str + + """ + + id_: int + outputs: str +``` + +## Linting generated code + +If you run a docstring linter (ruff's `pydocstyle` rules, for example) over the +generated package, set its convention to match the one you generated, or every +file will report style violations. In `ruff.toml`: + +```toml +[lint.pydocstyle] +convention = "google" +``` diff --git a/docs/content/docs/guide/drivers.md b/docs/content/docs/guide/drivers.md new file mode 100644 index 00000000..1e931258 --- /dev/null +++ b/docs/content/docs/guide/drivers.md @@ -0,0 +1,84 @@ +--- +title: Drivers +weight: 20 +prev: /docs/guide/configuration +next: /docs/guide/model-types +--- + +The `sql_driver` option picks which database library the generated code targets. +It must match your `engine`. Three drivers are supported: + +| Driver | Engine | Style | +|---|---|---| +| `asyncpg` | `postgresql` | async | +| `aiosqlite` | `sqlite` | async | +| `sqlite3` | `sqlite` | sync | + +Every generated query function takes the connection as its first argument, so you +open and manage the connection yourself and pass it in. + +## asyncpg (PostgreSQL) + +```python +import asyncio + +import asyncpg + +from app.db import queries + + +async def main() -> None: + conn = await asyncpg.connect("postgresql://user:pass@localhost/db") + user = await queries.get_field_naming(conn, id_=1) + + +asyncio.run(main()) +``` + +asyncpg is the only driver that supports `:copyfrom` (bulk insert via +`copy_records_to_table`). + +## aiosqlite (async SQLite) + +```python +import asyncio +import sqlite3 + +import aiosqlite + +from app.db import queries + + +async def main() -> None: + async with aiosqlite.connect("app.db", detect_types=sqlite3.PARSE_DECLTYPES) as conn: + user = await queries.get_field_naming(conn, id_=1) + + +asyncio.run(main()) +``` + +## sqlite3 (sync SQLite) + +```python +import sqlite3 + +from app.db import queries + +conn = sqlite3.connect("app.db", detect_types=sqlite3.PARSE_DECLTYPES) +user = queries.get_field_naming(conn, id_=1) +``` + +{{< callout type="warning" >}} + When a query *returns* a `date`, `datetime`/`timestamp`, `decimal`, `bool`, or + `blob` column, the generated code registers a converter for it - and converters + only run if the connection was opened with + `detect_types=sqlite3.PARSE_DECLTYPES`. Adapters, which send those types as + parameters, work without it. See + [SQLite type conversion](/docs/guide/sqlite-type-conversion). +{{< /callout >}} + +## Command support + +Not every [query command](/docs/guide/writing-queries) works on every driver - +for example `:copyfrom` is asyncpg-only and `:execlastid` is SQLite-only. The +full matrix is in the [feature support reference](/docs/reference/feature-support). diff --git a/docs/content/docs/guide/enums.md b/docs/content/docs/guide/enums.md new file mode 100644 index 00000000..26ed966d --- /dev/null +++ b/docs/content/docs/guide/enums.md @@ -0,0 +1,68 @@ +--- +title: Enums +weight: 50 +prev: /docs/guide/writing-queries +next: /docs/guide/type-overrides +--- + +PostgreSQL enum types become `enum.StrEnum` classes in a generated `enums.py` +module. Columns of that type are annotated with the class, and values read from +the database are coerced into it. + +## Example + +```sql +CREATE TYPE test_mood AS ENUM ('sad', 'ok', 'happy', '24h', '_hidden'); +``` + +generates: + +```python +class TestMood(enum.StrEnum): + SAD = "sad" + OK = "ok" + HAPPY = "happy" + VALUE_24H = "24h" + VALUE__HIDDEN = "_hidden" +``` + +Notice the member names are uppercased, and values that are not valid Python +identifiers are sanitized: the digit-leading `24h` becomes `VALUE_24H` and the +underscore-leading `_hidden` becomes `VALUE__HIDDEN`. The string *values* are +untouched, so round-tripping to the database is exact. See +[Naming and identifiers](/docs/guide/naming) for the full sanitization rules. + +## Using enums + +A column of an enum type is annotated with the generated class, and nullable +columns get `| None`: + +```sql +CREATE TABLE test_enum_types +( + id int PRIMARY KEY NOT NULL, + mood test_mood NOT NULL, + maybe_mood test_mood +); +``` + +```python +class TestEnumType(msgspec.Struct): + id_: int + mood: enums.TestMood + maybe_mood: enums.TestMood | None +``` + +Query functions coerce database values into these classes automatically, so you +get real `TestMood` members back, not bare strings. + +## Enums in other schemas + +Enums in a non-default schema get schema-qualified class names so same-named +enums never collide - for example `custom.mood` becomes `CustomMood`, distinct +from a `public.mood` that would become `Mood`. + +{{< callout type="info" >}} + Enum classes are a PostgreSQL feature - SQLite has no native enum type, so this + applies to the `asyncpg` driver. +{{< /callout >}} diff --git a/docs/content/docs/guide/model-types.md b/docs/content/docs/guide/model-types.md new file mode 100644 index 00000000..0d5b14a1 --- /dev/null +++ b/docs/content/docs/guide/model-types.md @@ -0,0 +1,103 @@ +--- +title: Model types +weight: 30 +prev: /docs/guide/drivers +next: /docs/guide/writing-queries +--- + +The `model_type` option controls what kind of class the plugin generates for each +table. All four produce the same fields with the same annotations - they differ +only in the class machinery and dependencies. + +```yaml +options: + model_type: "msgspec" # dataclass | attrs | msgspec | pydantic +``` + +## The four forms + +Given this table: + +```sql +CREATE TABLE test_field_namings +( + id bigint PRIMARY KEY NOT NULL, + outputs jsonb NOT NULL +); +``` + +the plugin generates (shown with the default `docstrings: none`): + +{{< tabs >}} + + {{< tab name="dataclass" >}} + +```python +@dataclasses.dataclass() +class TestFieldNaming: + id_: int + outputs: str +``` + +Standard library, no dependencies. The safe default. + {{< /tab >}} + + {{< tab name="attrs" >}} + +```python +@attrs.define() +class TestFieldNaming: + id_: int + outputs: str +``` + +Requires `attrs`. Slotted classes with concise definitions. + {{< /tab >}} + + {{< tab name="msgspec" >}} + +```python +class TestFieldNaming(msgspec.Struct): + id_: int + outputs: str +``` + +Requires `msgspec`. Very fast serialization/validation; pairs well with +[working with JSON](/docs/guide/working-with-json). + {{< /tab >}} + + {{< tab name="pydantic" >}} + +```python +class TestFieldNaming(pydantic.BaseModel): + model_config = pydantic.ConfigDict(arbitrary_types_allowed=True) + + id_: int + outputs: str +``` + +Requires `pydantic >= 2.9`. Full runtime validation on construction. + {{< /tab >}} + +{{< /tabs >}} + +Which to pick: `dataclass` if you want zero dependencies, `msgspec` for speed and +JSON, `pydantic` for runtime validation, `attrs` for its ergonomics. + +## Pydantic specifics + +`model_type: pydantic` differs from the others in two ways worth knowing: + +- **`arbitrary_types_allowed=True`** is set on every model. It is required for + field types pydantic has no core schema for (`memoryview` for `bytea`/`blob`, + and custom [override](/docs/guide/type-overrides) types). Those fields are still + validated with an `isinstance` check; all other fields get full validation. +- **Runtime annotations.** Unlike the other model types, pydantic resolves field + annotations at runtime, so generated files import referenced modules at module + level instead of inside `if typing.TYPE_CHECKING:`. + +{{< callout type="info" >}} + If you lint generated pydantic code with ruff, set + `lint.flake8-type-checking.runtime-evaluated-base-classes = ["pydantic.BaseModel"]` + so it does not try to move those runtime imports into a `TYPE_CHECKING` block. +{{< /callout >}} diff --git a/docs/content/docs/guide/naming.md b/docs/content/docs/guide/naming.md new file mode 100644 index 00000000..b2d579cb --- /dev/null +++ b/docs/content/docs/guide/naming.md @@ -0,0 +1,97 @@ +--- +title: Naming and identifiers +weight: 100 +prev: /docs/guide/sqlite-type-conversion +--- + +SQL identifiers are not always valid Python names, and SQL naming conventions are +not Python's. This page covers how table, column, and enum names are turned into +Python names, and how invalid ones are sanitized. + +## Table names to class names + +Table names are converted to `CamelCase` and singularized, so a table of many +rows yields a class describing one row: + +| Table | Model | +|---|---| +| `test_field_namings` | `TestFieldNaming` | +| `test_invalid_identifiers` | `TestInvalidIdentifier` | + +Two options change this: + +- **`emit_exact_table_names: true`** - skip singularization; model names mirror + table names exactly. +- **`inflection_exclude_table_names`** - a list of tables to leave + un-singularized while everything else is still singularized. Entries match both + bare and schema-qualified names. + +## Initialisms + +Segments listed in `initialisms` are fully upper-cased when building class names, +so you get `AppID` rather than `AppId`. It defaults to `["id"]`: + +```yaml +options: + initialisms: ["id", "url", "api"] +``` + +## Column names to field names + +Column names stay `snake_case`. Names that collide with Python keywords or +builtins get a trailing underscore - which is why an `id` column becomes `id_`: + +```python +class TestFieldNaming(msgspec.Struct): + id_: int + outputs: str +``` + +## Sanitization + +Identifiers that are not valid Python names are rewritten. Invalid characters +become underscores, and prefixes are added where a name would otherwise be +illegal: + +```sql +CREATE TABLE IF NOT EXISTS test_invalid_identifiers +( + id bigint PRIMARY KEY NOT NULL, + "3p%" text, + "new notes" text NOT NULL, + "%pct" text +); +``` + +```python +class TestInvalidIdentifier(msgspec.Struct): + id_: int + column_3p_: str | None + new_notes: str + column__pct: str | None +``` + +The rules at work: + +| Situation | Rule | Example | +|---|---|---| +| Invalid characters | Replaced with `_` | `new notes` -> `new_notes` | +| Column starts with a digit | Prefixed with `column_` | `3p%` -> `column_3p_` | +| Field starts with `_` | Prefixed with `column_` | `%pct` -> `column__pct` | +| Table yields a digit-leading, keyword, or empty class name | Prefixed with `Model` | `3rd_party_stats` -> `Model3RdPartyStat` | +| Enum value starts with a digit or `_` | Prefixed with `VALUE_` | `24h` -> `VALUE_24H` | +| Two results collide | Numeric suffix | `outputs`, `outputs` -> `outputs`, `outputs_2` | + +A leading underscore is prefixed rather than kept because `attrs` and `pydantic` +treat underscore-leading fields as private. + +The collision rule also covers queries that select the same column name twice - +for example a self-join selecting `a.outputs, b.outputs` produces a row class with +`outputs` and `outputs_2`. + +{{< callout type="warning" >}} + Sanitization checks characters with Go's Unicode tables, which differ from + Python's identifier rules in exotic cases - characters that may not *start* an + identifier in Python, or two names Python normalizes to the same identifier via + NFKC. Such schemas are not detected. Stick to ASCII identifiers if in doubt. +{{< /callout >}} diff --git a/docs/content/docs/guide/sqlite-type-conversion.md b/docs/content/docs/guide/sqlite-type-conversion.md new file mode 100644 index 00000000..99532ad5 --- /dev/null +++ b/docs/content/docs/guide/sqlite-type-conversion.md @@ -0,0 +1,100 @@ +--- +title: SQLite type conversion +weight: 90 +prev: /docs/guide/docstrings +next: /docs/guide/naming +--- + +SQLite only stores a handful of native types, so dates, decimals, booleans and +blobs have to be translated in both directions. For the `sqlite3` and `aiosqlite` +drivers the generated code registers **adapters** (Python value -> SQLite) and +**converters** (SQLite value -> Python) for you. + +## Enable declared-type parsing + +Converters only run when the connection parses declared column types, so you must +open the connection with `PARSE_DECLTYPES`: + +{{< tabs >}} + + {{< tab name="sqlite3" >}} + +```python +conn = sqlite3.connect("app.db", detect_types=sqlite3.PARSE_DECLTYPES) +``` + + {{< /tab >}} + + {{< tab name="aiosqlite" >}} + +```python +import sqlite3 + +import aiosqlite + + +async def main() -> None: + async with aiosqlite.connect("app.db", detect_types=sqlite3.PARSE_DECLTYPES) as conn: + ... +``` + + {{< /tab >}} + +{{< /tabs >}} + +{{< callout type="warning" >}} + Without `detect_types=PARSE_DECLTYPES` the converters never fire and you get raw + strings and bytes back instead of `datetime`, `Decimal`, `bool`, or `memoryview` + values. This is the most common SQLite setup mistake. +{{< /callout >}} + +## What gets converted + +| Declared SQL type | Python type | Sent to SQLite as | Read back with | +|---|---|---|---| +| `date` | `datetime.date` | `str` | `datetime.date.fromisoformat` | +| `datetime`, `timestamp` | `datetime.datetime` | `str` | `datetime.datetime.fromisoformat` | +| `decimal`, `decimal(p,s)` | `decimal.Decimal` | `str` | `decimal.Decimal` | +| `bool`, `boolean` | `bool` | `int` | `bool(int(...))` | +| `blob` | `memoryview` | `bytes` | `memoryview` | + +The converter is keyed on the column's **declared type name**, which is why the +declared types in your schema matter for SQLite. + +## Only what each module needs + +Registration is emitted per generated module, and only for what that module +actually uses: + +- A type used as a **parameter** gets its **adapter** registered. +- A type used in a **returned column** gets its **converter** registered. + +{{< callout type="info" >}} + `register_converter` is process-global in Python's `sqlite3`, so a registration + made by one generated module applies to every connection in the process. + Registering only what a module needs avoids installing conversions it never + uses - it does not isolate modules from one another, and two modules + registering the same declared type will share that conversion. +{{< /callout >}} + +## Interaction with overrides + +A column with a [type override](/docs/guide/type-overrides) or +[converter](/docs/guide/converters) is converted inline by the generated code, so +**no** SQLite converter is registered for it - registering one would convert the +value twice. Overridden values used as *parameters* still need the adapter, so +that half is still registered. + +## Faster date parsing + +`speedups: true` swaps the standard-library date parsing for +[`ciso8601`](https://github.com/closeio/ciso8601): + +```yaml +options: + speedups: true +``` + +This affects the `date` and `datetime`/`timestamp` converters only - `decimal`, +`bool` and `blob` are unchanged. It adds `ciso8601` as a runtime dependency of the +generated code, so only enable it if you install that package. diff --git a/docs/content/docs/guide/type-overrides.md b/docs/content/docs/guide/type-overrides.md new file mode 100644 index 00000000..29d6da27 --- /dev/null +++ b/docs/content/docs/guide/type-overrides.md @@ -0,0 +1,90 @@ +--- +title: Type overrides +weight: 60 +prev: /docs/guide/enums +next: /docs/guide/converters +--- + +An override replaces the Python type the plugin would pick for a column with one +of your own. You can target a specific column or every column of a given SQL +type. + +## Specification + +Each entry in `overrides` matches columns in one of two ways (exactly one is +required): + +- **`column`** - a specific column, as `[schema.]table.column` (wildcards + allowed). +- **`db_type`** - every column of a SQL type, e.g. `text`. + +and supplies a `py_type` describing the replacement type and how to import it +(`import`, `package`, `type`). Full schema in the +[reference](/docs/reference/configuration-options#override). + +## Example + +Override one `text` column with `collections.UserString`: + +```yaml +options: + overrides: + - column: test_type_override.text_test + py_type: + import: collections + package: UserString + type: UserString +``` + +The model field now uses your type: + +```python +class TestTypeOverride(msgspec.Struct): + id_: int + text_test: UserString | None +``` + +and query functions convert on the way in and out: + +```python +# read: construct your type from the column value +return models.TestTypeOverride(id_=row[0], text_test=UserString(row[1]) if row[1] is not None else None) + +# write: convert back to the column's default type (str for text) +await conn.execute(INSERT_TYPE_OVERRIDE, id_, str(text_test) if text_test is not None else None) +``` + +## How conversion works + +An override converts by **calling the type**. On read the plugin builds the value +with `YourType(column_value)`; on write it calls the column's default type +(`str(...)` here) to turn it back. Nullable columns are guarded, so a SQL `NULL` +stays `None` and the type is never called with `None`. + +This is why an override works for anything constructible from its wire value +(`UserString(str)`, `PurePosixPath(str)`, ...) but *not* for things like JSON, +where `MyModel(json_string)` is meaningless. For those, use a +[converter](/docs/guide/converters), which names your own encode/decode functions +instead. + +## Matching by SQL type + +Use `db_type` to override every column of a type at once: + +```yaml +overrides: + - db_type: text + py_type: + import: collections + package: UserString + type: UserString +``` + +When both a `column` and a `db_type` override could match a column, the `column` +override wins. + +{{< callout type="warning" >}} + Column overrides do not attach to `ANY($1::type[])` array parameters - sqlc does + not link a column override to those. Only `db_type` overrides (and converters + matched by `db_type`) reach them. +{{< /callout >}} diff --git a/docs/content/docs/guide/working-with-json.md b/docs/content/docs/guide/working-with-json.md new file mode 100644 index 00000000..f02f621f --- /dev/null +++ b/docs/content/docs/guide/working-with-json.md @@ -0,0 +1,264 @@ +--- +title: Working with JSON +weight: 75 +prev: /docs/guide/converters +next: /docs/guide/docstrings +--- + +A `json`/`jsonb` column maps to `str` by default - correct, but untyped: you get +a string and parse it yourself everywhere. With a [converter](/docs/guide/converters) +and a `msgspec.Struct` you get a fully typed, validated Python object instead, +and the plugin inserts the encode/decode calls for you. + +## Why JSON needs a converter + +A [type override](/docs/guide/type-overrides) converts by *calling the type*, so +it would generate `Preferences(row[2])` - handing a JSON string to a struct +constructor, which is meaningless. JSON needs a real parse step, which is exactly +what a converter's `from_db`/`to_db` pair provides. + +## The wire type contract + +This is the one rule to internalize: + +{{< callout type="info" >}} + A `jsonb` (or `json`) column's wire type is **`str`**. So `to_db` must *return* + `str`, and `from_db` *receives* `str`. +{{< /callout >}} + +That matters for `msgspec`, because `msgspec.json.encode()` returns **`bytes`** - +you must `.decode()` it. `msgspec.json.decode()` happily accepts either. + +## A complete example + +A `users` table with a nested preferences document: + +```sql +CREATE TABLE users +( + id bigint PRIMARY KEY NOT NULL, + name text NOT NULL, + preferences jsonb NOT NULL +); +``` + +```sql +-- name: GetUser :one +SELECT * FROM users WHERE id = $1; + +-- name: CreateUser :exec +INSERT INTO users (id, name, preferences) VALUES ($1, $2, $3); +``` + +{{< tabs >}} + + {{< tab name="models.py" >}} + +```python +import msgspec + + +class Theme(msgspec.Struct): + name: str + dark_mode: bool = False + + +class Preferences(msgspec.Struct): + theme: Theme + languages: list[str] = msgspec.field(default_factory=list) + email_opt_in: bool | None = None +``` + +Structs nest freely - `Preferences` holds a `Theme`, plus a list and an optional +field. `msgspec` handles all of it. + {{< /tab >}} + + {{< tab name="converters.py" >}} + +```python +import msgspec + +from myapp.models import Preferences + + +def encode_preferences(value: Preferences) -> str: + # jsonb's wire type is str, and msgspec encodes to bytes. + return msgspec.json.encode(value).decode() + + +def decode_preferences(value: str) -> Preferences: + return msgspec.json.decode(value, type=Preferences) +``` + +The `type=` argument is what makes decoding *typed* - msgspec builds real +`Preferences` objects and validates the payload while parsing. + {{< /tab >}} + + {{< tab name="sqlc.yaml" >}} + +```yaml +options: + model_type: "msgspec" + converters: + - name: preferences + py_type: + import: myapp.models + package: Preferences + type: Preferences + to_db: myapp.converters.encode_preferences + from_db: myapp.converters.decode_preferences + overrides: + - column: users.preferences + converter: preferences +``` + + {{< /tab >}} + +{{< /tabs >}} + +### What gets generated + +The model field is typed as your struct, and both query functions call your +converter: + +```python +class User(msgspec.Struct): + id_: int + name: str + preferences: Preferences + + +async def get_user(conn: ConnectionLike, *, id_: int) -> models.User | None: + row = await conn.fetchrow(GET_USER, id_) + if row is None: + return None + return models.User(id_=row[0], name=row[1], preferences=myapp.converters.decode_preferences(row[2])) + + +async def create_user(conn: ConnectionLike, *, id_: int, name: str, preferences: Preferences) -> None: + await conn.execute(CREATE_USER, id_, name, myapp.converters.encode_preferences(preferences)) +``` + +### Using it + +```python +async def main() -> None: + await create_user( + conn, + id_=1, + name="ada", + preferences=Preferences(theme=Theme(name="solarized", dark_mode=True), languages=["en"]), + ) + + user = await get_user(conn, id_=1) + if user is not None: + print(user.preferences.theme.dark_mode) # True - typed all the way down +``` + +No `json.loads`, no `dict["theme"]["dark_mode"]`, and pyright checks every access. + +## Several JSON columns with different shapes + +Match **per column**, one converter per struct: + +```yaml +options: + converters: + - name: preferences + py_type: { import: myapp.models, package: Preferences, type: Preferences } + to_db: myapp.converters.encode_preferences + from_db: myapp.converters.decode_preferences + - name: audit_meta + py_type: { import: myapp.models, package: AuditMeta, type: AuditMeta } + to_db: myapp.converters.encode_audit_meta + from_db: myapp.converters.decode_audit_meta + overrides: + - column: users.preferences + converter: preferences + - column: audit_log.meta + converter: audit_meta +``` + +{{< callout type="warning" >}} + Reaching for `db_type: jsonb` applies **one** struct to *every* `jsonb` column + in your schema. That is only what you want when all of them genuinely share a + shape - otherwise match by `column`. +{{< /callout >}} + +## Nullable and array JSON columns + +- **Nullable** (`preferences jsonb` without `NOT NULL`): the field becomes + `Preferences | None` and the plugin guards the call, so `decode_preferences` + never receives `None`. +- **Arrays** (`jsonb[]`): the field becomes + `collections.abc.Sequence[Preferences]` and conversion happens element-wise - + your functions still deal with a single value at a time. To also cover an + `ANY($1::jsonb[])` parameter, match with `db_type: jsonb`. + +## Validation + +`msgspec.json.decode(..., type=Preferences)` validates while parsing and raises +`msgspec.ValidationError` when a stored document does not match the struct (a +missing required field, a wrong type). That is usually what you want - it surfaces +bad data at the boundary. If you would rather tolerate legacy rows, give fields +defaults or widen them to `| None`. + +## Other libraries + +The converter contract is just "return the wire type, accept the wire type", so +any library works - as long as your model and your converter use the same one. +The examples below assume `Theme`/`Preferences` are defined with that library +rather than as `msgspec.Struct`s. + +{{< tabs >}} + + {{< tab name="stdlib json" >}} +With `Theme` and `Preferences` defined as `dataclasses.dataclass`: + +```python +import dataclasses +import json + +from myapp.models import Preferences, Theme + + +def encode_preferences(value: Preferences) -> str: + return json.dumps(dataclasses.asdict(value)) + + +def decode_preferences(value: str) -> Preferences: + raw = json.loads(value) + return Preferences( + theme=Theme(**raw["theme"]), + languages=raw.get("languages", []), + email_opt_in=raw.get("email_opt_in"), + ) +``` + +No dependencies, but no validation - and nested objects must be rebuilt by hand, +since `json.loads` leaves `theme` as a plain `dict`. + {{< /tab >}} + + {{< tab name="pydantic" >}} +With `Theme` and `Preferences` defined as `pydantic.BaseModel`: + +```python +from myapp.models import Preferences + + +def encode_preferences(value: Preferences) -> str: + return value.model_dump_json() + + +def decode_preferences(value: str) -> Preferences: + return Preferences.model_validate_json(value) +``` + +Full validation, nested models rebuilt for you, and `model_validate_json` parses +straight from the string. + {{< /tab >}} + +{{< /tabs >}} + +The same approach works for the SQLite drivers: a `json` column is `str` there +too, so the identical converter applies. diff --git a/docs/content/docs/guide/writing-queries.md b/docs/content/docs/guide/writing-queries.md new file mode 100644 index 00000000..7f4dd034 --- /dev/null +++ b/docs/content/docs/guide/writing-queries.md @@ -0,0 +1,175 @@ +--- +title: Writing queries +weight: 40 +prev: /docs/guide/model-types +next: /docs/guide/enums +--- + +Each query `.sql` file becomes one Python module, and each annotated query in it +becomes one typed function. This page shows how the `-- name:` annotation maps to +a function and what each query command generates. + +## Anatomy of a query + +A query is a `-- name: :` comment followed by SQL: + +```sql +-- name: GetFieldNaming :one +SELECT id, outputs +FROM test_field_namings +WHERE id = $1 LIMIT 1; +``` + +From that the plugin generates: + +```python +GET_FIELD_NAMING: typing.Final[str] = """-- name: GetFieldNaming :one +SELECT id, outputs +FROM test_field_namings +WHERE id = $1 LIMIT 1 +""" + + +async def get_field_naming(conn: ConnectionLike, *, id_: int) -> models.TestFieldNaming | None: + row = await conn.fetchrow(GET_FIELD_NAMING, id_) + if row is None: + return None + return models.TestFieldNaming(id_=row[0], outputs=row[1]) +``` + +- The SQL is stored once as a module-level constant. +- `GetFieldNaming` becomes the function `get_field_naming`. +- The connection is the first positional argument (`conn`); its type + `ConnectionLike` is a generated alias for the driver's connection (or pool). +- Query parameters (`$1`) become keyword-only arguments (`id_`). +- The `:one` command sets the return type and body. + +## Query commands + +### `:one` + +Returns the row or `None`. The row is a `models.*` class when the query's columns +match a table, a generated `Row` class when they do not, or a bare scalar +when the query selects a single column. Shown above. + +When a query's columns do not match one table exactly (a join, a partial select, +an aggregate), the plugin generates a dedicated `Row` class instead of +reusing a table model: + +```python +class GetJoinedFieldNamingsRow(msgspec.Struct): + outputs: str + outputs_2: str + + +async def get_joined_field_namings(conn: ConnectionLike, *, id_: int) -> GetJoinedFieldNamingsRow | None: + row = await conn.fetchrow(GET_JOINED_FIELD_NAMINGS, id_) + if row is None: + return None + return GetJoinedFieldNamingsRow(outputs=row[0], outputs_2=row[1]) +``` + +### `:many` + +Returns a `QueryResults[T]` - a helper that supports both async iteration and +one-shot fetching, so you do not pay for materializing the whole result set +unless you want it: + +```python +def get_many_test_timestamp_postgres_type(conn: ConnectionLike, *, id_: int) -> QueryResults[datetime.datetime]: + return QueryResults(conn, GET_MANY_TEST_TIMESTAMP_POSTGRES_TYPE, operator.itemgetter(0), id_) +``` + +Here the query selects one column, so `T` is a scalar (`datetime.datetime`). A +`:many` that selects full rows returns `QueryResults[models.Foo]` (or a +`Row`). + +{{< callout type="info" >}} + Note that a `:many` function is **not** a coroutine, even on the async drivers - + it returns the `QueryResults` helper synchronously. You await (or iterate) the + helper, not the call. +{{< /callout >}} + +### `:exec` + +Runs the statement and returns `None`: + +```python +async def set_field_naming_outputs(conn: ConnectionLike, *, id_: int, outputs: str, outputs_2: str) -> None: + await conn.execute(SET_FIELD_NAMING_OUTPUTS, id_, outputs, outputs_2) +``` + +### `:execresult`, `:execrows`, `:execlastid` + +Variants of `:exec` that return something about the write: + +- **`:execrows`** - the number of affected rows (`int`). For statements that + affect no rows, such as `CREATE TABLE`, asyncpg reports `0` and the SQLite + drivers report `-1`. +- **`:execlastid`** - the cursor's `lastrowid`, typed `int | None` - it is `None` + when no row was affected. SQLite drivers only, and note it is the last + *affected* row, not strictly the last inserted one. +- **`:execresult`** - the driver's raw result, which differs per driver: a `str` + status tag on asyncpg, and a `sqlite3.Cursor` / `aiosqlite.Cursor` on the + SQLite drivers. + +See the [feature support matrix](/docs/reference/feature-support) for which +driver supports which. + +### `:copyfrom` + +asyncpg only. Bulk-inserts rows via `copy_records_to_table`, taking a sequence of +generated `Params` objects and returning the affected row count: + +```python +async def test_copy_from(conn: ConnectionLike, *, params: collections.abc.Sequence[TestCopyFromParams]) -> int: + records = [(param.id_, param.float_test, param.int_test) for param in params] + r = await conn.copy_records_to_table("test_copy_from", columns=["id", "float_test", "int_test"], records=records) + return int(n) if (p := r.split()) and (n := p[-1]).isdigit() else 0 +``` + +## Parameters + +By default every parameter is **keyword-only** (note the `*,` in the signatures +above), which keeps call sites readable and order-independent. Two options tune +this: + +- **`omit_kwargs_limit`** - queries with this many parameters or fewer allow + positional arguments. Defaults to `0` (always keyword-only). +- **`query_parameter_limit`** - when set to a non-negative value, queries with + more parameters than the limit bundle them into a single generated + `Params` object instead of expanding them into the signature. Left unset + or negative, parameters are never bundled. `:copyfrom` always uses a params + class regardless. + +## Grouping into a class + +With `emit_classes: true`, the standalone functions of each query file become +methods on a class named after that file - `queries_field_namings.sql` yields +`QueriesFieldNamings` - so you get one class per query module rather than one +class overall. + +The connection is passed once to the constructor and is also exposed as a +read-only `conn` property. The bodies are otherwise unchanged, except that `conn` +becomes `self._conn`: + +```python +class QueriesFieldNamings: + __slots__ = ("_conn",) + + def __init__(self, conn: ConnectionLike) -> None: + self._conn = conn + + @property + def conn(self) -> ConnectionLike: + return self._conn + + async def get_field_naming(self, *, id_: int) -> models.TestFieldNaming | None: + row = await self._conn.fetchrow(GET_FIELD_NAMING, id_) + ... +``` + +so the call becomes +`user = await QueriesFieldNamings(conn).get_field_naming(id_=1)` - without the +`await` on the synchronous `sqlite3` driver, where the method is a plain +function. diff --git a/docs/content/docs/reference/_index.md b/docs/content/docs/reference/_index.md new file mode 100644 index 00000000..9c91167a --- /dev/null +++ b/docs/content/docs/reference/_index.md @@ -0,0 +1,16 @@ +--- +title: Reference +weight: 3 +prev: /docs/guide +sidebar: + open: true +--- + +Canonical lookup material. The [Guide](/docs/guide) explains when and why to use +a feature; the Reference is the exhaustive *what*. + +{{< cards >}} + {{< card link="configuration-options" title="Configuration options" subtitle="Every plugin option, its type, default, and meaning." >}} + {{< card link="type-mappings" title="Type mappings" subtitle="How SQL types map to Python types, per engine." >}} + {{< card link="feature-support" title="Feature support" subtitle="Supported macros and per-driver query commands." >}} +{{< /cards >}} diff --git a/docs/content/docs/reference/configuration-options.md b/docs/content/docs/reference/configuration-options.md new file mode 100644 index 00000000..f3543a0d --- /dev/null +++ b/docs/content/docs/reference/configuration-options.md @@ -0,0 +1,78 @@ +--- +title: Configuration options +weight: 10 +prev: /docs/reference +next: /docs/reference/type-mappings +--- + +Every option the plugin accepts, set under `options:` in the plugin's `codegen` +block. See [Configuration](/docs/guide/configuration) for where this block lives +in `sqlc.yaml`. + +## Options + +`package`, `sql_driver`, and `emit_init_file` are required; everything else is +optional. + +| Option | Type | Default | Description | +|---|---|---|---| +| `package` | string | *required* | Name of the generated package. | +| `sql_driver` | string | *required* | One of `asyncpg`, `aiosqlite`, `sqlite3`. Must match the engine (`asyncpg` -> `postgresql`; the other two -> `sqlite`). | +| `emit_init_file` | bool | *required* | Whether to emit an `__init__.py` in the package. Must be set explicitly. Set `false` only if the package already has one. | +| `model_type` | string | `dataclass` | One of `dataclass`, `attrs`, `msgspec`, `pydantic`. See [Model types](/docs/guide/model-types). | +| `initialisms` | list[string] | `["id"]` | Identifier segments to upper-case, e.g. `app_id` -> `AppID`. | +| `emit_exact_table_names` | bool | `false` | If `true`, model names mirror table names; otherwise plural table names are singularized. | +| `emit_classes` | bool | `false` | If `true`, each query file's functions become methods on a class named after that file (`queries_field_namings.sql` -> `QueriesFieldNamings`). See [Writing queries](/docs/guide/writing-queries). | +| `inflection_exclude_table_names` | list[string] | `[]` | Table names to leave un-singularized (only used when `emit_exact_table_names` is `false`). | +| `omit_unused_models` | bool | `false` | If `true`, tables not referenced by any query are not turned into models. | +| `omit_typechecking_block` | bool | `false` | If `true`, non-builtin types are imported at module level instead of inside a `typing.TYPE_CHECKING` block. | +| `docstrings` | string | `none` | One of `none`, `google`, `numpy`, `pep257`. See [Docstrings](/docs/guide/docstrings). | +| `docstrings_emit_sql` | bool | `true` | Include each query's SQL in its docstring. Ignored when `docstrings` is `none`. | +| `query_parameter_limit` | int | *unset* | When set to a non-negative value, queries with more parameters than the limit bundle them into a single `params` argument. Unset or negative never bundles (except `:copyfrom`, which always uses a params class). | +| `omit_kwargs_limit` | int | `0` | Queries with this many parameters or fewer do not require keyword arguments. `0` makes every parameter keyword-only. Must not be negative. | +| `speedups` | bool | `false` | Use faster third-party libraries for type conversion. Currently affects `sqlite3`/`aiosqlite` only (uses `ciso8601`). See [SQLite type conversion](/docs/guide/sqlite-type-conversion). | +| `overrides` | list[Override] | `[]` | Replace the Python type of matching columns. See [Type overrides](/docs/guide/type-overrides). | +| `converters` | list[Converter] | `[]` | Named encode/decode function pairs referenced by an override. See [Converters](/docs/guide/converters). | +| `indent_char` | string | `" "` (space) | Character used for one indent step. | +| `chars_per_indent_level` | int | `4` | Number of `indent_char`s per indent level. | +| `debug` | bool | `false` | Emit a `log.txt` debug log during `sqlc generate`. | + +## Object schemas + +Some options take structured objects rather than scalars. + +### `py_type` + +Describes a Python type and how to import it. Used by an `override` and by a +`converter`. + +| Field | Type | Description | +|---|---|---| +| `type` | string | *required* - the type expression used in annotations, e.g. `UserString` or `collections.UserString`. | +| `import` | string | Module to import, e.g. `collections`. | +| `package` | string | Name to import from `import` (`from import `). If empty, emits `import `. | + +### `override` + +An entry in `overrides`. Matches columns either by SQL type or by column name; +specifying both, or neither, is an error. + +| Field | Type | Description | +|---|---|---| +| `db_type` | string | Match a SQL type exactly (case-insensitive), e.g. `text`, `pg_catalog.int4`. Mutually exclusive with `column`. | +| `column` | string | Match `[catalog.][schema.]table.column`; wildcards are supported. Mutually exclusive with `db_type`. | +| `py_type` | py_type | Replacement type. Required unless `converter` is set. | +| `converter` | string | Name of a `converters` entry, which supplies the type and its encode/decode functions instead of `py_type`. | + +When both a `column` and a `db_type` override could match, the column match wins. + +### `converter` + +An entry in `converters`. See [Converters](/docs/guide/converters). + +| Field | Type | Description | +|---|---|---| +| `name` | string | *required* - unique name, referenced by an override's `converter`. | +| `py_type` | py_type | *required* - the Python type the converter produces and accepts. | +| `to_db` | string | *required* - dotted path to a function that turns the Python value into the column's wire type. | +| `from_db` | string | *required* - dotted path to a function that turns the wire value into the Python type. | diff --git a/docs/content/docs/reference/feature-support.md b/docs/content/docs/reference/feature-support.md new file mode 100644 index 00000000..bc020d22 --- /dev/null +++ b/docs/content/docs/reference/feature-support.md @@ -0,0 +1,53 @@ +--- +title: Feature support +weight: 30 +prev: /docs/reference/type-mappings +--- + +Which sqlc features and query commands the plugin supports. + +## Macros + +Every [sqlc macro](https://docs.sqlc.dev/en/latest/reference/macros.html) is +supported (`sqlc.arg`, `sqlc.narg`, `sqlc.embed`, `sqlc.slice`). + +{{< callout type="info" >}} + Known issue in `v0.5.1`: a query using `sqlc.slice` is generated with sqlc's + `/*SLICE:name*/` placeholder left in place, so it fails at runtime. A fix is + landing before the next release - see + [issue #210](https://github.com/rayakame/sqlc-gen-better-python/issues/210). + On PostgreSQL, `= ANY($1::type[])` is the idiomatic form anyway and is + unaffected. +{{< /callout >}} + +## Query commands + +The supported [query annotations](https://docs.sqlc.dev/en/latest/reference/query-annotations.html) +depend on the driver: + +| Command | aiosqlite | sqlite3 | asyncpg | +|---|---|---|---| +| `:one` | yes | yes | yes | +| `:many` | yes | yes | yes | +| `:exec` | yes | yes | yes | +| `:execresult` | yes | yes | yes | +| `:execrows` | yes | yes | yes | +| `:execlastid` | yes | yes | no | +| `:copyfrom` | no | no | yes | + +See [Writing queries](/docs/guide/writing-queries) for what each command +generates. + +{{< callout type="info" >}} + `:execlastid` relies on a last-inserted-row id, which asyncpg/PostgreSQL do not + provide; use a `RETURNING` clause with `:one` instead. `:copyfrom` maps to + asyncpg's bulk `copy_records_to_table`, which the SQLite drivers have no + equivalent for. +{{< /callout >}} + +## Not supported + +- **`:batch*` commands** (`:batchexec`, `:batchmany`, `:batchone`) are not + supported and likely never will be. +- **Prepared queries** are not planned for the near future. +- **`psycopg2` and `mysql`** drivers are not currently supported. diff --git a/docs/content/docs/reference/type-mappings.md b/docs/content/docs/reference/type-mappings.md new file mode 100644 index 00000000..187f05e7 --- /dev/null +++ b/docs/content/docs/reference/type-mappings.md @@ -0,0 +1,74 @@ +--- +title: Type mappings +weight: 20 +prev: /docs/reference/configuration-options +next: /docs/reference/feature-support +--- + +How the plugin maps SQL types to Python types, per engine. These are the +built-in defaults; a [type override](/docs/guide/type-overrides) or +[converter](/docs/guide/converters) replaces the type for a specific column or +SQL type. + +## How a column's type is built + +The tables below give the **base** Python type for a SQL type. Two modifiers are +applied on top, based on the column: + +- **Nullable** (the column is not `NOT NULL`): `| None` is appended, e.g. + `str | None`. +- **Array / list** (a Postgres array column, or a `sqlc.slice` parameter): the + base type is wrapped as `collections.abc.Sequence[T]`. + +Both can combine: a nullable `bytea[]` becomes +`collections.abc.Sequence[memoryview] | None`. + +## PostgreSQL + +| SQL type | Python type | +|---|---| +| `smallint`, `integer`, `bigint` (and `int2`/`int4`/`int8`, `smallserial`/`serial`/`bigserial`, `pg_catalog.*` forms) | `int` | +| `real`, `double precision` (`float4`/`float8`) | `float` | +| `numeric` | `decimal.Decimal` | +| `money` | `str` | +| `boolean` (`bool`) | `bool` | +| `json`, `jsonb` | `str` | +| `bytea` | `memoryview` | +| `date` | `datetime.date` | +| `time`, `timetz` | `datetime.time` | +| `timestamp`, `timestamptz` | `datetime.datetime` | +| `interval` | `datetime.timedelta` | +| `text`, `varchar`, `char`/`bpchar`, `citext` | `str` | +| `uuid` | `uuid.UUID` | +| `inet`, `cidr`, `macaddr`, `macaddr8` | `str` | +| `ltree`, `lquery`, `ltxtquery` | `str` | +| a user-defined `ENUM` type | the generated [enum class](/docs/guide/enums), e.g. `enums.Mood` | +| anything else | `typing.Any` | + +{{< callout type="info" >}} + A SQL type the plugin does not recognize falls back to `typing.Any`. If that + happens for a type you use often, add a [type override](/docs/guide/type-overrides) + or [converter](/docs/guide/converters) so it gets a real Python type. +{{< /callout >}} + +## SQLite + +SQLite type names are matched case-insensitively, by exact name first and then by +prefix (so `varchar(255)` matches `varchar`, and `decimal(10,5)` matches +`decimal`). + +| SQL type | Python type | +|---|---| +| `int`, `integer`, `tinyint`, `smallint`, `mediumint`, `bigint`, `int2`, `int8`, `bigserial` | `int` | +| `real`, `double`, `double precision`, `float`, `numeric` | `float` | +| `decimal`, `decimal(p,s)` | `decimal.Decimal` | +| `bool`, `boolean` | `bool` | +| `date` | `datetime.date` | +| `datetime`, `timestamp` | `datetime.datetime` | +| `text`, `clob`, `json`, `character`, `varchar`, `varyingcharacter`, `nchar`, `nativecharacter`, `nvarchar` | `str` | +| `blob` | `memoryview` | +| anything else | `typing.Any` | + +For the two SQLite drivers, several of these types also need runtime adapters and +converters (registered in the generated code) to round-trip correctly - see +[SQLite type conversion](/docs/guide/sqlite-type-conversion). diff --git a/docs/i18n/en.yaml b/docs/i18n/en.yaml new file mode 100644 index 00000000..5258e9fc --- /dev/null +++ b/docs/i18n/en.yaml @@ -0,0 +1,4 @@ +# Overrides theme defaults. Hugo merges these over the theme's own i18n strings. + +# Footer copyright. Kept in step with the LICENSE holder. +copyright: "Copyright (c) 2025-present Rayakame"