Skip to content

Support AUTOINCREMENT in public creation APIs and CLI #894

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
MbappeWU wants to merge 1 commit into
base: main
Choose a base branch
from
+639 −78
Open

Support AUTOINCREMENT in public creation APIs and CLI #894

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/changelog.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@
Changelog
===========

Unreleased
----------

- The Python table creation, insert, upsert and lookup APIs now accept ``autoincrement=`` for single integer primary keys. The ``create-table``, ``insert``, ``upsert`` and ``transform`` commands accept ``--autoincrement`` and ``--no-autoincrement``. Existing table mode changes can be applied or previewed with ``transform()`` and ``transform_sql()``. (:issue:`664`)

.. _v4_2_1:

4.2.1 (2026-08-13)
Expand Down
171 changes: 96 additions & 75 deletions docs/cli-reference.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -287,40 +287,47 @@ See :ref:`cli_inserting_data`, :ref:`cli_insert_csv_tsv`, :ref:`cli_insert_unstr
' --pk id

Options:
--pk TEXT Columns to use as the primary key, e.g. id
--code TEXT Python code defining a rows() function or iterable
of rows to insert
--flatten Flatten nested JSON objects, so {"a": {"b": 1}}
becomes {"a_b": 1}
--nl Expect newline-delimited JSON
-c, --csv Expect CSV input
--tsv Expect TSV input
--empty-null Treat empty strings as NULL
--lines Treat each line as a single value called 'line'
--text Treat input as a single value called 'text'
--convert TEXT Python code to convert each item
--import TEXT Python modules to import
--delimiter TEXT Delimiter to use for CSV files
--quotechar TEXT Quote character to use for CSV/TSV
--sniff Detect delimiter and quote character
--no-headers CSV file has no header row
--encoding TEXT Character encoding for input, defaults to utf-8
--batch-size INTEGER Commit every X records
--stop-after INTEGER Stop after X records
--alter Alter existing table to add any missing columns
--not-null TEXT Columns that should be created as NOT NULL
--default <TEXT TEXT>... Default value that should be set for a column
--type <TEXT CHOICE>... Column types to use when creating the table
--no-detect-types Treat all CSV/TSV columns as TEXT
--analyze Run ANALYZE at the end of this operation
--load-extension TEXT Path to SQLite extension, with optional :entrypoint
--silent Do not show progress bar
--strict Apply STRICT mode to created table
--ignore Ignore records if pk already exists
--replace Replace records if pk already exists
--truncate Truncate table before inserting records, if table
already exists
-h, --help Show this message and exit.
--pk TEXT Columns to use as the primary key, e.g. id
--code TEXT Python code defining a rows() function or
iterable of rows to insert
--flatten Flatten nested JSON objects, so {"a": {"b":
1}} becomes {"a_b": 1}
--nl Expect newline-delimited JSON
-c, --csv Expect CSV input
--tsv Expect TSV input
--empty-null Treat empty strings as NULL
--lines Treat each line as a single value called
'line'
--text Treat input as a single value called 'text'
--convert TEXT Python code to convert each item
--import TEXT Python modules to import
--delimiter TEXT Delimiter to use for CSV files
--quotechar TEXT Quote character to use for CSV/TSV
--sniff Detect delimiter and quote character
--no-headers CSV file has no header row
--encoding TEXT Character encoding for input, defaults to
utf-8
--batch-size INTEGER Commit every X records
--stop-after INTEGER Stop after X records
--alter Alter existing table to add any missing
columns
--not-null TEXT Columns that should be created as NOT NULL
--default <TEXT TEXT>... Default value that should be set for a column
--type <TEXT CHOICE>... Column types to use when creating the table
--no-detect-types Treat all CSV/TSV columns as TEXT
--analyze Run ANALYZE at the end of this operation
--load-extension TEXT Path to SQLite extension, with optional
:entrypoint
--silent Do not show progress bar
--strict Apply STRICT mode to created table
--autoincrement / --no-autoincrement
Use AUTOINCREMENT for the INTEGER PRIMARY KEY
when creating the table
--ignore Ignore records if pk already exists
--replace Replace records if pk already exists
--truncate Truncate table before inserting records, if
table already exists
-h, --help Show this message and exit.


.. _cli_ref_upsert:
Expand Down Expand Up @@ -351,36 +358,43 @@ See :ref:`cli_upsert`.
]' | sqlite-utils upsert data.db chickens - --pk id

Options:
--pk TEXT Columns to use as the primary key, e.g. id
--code TEXT Python code defining a rows() function or iterable
of rows to insert
--flatten Flatten nested JSON objects, so {"a": {"b": 1}}
becomes {"a_b": 1}
--nl Expect newline-delimited JSON
-c, --csv Expect CSV input
--tsv Expect TSV input
--empty-null Treat empty strings as NULL
--lines Treat each line as a single value called 'line'
--text Treat input as a single value called 'text'
--convert TEXT Python code to convert each item
--import TEXT Python modules to import
--delimiter TEXT Delimiter to use for CSV files
--quotechar TEXT Quote character to use for CSV/TSV
--sniff Detect delimiter and quote character
--no-headers CSV file has no header row
--encoding TEXT Character encoding for input, defaults to utf-8
--batch-size INTEGER Commit every X records
--stop-after INTEGER Stop after X records
--alter Alter existing table to add any missing columns
--not-null TEXT Columns that should be created as NOT NULL
--default <TEXT TEXT>... Default value that should be set for a column
--type <TEXT CHOICE>... Column types to use when creating the table
--no-detect-types Treat all CSV/TSV columns as TEXT
--analyze Run ANALYZE at the end of this operation
--load-extension TEXT Path to SQLite extension, with optional :entrypoint
--silent Do not show progress bar
--strict Apply STRICT mode to created table
-h, --help Show this message and exit.
--pk TEXT Columns to use as the primary key, e.g. id
--code TEXT Python code defining a rows() function or
iterable of rows to insert
--flatten Flatten nested JSON objects, so {"a": {"b":
1}} becomes {"a_b": 1}
--nl Expect newline-delimited JSON
-c, --csv Expect CSV input
--tsv Expect TSV input
--empty-null Treat empty strings as NULL
--lines Treat each line as a single value called
'line'
--text Treat input as a single value called 'text'
--convert TEXT Python code to convert each item
--import TEXT Python modules to import
--delimiter TEXT Delimiter to use for CSV files
--quotechar TEXT Quote character to use for CSV/TSV
--sniff Detect delimiter and quote character
--no-headers CSV file has no header row
--encoding TEXT Character encoding for input, defaults to
utf-8
--batch-size INTEGER Commit every X records
--stop-after INTEGER Stop after X records
--alter Alter existing table to add any missing
columns
--not-null TEXT Columns that should be created as NOT NULL
--default <TEXT TEXT>... Default value that should be set for a column
--type <TEXT CHOICE>... Column types to use when creating the table
--no-detect-types Treat all CSV/TSV columns as TEXT
--analyze Run ANALYZE at the end of this operation
--load-extension TEXT Path to SQLite extension, with optional
:entrypoint
--silent Do not show progress bar
--strict Apply STRICT mode to created table
--autoincrement / --no-autoincrement
Use AUTOINCREMENT for the INTEGER PRIMARY KEY
when creating the table
-h, --help Show this message and exit.


.. _cli_ref_bulk:
Expand Down Expand Up @@ -510,6 +524,9 @@ See :ref:`cli_transform_table`.
--drop-foreign-key TEXT Drop foreign key constraint for this column
--strict / --no-strict Enable or disable STRICT mode (default:
preserve current mode)
--autoincrement / --no-autoincrement
Enable or disable AUTOINCREMENT (default:
preserve current mode)
--sql Output SQL without executing it
--load-extension TEXT Path to SQLite extension, with optional
:entrypoint
Expand Down Expand Up @@ -966,17 +983,21 @@ See :ref:`cli_create_table`.
Valid column types are text, integer, real, float, blob and any.

Options:
--pk TEXT Column to use as primary key
--not-null TEXT Columns that should be created as NOT NULL
--default <TEXT TEXT>... Default value that should be set for a column
--fk <TEXT TEXT TEXT>... Column, other table, other column to set as a
foreign key
--ignore If table already exists, do nothing
--replace If table already exists, replace it
--transform If table already exists, try to transform the schema
--load-extension TEXT Path to SQLite extension, with optional :entrypoint
--strict Apply STRICT mode to created table
-h, --help Show this message and exit.
--pk TEXT Column to use as primary key
--not-null TEXT Columns that should be created as NOT NULL
--default <TEXT TEXT>... Default value that should be set for a column
--fk <TEXT TEXT TEXT>... Column, other table, other column to set as a
foreign key
--ignore If table already exists, do nothing
--replace If table already exists, replace it
--transform If table already exists, try to transform the
schema
--load-extension TEXT Path to SQLite extension, with optional
:entrypoint
--strict Apply STRICT mode to created table
--autoincrement / --no-autoincrement
Use AUTOINCREMENT for the INTEGER PRIMARY KEY
-h, --help Show this message and exit.


.. _cli_ref_create_index:
Expand Down
19 changes: 19 additions & 0 deletions docs/cli.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2079,6 +2079,22 @@ You can pass as many column-name column-type pairs as you like. Valid types are

Pass ``--pk`` more than once for a compound primary key that covers multiple columns.

For a single integer primary key, add ``--autoincrement`` to prevent reuse of deleted IDs:

.. code-block:: bash

sqlite-utils create-table events.db events id integer name text --pk id --autoincrement

The option also works when ``insert`` or ``upsert`` creates a new table:

.. code-block:: bash

echo '{"name": "first event"}' | sqlite-utils insert events.db events - --pk id --autoincrement

An omitted primary key column is created as an integer column. ``upsert`` still requires primary key values in the input. Compound and non-integer primary keys cannot use this option. Ordinary integer primary keys already generate missing IDs; ``AUTOINCREMENT`` prevents their reuse and does not guarantee consecutive values. Use ``--no-autoincrement`` to explicitly create a table without it.

Insert options do not change the mode of an existing table. A conflicting explicit mode is rejected; use ``transform --autoincrement`` or ``transform --no-autoincrement`` to change it. ``create-table --transform`` also applies an explicit mode change.

You can specify columns that should be NOT NULL using ``--not-null colname``. You can specify default values for columns using ``--default colname defaultvalue``.

.. code-block:: bash
Expand Down Expand Up @@ -2271,6 +2287,9 @@ Every option for this table (with the exception of ``--pk-none``) can be specifi
``--add-foreign-key column other_table other_column``
Add a foreign key constraint to ``column`` pointing to ``other_table.other_column``.

``--autoincrement`` / ``--no-autoincrement``
Enable or disable ``AUTOINCREMENT`` for a single integer primary key. If neither option is supplied the current mode is preserved. Enabling it tracks the current IDs, but cannot recover IDs deleted before it was enabled. ``--sql`` previews the same change without applying it.

``--strict``
Convert the table to a `SQLite STRICT table <https://www.sqlite.org/stricttables.html>`__. The command fails if the available SQLite version does not support strict tables. If existing rows contain values that are incompatible with their declared column types the transformation fails and the original table is left unchanged.

Expand Down
27 changes: 27 additions & 0 deletions docs/python-api.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -844,6 +844,33 @@ An ``ANY`` column can store integers, floating point values, text, binary data o
.. note::
In the CLI: :ref:`sqlite-utils create-table <cli_create_table>`

.. _python_api_autoincrement:

Preventing reuse of deleted IDs
-------------------------------

An ``INTEGER PRIMARY KEY`` already generates an ID when an inserted record omits that column. Pass ``autoincrement=True`` to prevent SQLite from reusing IDs of deleted rows:

.. code-block:: python

table = db["events"].create({"id": int, "name": str}, pk="id", autoincrement=True)
table.insert({"name": "first event"})

This option also works with ``db.create_table()``, ``db.create_table_sql()``, ``insert()``, ``insert_all()``, ``upsert()``, ``upsert_all()`` and ``lookup()``. To create the table on the first insert:

.. code-block:: python

db["events"].insert({"name": "first event"}, pk="id", autoincrement=True)

You can configure it as a table default using ``db.table("events", pk="id", autoincrement=True)``. The primary key must be a single ``INTEGER`` column; compound, text and hash primary keys cannot use this option. A primary key omitted from the column definitions is created as an integer column. ``upsert()`` still requires primary key values in every record.

The default ``None`` keeps the usual ID allocation when creating a new table. An explicit ``False`` creates a table without ``AUTOINCREMENT``. An explicit option that conflicts with an existing table is rejected by inserts and lookups; use ``table.transform(autoincrement=True)`` or ``table.transform(autoincrement=False)`` to change the mode. ``create(..., transform=True)`` also applies an explicit mode change. Ordinary transformations preserve the existing mode and its sequence.

Enabling the mode on an existing table starts tracking its current IDs. It cannot recover the history of IDs deleted before it was enabled. IDs need not be consecutive. SQLite documents the additional cost of this mode in `its AUTOINCREMENT documentation <https://www.sqlite.org/autoinc.html>`__.

.. note::
In the CLI: :ref:`sqlite-utils create-table <cli_create_table>` and :ref:`sqlite-utils transform <cli_transform_table>`.

.. _python_api_compound_primary_keys:

Compound primary keys
Expand Down
Loading
Loading