Skip to content

feat: Add SQLite compiletime native bindings#450

Merged
Frotty merged 3 commits into
wurstscript:masterfrom
Donach:feature/sqlite-externs
Jul 24, 2026
Merged

feat: Add SQLite compiletime native bindings#450
Frotty merged 3 commits into
wurstscript:masterfrom
Donach:feature/sqlite-externs

Conversation

@Donach

@Donach Donach commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Adds native @extern declarations for compiletime SQLite database operations in wurst/file/SQLite.wurst.

High-level API

  • SqlResult: Dynamic column storage using ArrayList<string>.
  • SqliteDb: OOP connection wrapper supporting raw and parameterised queries (execPrepared, selectPrepared, selectFirstPrepared, existsPrepared).
  • SQL: Configurable singleton wrapper.

Acceptance Criteria

Exposed Native Functions:

  • sqlite_open(string path) returns int
  • sqlite_prepare(int conn, string q) returns int
  • sqlite_bind_int(int stmt, int idx, int val)
  • sqlite_bind_real(int stmt, int idx, real val)
  • sqlite_bind_string(int stmt, int idx, string val)
  • sqlite_step(int stmt) returns boolean
  • sqlite_column_count(int stmt) returns int
  • sqlite_column_int(int stmt, int idx) returns int
  • sqlite_column_real(int stmt, int idx) returns real
  • sqlite_column_string(int stmt, int idx) returns string
  • sqlite_reset(int stmt)
  • sqlite_finalize(int stmt)
  • sqlite_close(int conn)
  • sqlite_exec(int conn, string q)

Comment thread wurst/file/SQLite.wurst Outdated

/** A row returned from a SELECT query. Access columns by index via col(). */
public class SqlResult
string array[2000] cols

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Big array members are not a good idea in wurst, I think just use ArrayList here and for select and remove LinkedList import.

@Frotty

Frotty commented Jul 22, 2026

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7db7067e44

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread wurst/file/SQLite.wurst Outdated

/** A row returned from a SELECT query. Access columns by index via col(). */
public class SqlResult
string array[2000] cols

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use dynamic storage for row columns

Because Wurst implements sized class-member arrays by generating one backing array per element, this declaration emits 2000 backing arrays/accessor branches for every project that uses SqlResult, even when a query returns only one or two columns. That is a large code-size/compile-time cost in a stdlib wrapper and makes every col() access go through the generated dispatcher; store the actual column values in a dynamic list/ArrayList instead of a 2000-slot member array.

Useful? React with 👍 / 👎.

@Donach
Donach force-pushed the feature/sqlite-externs branch from 7db7067 to 81cd773 Compare July 22, 2026 20:33
@Donach

Donach commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Updated this PR with the requested fixes:

  1. Parameterized Queries: Added execPrepared, selectPrepared, selectFirstPrepared, and existsPrepared methods on SqliteDb and SQL using sqlite_prepare and sqlite_bind_string to prevent SQL injection.
  2. Column Count Timing: Fixed sqlite_column_count timing by calling it after the first sqlite_step call when ResultSet metadata is present.
  3. Dynamic Column Storage: Updated SqlResult to store row values in a dynamic ArrayList<string> rather than a fixed 2000-element member array to avoid Wurst's array-backing code generation overhead.
  4. Test Annotations: Unified all test annotations to lower-case @test.
  5. Acceptance Criteria: Noted dependency on upstream WurstScript PR #1180.

Reborn Reviewer added 2 commits July 23, 2026 00:12
Adds @extern declarations for compiletime SQLite JDBC bindings exposed by
the WurstScript compiler (requires wurstscript/WurstScript PR #1180).

Native functions: sqlite_open, sqlite_prepare, sqlite_bind_*, sqlite_step,
sqlite_column_*, sqlite_reset, sqlite_finalize, sqlite_close, sqlite_exec.

High-level API:
- SqlResult: row accessor with col, colInt, colReal, colBool helpers
- SqliteDb: OOP connection wrapper with select, selectFirst, exists, count
- SQL: configurable singleton for single-database workflows

Parameterised query support:
- execPrepared, selectPrepared, selectFirstPrepared, existsPrepared
- Prevents SQL injection by using sqlite_prepare + sqlite_bind_string
- Includes injection prevention test

Fixes vs original draft:
- Moved sqlite_column_count after first sqlite_step (JDBC ResultSet
  metadata requires an executed statement)
- Added parameterised query variants to prevent SQL injection
- Added comprehensive doc comments about error behaviour and upstream dep
- Unified test annotations to @test (lowercase)
- Removed unused ErrorHandling import
- Fixed array size to use literal (WurstScript requirement)
Because Wurst implements sized class-member arrays by generating one
backing array per element, the previous 2000-slot member array
in SqlResult emitted massive code and compiled slowly.
This refactors SqlResult to use a dynamic ArrayList<string> to store
only the actual returned columns.

Additionally updates all LinkedList usages to ArrayList to match.
@Donach
Donach force-pushed the feature/sqlite-externs branch from 81cd773 to 59480f1 Compare July 22, 2026 22:12
@Frotty

Frotty commented Jul 23, 2026

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 59480f1d1e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread wurst/file/SQLite.wurst Outdated
Comment on lines +165 to +166
for p in params
sqlite_bind_string(stmt, idx, p)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Replace ArrayList foreach with index loops

When prepared queries are used, this iterates an ArrayList<string> with for p in params, but ArrayList in this repo does not expose an iterator and is documented/implemented for index-based traversal only. That makes the new SQLite package fail to typecheck as soon as this method is compiled; the same pattern is repeated in the new tests over rows, so these should be changed to for i = 0 to params.size() - 1 / params.get(i) style loops.

Useful? React with 👍 / 👎.

The API used `for x in ArrayList` in 6 places, but this stdlib's ArrayList
has no iterator by design (index loops only), so the PR did not compile.
Replaced those with index loops.

Also completed the binding surface against WurstScript PR #1180:
- expose sqlite_column_is_null and sqlite_clear_bindings natives (were missing)
- SqlResult now records per-column NULL flags and exposes colIsNull(), the
  only reliable way to tell SQL NULL apart from an empty string (col()
  returns "" for both)

Expanded SQLiteTests with a native-surface section proving every native:
typed bind/read-back, column_count, column_is_null, step-past-last-row,
reset (rewind + preserve bindings), clear_bindings, reset+rebind reuse,
multi-statement exec, trigger blocks, and finalize+reprepare.

All 354 stdlib tests pass (50 SQLite) with the PR #1180 compiler.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Donach

Donach commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Evaluation + update (compiled & tested against WurstScript #1180)

I built the compiler from #1180 and ran the stdlib test suite against it. Findings and the changes I pushed:

🔴 Critical: the PR did not compile

SqliteDb.bindParams and 5 tests used for x in ArrayList, but this stdlib's ArrayList has no iterator by design ("index loops keep hot paths allocation-free"). The compiler rejected all 6 sites:

Error in File SQLite.wurst:165: For loop target ArrayList<string> doesn't provide a iterator() function
Error in File SQLiteTests.wurst:72: For loop target ArrayList<SqlResult> doesn't provide a iterator() function
... (11 errors total)

Fixed by switching to index loops (for i = 0 to list.size() - 1).

🟡 Incomplete native surface vs #1180

#1180 ships 16 natives; this PR declared 14, omitting the two that make the cursor API complete:

  • sqlite_column_is_null — the only way to tell SQL NULL from "" (sqlite_column_string returns "" for both)
  • sqlite_clear_bindings — unbind a reused prepared statement

Both are now declared, and SqlResult records per-column NULL flags and exposes colIsNull(index). Previously test_..._nullValueReturnsEmptyString baked the NULL-vs-empty ambiguity in as intended behaviour; it's now disambiguated.

✅ Tests: full native-surface coverage

Added a native-surface section proving every native during compiletime: typed bind/read-back (bind_int/bind_real/column_int/column_real), column_count, column_is_null, step-past-last-row, reset (rewind and preserved bindings), clear_bindings, reset+rebind reuse, multi-statement exec, CREATE TRIGGER … BEGIN … END blocks, and finalize+reprepare.

Proof

Tests succeeded: 354/354
>> All tests have passed successfully!
compilation finished (errors: 0, warnings: 1)

50 of those are SQLite tests, all green. (Before the fix: 11 compile errors, 0 tests runnable.)

Note: the native list in the PR description is now slightly out of date — it should add sqlite_column_is_null and sqlite_clear_bindings.

@Frotty

Frotty commented Jul 24, 2026

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0c9bca1b30

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread wurst/file/SQLite.wurst
Comment thread wurst/file/SQLite.wurst
Comment thread wurst/file/SQLiteTests.wurst
@Frotty
Frotty merged commit 3af4815 into wurstscript:master Jul 24, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants