Optimize paginated payment listing queries#240
Conversation
Use a composite namespace/time/key index because paginated store queries first constrain primary_namespace and secondary_namespace, then page and order by creation_time DESC and key ASC. The prior creation_time-only index did not match that access pattern and could scan rows from unrelated namespaces or require extra sorting. Existing databases may retain idx_creation_time; this change only stops creating it for new databases and creates idx_paginated_kv_namespace_time_key instead.
Add a required list_values method to the paginated store abstraction and implement it in SqliteStore with a single query that returns each page of keys and serialized values. Keep the existing cursor predicate, ordering, and page-token behavior. Switch ListPayments and ListForwardedPayments to decode the returned values directly so each page avoids the per-key read loop.
|
👋 I see @valentinewallace was un-assigned. |
tnull
left a comment
tnull
left a comment
There was a problem hiding this comment.
We're about to drop the SQLite / local forwarded payment store from Server as we're close to finally add all parts to LDK Node. So I don't think it makes sense to make changes to it right now?
|
Opened this PR because I still had the code lying around from an older analysis, but I wasn’t fully up to date with what had happened in The main thing to distill from it is probably this question on One meta note: the split between |
Yes, well, that split is exactly what we're trying to reduce currently, i.e., move any remaining custom 'logic' into Node so that Server is really just responsible for the API, dealing with the daemon etc. |
|
Once And I still think keeping this split across two repos introduces unnecessary visibility and change friction, but you all know my stance on that. |
3 participants
This replaces the SQLite pagination index with one that matches the actual paginated store access pattern: namespace filters first, followed by
creation_time DESC, key ASCfor cursor ordering. The previouscreation_time-only index did not fit these queries well and could scan unrelated namespaces or require extra sorting.It also adds a required
list_valuespaginated store method and implements it inSqliteStorewith a single query returning keys and serialized values.ListPaymentsandListForwardedPaymentsnow decode those returned values directly, avoiding the previous page-level N+1 pattern where listing 100 items could require 101 SQLite queries.