fix(index): preserve row addresses in JSON index training

[lance-gatekeeper]

Root cause

The JSON index training pipeline projected and rebuilt every transformed batch with a hard-coded _rowid column. Inner trainers such as ZoneMap and FM-index request _rowaddr, so JSON extraction discarded the requested addresses before delegating to those trainers.

Fix

Preserve every scanner-provided row-location column through JSON extraction and type conversion. The existing JSON trainer test helper now follows the target training criteria, with regression cases verifying that ZoneMap and FM-index retain addresses from multiple fragments.

Validation

• cargo fmt --all -- --check
• cargo test -p lance-index scalar::json::tests (10 passed)
• cargo clippy --all --tests --benches -- -D warnings

Fixes #7859

[codecov]

Codecov Report

❌ Patch coverage is 96.47059% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-index/src/scalar/json.rs	96.47%	1 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

[lance-gatekeeper]

Gate recommendation: request changes. Preserving the scanner-provided row-location columns is the right mechanism, but the newly enabled JSON→ZoneMap path still weakens the delegated trainer’s address-order contract to arbitrary ordering. Preserve TrainingOrdering::Addresses in the wrapped request (only Values needs to become None so JSON can sort the extracted value) so every caller satisfying the advertised contract can train safely.

[lance-gatekeeper]

JSON→ZoneMap still advertises TrainingOrdering::None, so a caller may legally supply arbitrary row-address order even though ZoneMap requires Addresses; this head then fails training for [1, 0] with zone row offsets are out of order. Preserve the inner Addresses ordering (only map Values to None, since JSON sorts extracted values later) and cover unordered input.

Reproducer

I ran this disposable test on the reviewed head:

#[tokio::test]
async fn test_json_zonemap_none_order_rejects_legal_out_of_order_input() {
    use crate::progress::noop_progress;
    use arrow_array::{ArrayRef, LargeBinaryArray, UInt64Array};
    use futures::stream;

    let registry = IndexPluginRegistry::with_default_plugins();
    let plugin = registry.get_plugin_by_name("json").unwrap();
    let trainer = plugin.basic_trainer().unwrap();
    let request = trainer
        .new_training_request(
            r#"{\"target_index_type\":\"zonemap\",\"path\":\"value\"}"#,
            &Field::new(VALUE_COLUMN_NAME, DataType::LargeBinary, true),
        )
        .unwrap();
    assert_eq!(request.criteria().ordering, TrainingOrdering::None);
    assert!(request.criteria().needs_row_addrs);

    let jsonb = [r#"{\"value\":\"first\"}"#, r#"{\"value\":\"second\"}"#]
        .into_iter()
        .map(|doc| doc.parse::<jsonb::OwnedJsonb>().unwrap().to_vec())
        .collect::<Vec<_>>();
    let schema = Arc::new(Schema::new(vec![
        Field::new(VALUE_COLUMN_NAME, DataType::LargeBinary, true),
        Field::new(ROW_ADDR, DataType::UInt64, false),
    ]));
    let batch = RecordBatch::try_new(
        schema.clone(),
        vec![
            Arc::new(LargeBinaryArray::from(
                jsonb.iter().map(|value| Some(value.as_slice())).collect::<Vec<_>>(),
            )) as ArrayRef,
            Arc::new(UInt64Array::from(vec![1, 0])) as ArrayRef,
        ],
    )
    .unwrap();
    let data = Box::pin(RecordBatchStreamAdapter::new(
        schema,
        stream::iter(vec![Ok(batch)]),
    )) as SendableRecordBatchStream;
    let (store, _tmpdir) = local_json_index_store();

    let error = match trainer
        .train_index(data, store.as_ref(), request, None, noop_progress())
        .await
    {
        Ok(_) => panic!("zonemap unexpectedly accepted descending row addresses"),
        Err(error) => error,
    };
    assert!(error.to_string().contains("zone row offsets are out of order"));
}

Command: cargo test -p lance-index test_json_zonemap_none_order_rejects_legal_out_of_order_input -- --nocapture

The outer contract permits this input, so training should succeed; it instead returned zone row offsets are out of order.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The follow-up preserves the delegated trainer’s address-order requirement while still deferring Values ordering until after JSON extraction. Together with row-location passthrough, the wrapper now satisfies the inner training contract for ZoneMap, FM-index, and existing value-sorted targets.