[ivorbosloper]

The new es converters converts 50 provincial GeoPackages individually and merges the parts. The whole of Spain does not fit in memory in one run. Closes #336.

[m-mohr]

Does it handle collectio metadata correctly? Especially creating columns for e.g. different data colimng from there? License, variant/state/kanton, etc? Ideally it mimics what vec merge does and can be used as a generic replacement as vec merge in geopandas will be an issue anyway in the long run.

[ivorbosloper]

Good point. There's a bug in vec merge due to the "collection" properties (which are not read correctly). A constant property gets into collection, and merge_parquet keeps only what all parts agree on. So admin_province_code is lost in the full ES run and should be expanded again to a column-property. I'll fix merge_parquet in vecorel-cli to hydrate those per part, as vec merge does; this PR then only bumps the pin.

[ivorbosloper]

Fix: vecorel/cli#55. Checked on two real provinces: admin_province_code is a column again.

[ivorbosloper]

I'll wait with making this a Non-Draft PR, it needs a vecorel dependency update due to vecorel/cli#55

[m-mohr]

@ivorbosloper vecorel-cli v0.4.0 has been released. I must say I forgot what the next steps were after finishing vecorel-cli...

[ivorbosloper]

@m-mohr The next step was this PR: it only waited for vecorel/cli#55, which is in 0.4.0, and main already pins that. I merged main in and checked it:

• Álava and Ávila, converted per province and merged with vecorel-cli 0.4.0, validate; admin_province_code is a column again (01 and 05), while admin:country_code, the same everywhere, stays in the collection metadata.
• The test now merges two different provinces and checks that admin_province_code keeps both values, which is the case your question was about.

On a generic replacement for vec merge: merge_parquet already does what vec merge does with collection constants (keeps what all parts agree on, turns the rest into columns), so this mixin leaves that to it rather than repeating it.

[m-mohr]

Sorry, I don't understand what this PR is meant to do and the AI response doesn't help with understanding it. Could a human take over, please?

[ivorbosloper]

Sorry, I don't understand what this PR is meant to do and the AI response doesn't help with understanding it. Could a human take over, please?

@m-mohr Yes. The es-converter memory consumption was too high, all features from 50 source files were put in one gdf. I added a simple "PerFileConverterMixin" class that only overrides the convert method. It transforms all url-sources individually to intermediate parquet files, and then uses DuckDB to merge it.

[m-mohr]

Is this still needed given all the upstream improvements?
We didn't do this for ch either, instead of converting the individual converters and using vec merge...

[ivorbosloper]

Is this still needed given all the upstream improvements? We didn't do this for ch either, instead of converting the individual converters and using vec merge...

We now need between 30/40 Gb, with this PR merged it peeks during sort at around 10 Gb. ch is separate converters because of license/attribution changes. jp does something similar with DuckDB converter, but using this here requires more changes.

[m-mohr]

I've migrated the mixin to vecorel-cli and updated this PR accordingly.