Skip to content

JP: MAFF's own Fude Polygon downloads, adding 2025 and 2026 - #345

Merged
ivorbosloper merged 8 commits into
mainfrom
jp-maff
Oct 25, 2026
Merged

ivorbosloper merged 8 commits into
mainfrom
jp-maff

Conversation

@ivorbosloper

@ivorbosloper ivorbosloper commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

The JP converter now reads the Fude Polygons from MAFF's download service instead of Pacific Spatial's converted copy, which stopped at 2024. This adds the 2025 and 2026 editions.

  • MAFF publishes one zip of municipality GeoJSONs per prefecture. The download URLs only work when signed, and a signature lasts 30 minutes, so the converter asks the API to sign the files just before it downloads them; only MAFF's own URLs are signed.
  • MAFF's CDN only answers requests that look like a browser, so the converter sets request_headers (request_headers for downloads; DuckDB converter downloads archives also without a cache vecorel/cli#63). It refuses data-centre addresses outright (403); converting from such a machine needs the zips placed in the cache first.
  • Each prefecture zip becomes one GeoParquet in the cache (extracted.<zip>/) before the usual DuckDB conversion, which reads only parquet. Without a cache, vecorel downloads the zips to a temporary folder (request_headers for downloads; DuckDB converter downloads archives also without a cache vecorel/cli#63), so the converter no longer overrides convert.
  • The GeoJSONs declare JGD2011 per geometry, which GDAL ignores, so the CRS is set explicitly.
  • The attribution follows MAFF's terms (政府標準利用規約 2.0): each edition carries the wording with its year, and the collection carries it without the year.
  • The output columns are the same as before. Where they overlap with the old source, the ids, land types and municipality codes are identical, and the geometries match to floating-point precision.

Updates vecorel-cli to 0.4.1, which brings vecorel/cli#62 and #63.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread fiboa_cli/datasets/jp.py Resolved
Comment thread fiboa_cli/datasets/jp.py Resolved
Comment thread fiboa_cli/datasets/jp.py Resolved

This comment was marked as resolved.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ivorbosloper and others added 2 commits October 24, 2026 00:05
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ivorbosloper

Copy link
Copy Markdown
Collaborator Author

Can only be merged after a new vecorel release....

@m-mohr

m-mohr commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

vecorel-cli v0.4.1 has been released

ivorbosloper and others added 2 commits October 25, 2026 16:21

This comment was marked as resolved.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@m-mohr

m-mohr commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

CI is failing...

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@m-mohr
m-mohr requested review from m-mohr and a balanced review from Copilot October 25, 2026 20:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A cached-input regression remains unresolved, and compatibility across all six source editions still needs validation.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Preserve existing Parquet inputs when caching is enabled

fiboa_cli/​datasets/​jp.py:111

An existing .parquet or .geoparquet supplied through input_files is read directly by DuckDBBaseConverter.convert without a cache, but adding cache now routes the same file here and raises BadZipFile. Pass existing Parquet inputs through unchanged before opening the archive, so enabling caching does not break an otherwise supported input format.

2 resolved since last review

Review effort: Balanced

@ivorbosloper
ivorbosloper merged commit 2c1a135 into main Oct 25, 2026
8 checks passed
@ivorbosloper
ivorbosloper deleted the jp-maff branch October 25, 2026 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants