Repository navigation
Support tags and attributes in tree-model schema export/import - #19150
Open
CritasWang wants to merge 6 commits into
Open
CritasWang wants to merge 6 commits into
CritasWang wants to merge 6 commits into
Conversation
mapToString concatenated keys and values without any escaping, so a tag value containing a double quote produced invalid JSON, a backslash was silently reinterpreted by JSON parsers, and control characters such as tab or newline were emitted raw. Escape keys and values per RFC 8259 while keeping non-ASCII characters readable, and drop the duplicate implementation in LogicalViewSchemaSource in favour of the shared one. The output is unchanged for values that need no escaping.
export-schema wrote only Timeseries, Alias, DataType, Encoding and Compression, so tags and attributes were always lost; import-schema had no way to restore them even though createMultiTimeseries and createAlignedTimeseries accept them. Write Tags and Attributes as compact JSON columns (sorted keys, HTML escaping disabled, empty cell when absent) and restore them on import for both the normal and the aligned path. The schema CSV moves to RFC4180 quoting with no escape character, and files carrying the previous five-column header are still read with the legacy settings. A header that looks like the new format is rejected unless all expected columns are present.
writeScheme cleared only paths, dataTypes, encodings, compressors and measurementAlias when a batch was flushed, while pathsWithAlias and its parallel lists kept their entries. A file with more aliased records than the batch size therefore paired aliases (and tags) with the wrong paths and re-sent time series that had already been created.
The schema export/import tools parse the tags and attributes cells with gson, which was only available transitively; the dependency analysis in mvn verify rejects used-but-undeclared dependencies.
Field.getStringValue() renders a NULL cell as the text "null", so comparing the restored alias against null failed for every time series without an alias. Normalise the value in the cell helper, the same way the export tool distinguishes a missing value.
… round trip The test only created DOUBLE/GORILLA/LZ4 series, so a regression that lost a per-series type, encoding or compression, or one of the four tag/attribute presence combinations, could pass unnoticed. Cover six type/encoding/compression combinations over twenty series spread over twenty devices, keep all four presence combinations (both, tags only, attributes only, neither) in the single exported file, assert that one device mixes several of them across its measurements, and check the format of every row after the import as well.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tree-model
export-schemadropped tags and attributes entirely, andimport-schemahad no way to restore them even thoughcreateMultiTimeseries/createAlignedTimeseriesaccept both. This PR makes a full export → import cycle preserve them exactly, including every special character.It is two logically separate pieces, kept as three commits so they can be reviewed (or split) independently:
[Server] Render tags and attributes as valid JSON in SHOW TIMESERIES[Tools] Export and import tags and attributes in tree-model schema files[Tools] Clear the aliased lists at every import batch boundary1. Why the server change is required
The client has no API that returns tags as a
Map—Session/SessionPoolonly accepttagsListon write. Reading them is only possible through theTags/Attributescolumns ofSHOW TIMESERIES, andTimeSeriesSchemaSource.mapToStringconcatenated keys and values without escaping, so that column was not parseable data:a"b{"k":"a"b"}a\b{"k":"a\b"}a<backspace>— silent corruptiona<TAB>b{"k":"a<TAB>b"}The fix escapes per RFC 8259 and keeps non-ASCII characters readable; it also removes the duplicate implementation in
LogicalViewSchemaSource. Output is byte-identical for values that need no escaping, so ordinary schemas are unaffected.2. File format
Export now writes seven columns:
Tags/Attributesare compact JSON objects with sorted keys and HTML escaping disabled; an absent set is an empty cell.", no escape character, LF). JSON cells are therefore quoted with their inner quotes doubled, which is what makes them round-trip.3. Compatibility
quote='',escape=''`), so existing exports keep importing unchanged.TagsorAttributespresent), and such a header is rejected unless all seven expected columns are there.",\or control characters are now rendered escaped inSHOW TIMESERIES(previously the output was not valid JSON at all). Values without such characters are unchanged.","would need adjusting.4. Testing
Unit tests (real classes, no mocks except the session pool):
TimeSeriesSchemaSourceMapToStringTest(18 tests): every escaping case, round-trip through a standard JSON parser, and byte-identity with the old algorithm for values that need no escaping.SchemaTagsFidelityTest(14) andSchemaTagsImportTest(6): JSON cell encoding (sorted keys, no HTML escaping, empty cell), RFC4180 bytes, CSV round-trip for"\,tab newline CJK backtick, legacy detection, and the create-call arguments for both the normal and the aligned path. The batch test was validated by reverting the fix and observing it fail (expected:<4> but was:<2>).Full local runs:
mvn test -pl iotdb-client/cli→ 100/100; datanode test for the escaping change → 18/18; both locales (-P with-zh-locale) compile.End-to-end on a real standalone cluster (built from this branch): created 18 time series covering
",\, a literal\n, tab, newline, CJK+emoji, literalnull, empty string, double spaces, backtick, a key with a space, multiple tags, attributes-only, neither, and an alias; exported, dropped the database, imported, and compared the server's rendering field by field — 18/18 identical. Editing one tag in the exported file with a spreadsheet tool and re-importing also worked, leaving the other rows untouched.SchemaTagsRoundTripITwas added for CI. It has not been executed locally (iotdb-core/ainode/dist/ainode/, which the integration-test assembly references, is absent in this environment, and the test node template needs a full reactor build). It compiles, is spotless-clean, and passes the repository's IT-selection check, but its first real run will be in CI.5. Not addressed here
Logical views, schema templates and device alignment are still not exported (they need different creation statements or device-level information); the import batch fix above is included because misaligned tags would otherwise corrupt exactly this feature's data.