Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions Changes.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,25 @@
## next release

- `MMDB_open()` now returns `MMDB_INVALID_METADATA_ERROR` if the top-level
metadata map or the `description` map repeats a key. Before, top-level
metadata lookups used only the first value of a repeated key, and the
`description` list kept every pair, while other readers may keep the last
value. Keys are compared up to the first NUL byte, so keys that differ only
after a NUL count as repeated. A top-level metadata map or `description` map
with more than 1,024 keys also causes `MMDB_INVALID_METADATA_ERROR`. Databases
with repeated metadata keys no longer open.
- `MMDB_open()` now reads every top-level metadata key, not only the required
keys, and skips each value without following pointers in it. A malformed key
or value, or a map that claims more entries than it has, now causes
`MMDB_INVALID_DATA_ERROR`. A value nested deeper than the depth limit, or keys
whose total size exceeds the payload limit, causes
`MMDB_INVALID_METADATA_ERROR`. Some of these databases opened before. Metadata
whose root is not a map now causes `MMDB_INVALID_METADATA_ERROR`.
- `MMDB_get_value()` and its variants, and `MMDB_open()`, now return
`MMDB_INVALID_DATA_ERROR` when they skip a nested map whose key is a map or an
array. Before, they skipped only the key's header and read the data after it
out of step. A lookup could return the wrong value.

## 1.14.1 - 2026-09-17

- Fixed a `-Wsign-compare` warning in `MMDB_open()` that broke builds with
Expand Down
17 changes: 15 additions & 2 deletions doc/libmaxminddb.md
Original file line number Diff line number Diff line change
Expand Up @@ -381,8 +381,9 @@ status codes are:
its major version is not 2. This library can only handle major version 2.
- `MMDB_OUT_OF_MEMORY_ERROR` - a memory allocation call (`malloc`, etc.) failed.
- `MMDB_INVALID_DATA_ERROR` - an entry in the data section contains invalid
data. For example, a `uint16` field is claiming to be more than 2 bytes long.
The database is probably damaged or was generated incorrectly.
data. For example, a `uint16` field is claiming to be more than 2 bytes long,
or a map has a key that is a map or an array. The database is probably damaged
or was generated incorrectly.
- `MMDB_INVALID_LOOKUP_PATH_ERROR` - The lookup path passed to `MMDB_get_value`,
`MMDB_vget_value`, or `MMDB_aget_value` contains an array offset that is
larger than LONG_MAX or smaller than LONG_MIN.
Expand Down Expand Up @@ -459,6 +460,15 @@ If metadata processing exceeds one of the decoder resource limits described
under `MMDB_get_entry_data_list()`, this function returns
`MMDB_INVALID_METADATA_ERROR`.

If the metadata root is not a map, or the top-level metadata map or the
`description` map repeats a key or has more than 1,024 keys, this function
returns `MMDB_INVALID_METADATA_ERROR`. It compares keys up to the first NUL
byte, so keys that differ only after a NUL count as repeated.

If a top-level metadata key or value is malformed, this function returns
`MMDB_INVALID_DATA_ERROR`. It does not follow pointers in values that it does
not use, so it does not detect a bad pointer target there.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The manual does not say that MMDB_get_value and MMDB_aget_value can now return MMDB_INVALID_DATA_ERROR when a nested map that they skip has a map or array key. Only Changes.md says this.

The MMDB_get_value section and the error code list (around line 383, outside this diff) are the places to add it. Users who read those sections do not learn that lookups which returned data before (possibly the wrong value) now fail.

🤖 Comment by Claude Opus 5.5.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From Claude (Claude Opus 5.5), on Greg's behalf: Valid. The Data Lookup Functions section and the MMDB_INVALID_DATA_ERROR entry in the status code list now describe the map or array key error. Fixed in 522dd44.


## `MMDB_close()`

```c
Expand Down Expand Up @@ -631,6 +641,9 @@ calls.
For each of the three functions, the return value is a status code as defined
above.

Map keys must be strings. If a lookup skips a nested map whose key is a map or
an array, these functions return `MMDB_INVALID_DATA_ERROR`.

## `MMDB_get_entry_data_list()`

```c
Expand Down
177 changes: 175 additions & 2 deletions src/maxminddb.c
Original file line number Diff line number Diff line change
Expand Up @@ -178,9 +178,17 @@ typedef struct decode_state_s {
uint64_t bytes;
} decode_state_s;

typedef struct metadata_key_s {
const char *name;
uint32_t size;
} metadata_key_s;

#define METADATA_MARKER "\xab\xcd\xefMaxMind.com"
/* This is 128kb */
#define METADATA_BLOCK_MAX_SIZE 131072
/* Real metadata has about 10 top-level keys and a few description languages.
* The cap bounds the key sort, because C does not bound the cost of qsort. */
#define METADATA_MAP_MAX_KEYS 1024

// 64 leads us to allocating 4 KiB on a 64bit system.
#define MMDB_POOL_INIT_SIZE 64
Expand All @@ -191,6 +199,10 @@ static const uint8_t *find_metadata(const uint8_t *file_content,
uint32_t *metadata_size);
static int read_metadata(MMDB_s *mmdb);
static MMDB_s make_fake_metadata_db(const MMDB_s *const mmdb);
static int check_metadata_keys_unique(const MMDB_s *const metadata_db);
static int next_map_key(const MMDB_s *const mmdb,
uint32_t *offset,
MMDB_entry_data_s *key);
static int
value_for_key_as_uint16(MMDB_entry_s *start, char *key, uint16_t *value);
static int
Expand All @@ -205,6 +217,9 @@ static int populate_languages_metadata(MMDB_s *mmdb,
static int populate_description_metadata(MMDB_s *mmdb,
MMDB_s *metadata_db,
MMDB_entry_s *metadata_start);
static metadata_key_s metadata_key(const char *name, uint32_t size);
static int check_keys_unique(metadata_key_s *keys, uint32_t count);
static int compare_metadata_keys(const void *a, const void *b);
static int resolve_any_address(const char *ipstr, struct addrinfo **addresses);
static int find_address_in_search_tree(const MMDB_s *const mmdb,
uint8_t const *address,
Expand Down Expand Up @@ -589,10 +604,28 @@ static int read_metadata(MMDB_s *mmdb) {
want to use the same functions we use for the data section to get
metadata values. */
MMDB_s metadata_db = make_fake_metadata_db(mmdb);

MMDB_entry_s metadata_start = {.mmdb = &metadata_db, .offset = 0};

int status = value_for_key_as_uint32(
// Check the major version before the strict walk below. A newer format
// can have metadata that this version cannot decode, and the caller must
// still get MMDB_UNKNOWN_DATABASE_FORMAT_ERROR for it. If this lookup
// fails or finds 0, the full read below reports the error.
uint16_t major_version = 0;
int status = value_for_key_as_uint16(
&metadata_start, "binary_format_major_version", &major_version);
if (MMDB_SUCCESS == status && 0 != major_version && 2 != major_version) {
DEBUG_MSGF("unsupported binary_format_major_version %i", major_version);
return MMDB_UNKNOWN_DATABASE_FORMAT_ERROR;
}

// Key lookups return the first match, but other readers can keep the
// last one. A repeated key could give each reader a different value.
status = check_metadata_keys_unique(&metadata_db);
if (MMDB_SUCCESS != status) {
return status;
}

status = value_for_key_as_uint32(
&metadata_start, "node_count", &mmdb->metadata.node_count);
if (MMDB_SUCCESS != status) {
return status;
Expand Down Expand Up @@ -699,6 +732,85 @@ static MMDB_s make_fake_metadata_db(const MMDB_s *const mmdb) {
return fake_metadata_db;
}

static int check_metadata_keys_unique(const MMDB_s *const metadata_db) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design: two separate parsers read the same map.

This function checks for unique keys in a separate walk. The 9 required fields are still read by first-match scans (value_for_key_as_*). The two parsers must stay in step. Commit de11e2a had to fix this kind of problem: a nested array key made the checker read something different from what the lookups read.

A single walk that dispatches on the key name, fills the metadata fields, and fails on a second occurrence of a key would remove that class of problem. It would also replace about 10 scans of the metadata with one.

🤖 Comment by Claude Opus 5.5.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From Claude (Claude Opus 5.5), on Greg's behalf: Agreed that one walk is the better design, but it rewrites read_metadata, so I filed it as a follow-up: https://linear.app/maxmind/issue/STF-2363/read-libmaxminddb-metadata-in-one-walk. The two parsers now use the same skip_map_or_array, so the de11e2a class of problem is fixed in the shared code.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did do performance testing on an open and the increase from this seemed minimal. I think it was a few percent.

MMDB_entry_data_s map;
CHECKED_DECODE_ONE_FOLLOW(metadata_db, 0, &map);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out-of-bounds read when the metadata section is shorter than 3 bytes.

This bug was already in the code before this PR. But check_metadata_keys_unique is now the first code that decodes the metadata, so it is now the first code to hit the bug. MMDB_open checks for a data section shorter than 3 bytes, but it does not do the same check for the metadata section.

Example: a 4096-byte file that ends in the metadata marker plus the two bytes \x9f\x03 gives a 2-byte metadata section. In decode_one, case 31 calculates data_section_size - 3. That value wraps to UINT32_MAX, so the bounds check passes, and get_uint24 reads 2 bytes past the end of the mapping. I reproduced this with ASan through mmdblookup --file. Without ASan, a page-aligned file can cause SIGBUS or SIGSEGV.

Suggested fix: reject metadata_section_size < 3 in MMDB_open.

🤖 Comment by Claude Opus 5.5.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From Claude (Claude Opus 5.5), on Greg's behalf: Valid, and the bug is already in main: before this PR, the first required-key lookup decoded the same bytes. #514 fixes it. Its first commit guards the size-code 30 and 31 subtractions in decode_one, and its tests cover a 2-byte metadata section, which now returns MMDB_INVALID_DATA_ERROR. So I did not change this PR.

if (MMDB_DATA_TYPE_MAP != map.type) {
DEBUG_MSG("metadata value is not a map");
return MMDB_INVALID_METADATA_ERROR;
}
if (0 == map.data_size) {
return MMDB_SUCCESS;
}
// Each entry takes at least a key byte and a value byte. Reject a larger
// count before it sizes the allocation, with the error the walk would
// return when it reaches the end of the section.
if (map.offset_to_next > metadata_db->data_section_size ||
map.data_size >
(metadata_db->data_section_size - map.offset_to_next) / 2) {
DEBUG_MSG("metadata map size exceeds the metadata section");
return MMDB_INVALID_DATA_ERROR;
}
if (map.data_size > METADATA_MAP_MAX_KEYS) {
DEBUG_MSG("metadata map has too many keys");
return MMDB_INVALID_METADATA_ERROR;
}

metadata_key_s *keys = calloc(map.data_size, sizeof(metadata_key_s));
if (NULL == keys) {
return MMDB_OUT_OF_MEMORY_ERROR;
}

int status = MMDB_SUCCESS;
uint32_t offset = map.offset_to_next;
uint64_t key_bytes = 0;
for (uint32_t i = 0; i < map.data_size; i++) {
MMDB_entry_data_s key;
status = next_map_key(metadata_db, &offset, &key);
if (MMDB_SUCCESS != status) {
goto cleanup;
}
// Pointer keys can reuse one long string many times. Charge each key
// against the payload budget to bound the scan and the sort below.
key_bytes += key.data_size;
if (key_bytes > (uint64_t)(MAXIMUM_DATA_STRUCTURE_BYTES)) {
DEBUG_MSG("metadata keys exceed the payload budget");
status = MMDB_DECODER_LIMIT_ERROR;
goto cleanup;
}
keys[i] = metadata_key(key.utf8_string, key.data_size);
}

status = check_keys_unique(keys, map.data_size);

cleanup:
free(keys);
return status;
}

/* Decodes the map key at *offset, skips its value, and sets *offset to the
* next key. */
static int next_map_key(const MMDB_s *const mmdb,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reuse: next_map_key repeats the sequence in lookup_path_in_map (around lines 1542-1566): decode the key, check that it is a string, decode the value, call skip_map_or_array, and move the offset. The map-size bound check at lines 735-737 is also a copy of the check in get_entry_data_list (around lines 2038-2039).

The same logic now exists in two places. A later fix to one key-skipping path (like the fix in de11e2a) can miss the other. lookup_path_in_map could call next_map_key, and the bound check could move into one helper function.

🤖 Comment by Claude Opus 5.5.

@oschwald oschwald Nov 6, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From Claude (Claude Opus 5.5), on Greg's behalf: Won't fix. lookup_path_in_map is on the lookup hot path, and its loop must stop at the matching key and decode that value, while next_map_key always skips the value. Sharing the code would change the hot loop. For de11e2a, small changes on the same lookup path in skip_map_or_array cost 0.5% to 1.85% on a lookup benchmark, so I would not change it without a strong reason. The two map-size bounds return different errors at different points, so one helper would not save much.

uint32_t *offset,
MMDB_entry_data_s *key) {
CHECKED_DECODE_ONE_FOLLOW(mmdb, *offset, key);
if (MMDB_DATA_TYPE_UTF8_STRING != key->type) {
return MMDB_INVALID_DATA_ERROR;
}

// A top-level value is at depth 1, as in
// MMDB_get_metadata_as_entry_data_list, so both apply the same limit.
MMDB_entry_data_s value;
CHECKED_DECODE_ONE(mmdb, key->offset_to_next, &value);
int status = skip_map_or_array(mmdb, &value, 1);
if (MMDB_SUCCESS != status) {
return status;
}
*offset = value.offset_to_next;

return MMDB_SUCCESS;
}

static int
value_for_key_as_uint16(MMDB_entry_s *start, char *key, uint16_t *value) {
MMDB_entry_data_s entry_data;
Expand Down Expand Up @@ -864,6 +976,7 @@ static int populate_description_metadata(MMDB_s *mmdb,
}

MMDB_entry_data_list_s *first_member = member;
metadata_key_s *keys = NULL;

uint32_t map_size = member->entry_data.data_size;
mmdb->metadata.description.count = 0;
Expand All @@ -874,6 +987,17 @@ static int populate_description_metadata(MMDB_s *mmdb,
MAYBE_CHECK_SIZE_OVERFLOW(map_size,
SIZE_MAX / sizeof(MMDB_description_s *),
MMDB_INVALID_METADATA_ERROR);
if (map_size > METADATA_MAP_MAX_KEYS) {
DEBUG_MSG("description map has too many keys");
status = MMDB_INVALID_METADATA_ERROR;
goto cleanup;
}

keys = calloc(map_size, sizeof(metadata_key_s));
if (NULL == keys) {
status = MMDB_OUT_OF_MEMORY_ERROR;
goto cleanup;
}

mmdb->metadata.description.descriptions =
calloc(map_size, sizeof(MMDB_description_s *));
Expand Down Expand Up @@ -901,6 +1025,8 @@ static int populate_description_metadata(MMDB_s *mmdb,
goto cleanup;
}

keys[i] = metadata_key(member->entry_data.utf8_string,
member->entry_data.data_size);
mmdb->metadata.description.descriptions[i]->language = mmdb_strndup(
member->entry_data.utf8_string, member->entry_data.data_size);

Expand All @@ -925,12 +1051,51 @@ static int populate_description_metadata(MMDB_s *mmdb,
}
}

status = check_keys_unique(keys, map_size);

cleanup:
free(keys);
MMDB_free_entry_data_list(first_member);

return status;
}

/* Returns the key to compare for a metadata map key. Keys are compared up to
* the first NUL, because some callers, such as the PHP extension, cut keys
* there. */
static metadata_key_s metadata_key(const char *name, uint32_t size) {
metadata_key_s key = {.name = name, .size = size};
const char *nul = memchr(name, '\0', size);
if (NULL != nul) {
key.size = (uint32_t)(nul - name);
}
return key;
}

/* Sorts keys and returns MMDB_INVALID_METADATA_ERROR if two are equal. */
static int check_keys_unique(metadata_key_s *keys, uint32_t count) {
qsort(keys, count, sizeof(metadata_key_s), compare_metadata_keys);
for (uint32_t i = 1; i < count; i++) {
if (0 == compare_metadata_keys(&keys[i - 1], &keys[i])) {
DEBUG_MSG("duplicate key in metadata");
return MMDB_INVALID_METADATA_ERROR;
}
}
return MMDB_SUCCESS;
}

static int compare_metadata_keys(const void *a, const void *b) {
const metadata_key_s *key_a = a;
const metadata_key_s *key_b = b;
if (key_a->size < key_b->size) {
return -1;
}
if (key_a->size > key_b->size) {
return 1;
}
return memcmp(key_a->name, key_b->name, key_a->size);
}

MMDB_lookup_result_s MMDB_lookup_string(const MMDB_s *const mmdb,
const char *const ipstr,
int *const gai_error,
Expand Down Expand Up @@ -1432,6 +1597,14 @@ static int skip_map_or_array(const MMDB_s *const mmdb,
while (size-- > 0) {
CHECKED_DECODE_ONE(
mmdb, entry_data->offset_to_next, entry_data); // key
// Keys must be strings. This loop skips only a key's header, so
// the contents of a map or array key would be read as the keys and
// values that follow it.
if (MMDB_DATA_TYPE_MAP == entry_data->type ||

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check also rejects keys that are an empty map (0xE0) or an empty array (0x00 0x04). When the code skips the header of an empty map or empty array, it skips the whole key, so these keys never caused the misaligned read. Only keys that have contents caused it.

Before this change, MMDB_get_value on a path past such a nested map worked. Now it returns MMDB_INVALID_DATA_ERROR, so lookups on some lenient third-party databases start to fail for no reason. Should the check apply only to a map or array key that has contents?

🤖 Comment by Claude Opus 5.5.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Weird feedback. Seems like the change is an improvement.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From Claude (Claude Opus 5.5), on Greg's behalf: I agree with you: the check stays. The spec says map keys are always UTF-8 strings, so a map or array key is invalid even when it is empty, and rejecting it in one place is simpler than a special case for empty containers.

MMDB_DATA_TYPE_ARRAY == entry_data->type) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MMDB_get_value and MMDB_get_entry_data_list now give different results for the same data.

MMDB_get_value and MMDB_aget_value reject a map or array key through skip_map_or_array. But MMDB_get_entry_data_list still accepts it, because get_entry_data_list decodes the whole key (around line 2051).

Example: a record has a nested map with an array key. MMDB_get_entry_data_list(&entry) succeeds and returns the full structure. MMDB_get_value(&entry, ..., "sibling", NULL) on the same entry returns MMDB_INVALID_DATA_ERROR. Changes.md describes only the second behavior.

🤖 Comment by Claude Opus 5.5.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From Claude (Claude Opus 5.5), on Greg's behalf: Valid. #516 makes get_entry_data_list reject a non-string map key with MMDB_INVALID_DATA_ERROR, so the two functions agree once both PRs merge. I did not duplicate that change here.

DEBUG_MSG("map key is a map or an array");
return MMDB_INVALID_DATA_ERROR;
}
CHECKED_DECODE_ONE(
mmdb, entry_data->offset_to_next, entry_data); // value
int status = skip_map_or_array(mmdb, entry_data, depth + 1);
Expand Down
2 changes: 2 additions & 0 deletions t/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ set(TEST_TARGET_NAMES
overflow_bounds_t
pointer_dos_t
read_node_t
repeated_metadata_key_t
version_t
)

Expand All @@ -33,6 +34,7 @@ if(UNIX) # or if (NOT WIN32)
bad_data_size_t
bad_epoch_t
bad_indent_t
duplicate_metadata_key_t
empty_container_metadata_t
invalid_sockaddr_t
max_depth_t
Expand Down
5 changes: 3 additions & 2 deletions t/Makefile.am
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,13 @@ check_PROGRAMS = \
bad_pointers_t bad_databases_t bad_data_size_t bad_epoch_t bad_indent_t \
bad_search_tree_t \
basic_lookup_t data_entry_list_t \
data-pool-t data_types_t double_close_t dump_t empty_container_metadata_t \
data-pool-t data_types_t double_close_t dump_t duplicate_metadata_key_t \
empty_container_metadata_t \
gai_error_t get_value_t \
get_value_pointer_bug_t invalid_sockaddr_t \
ipv4_start_cache_t ipv6_lookup_in_ipv4_t max_depth_t metadata_t \
metadata_marker_t metadata_pointers_t no_map_get_value_t \
overflow_bounds_t pointer_dos_t read_node_t \
overflow_bounds_t pointer_dos_t read_node_t repeated_metadata_key_t \
threads_t version_t

data_pool_t_LDFLAGS = $(AM_LDFLAGS) -lm
Expand Down
Loading
Loading