From 6372a8763a8668d010736e9a2b9573687fd5b697 Mon Sep 17 00:00:00 2001 From: Gregory Oschwald Date: Fri, 6 Nov 2026 22:07:32 +0000 Subject: [PATCH 1/9] Read the database file in 4 KiB windows FileReader made one pread system call for each read. The decoder reads a record as many small, mostly adjacent values, so a GeoLite City lookup made about 290 system calls. A read that misses now reads 4 KiB and keeps it as a window. The reader keeps the last two windows, because a data pointer often leaves a map for another window and the next read returns to the map. A City lookup now makes about 16 system calls. The file does not change, so a window does not go stale. Each window is frozen and set in a single assignment, so CRuby needs no lock. JRuby has no global lock, so it takes a mutex to find or replace a window. A returned String must not share the window's buffer, or a cached lookup result keeps 4 KiB alive. CRuby shares the buffer only for a slice that ends at the window end, so only that slice is copied. JRuby shares it for every slice, so JRuby copies every slice. FileReader also overrides inspect, as MemoryReader does, to keep the windows out of irb. GeoLite2-City, 30,000 random IPv4 lookups in MODE_FILE, alternating runs: - CRuby 4.0.9: origin/main takes 157.5 to 161.7 us per lookup and this branch 78.8 to 81.1, about 2x as fast. - JRuby 10.1.3.0: origin/main takes 105.3 to 108.0 us and this branch 93.9 to 94.7, about 11% faster. Results match MODE_MEMORY on 50,000 random lookups. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 8 +++ lib/maxmind/db/file_reader.rb | 85 +++++++++++++++++++++++-- test/test_file_reader.rb | 114 ++++++++++++++++++++++++++++++++++ 3 files changed, 203 insertions(+), 4 deletions(-) create mode 100644 test/test_file_reader.rb diff --git a/CHANGELOG.md b/CHANGELOG.md index a34acab..6c35cae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # Changelog +## 1.5.1 + +* Lookups that read from the file, in `MODE_FILE` and the default + `MODE_AUTO`, are faster. The reader now reads 4 KiB at a time and keeps the + last two blocks for the reads that follow. Before, it made one `pread` + system call for each read. GeoLite City lookups are about 2x as fast on + CRuby 4.0 and about 11% faster on JRuby 10.1. + ## 1.5.0 (2026-09-10) * Fixed two denial-of-service issues in the decoder. A crafted database could diff --git a/lib/maxmind/db/file_reader.rb b/lib/maxmind/db/file_reader.rb index eb73b53..b73dba3 100644 --- a/lib/maxmind/db/file_reader.rb +++ b/lib/maxmind/db/file_reader.rb @@ -33,30 +33,107 @@ def pread(size, offset) end end + # Number of bytes read from the file when a read misses both windows. + WINDOW_SIZE = 4096 + private_constant :WINDOW_SIZE + + JRUBY = defined?(JRUBY_VERSION) ? true : false + private_constant :JRUBY + def initialize(filename) @fh = PReadFile.new(filename, 'rb') @size = @fh.size + @window = nil + @previous_window = nil + # JRuby has no global lock, so an unsynchronized read of @window could + # see the new Array before its elements. + @mutex = JRUBY ? Mutex.new : nil end attr_reader :size + # Override to not show the windows in inspect to avoid showing them in + # irb. + def inspect + "#<#{self.class.name}:0x#{object_id.to_s(16)}, @size=#{@size.inspect}>" + end + def close @fh.close + @window = nil + @previous_window = nil end # Return the byte at +offset+ as an Integer. def getbyte(offset) - read(offset, 1).ord + window = window_covering(offset, 1) + window[2].getbyte(offset - window[0]) end + # Return +size+ bytes at +offset+ as a new String. + # + # The decoder reads a record in many small, mostly adjacent reads. A pread + # for each one costs a system call, so a read that misses reads + # WINDOW_SIZE bytes and keeps them for the reads that follow. def read(offset, size) return ''.b if size == 0 - buf = @fh.pread(size, offset) + if size > WINDOW_SIZE + buf = @fh.pread(size, offset) + raise_bad_data if buf.nil? || buf.bytesize != size + + return buf + end + + window = window_covering(offset, size) + buf = window[2].byteslice(offset - window[0], size) + + # The caller can keep the String, so it must not share the window's + # buffer. CRuby shares it only for a slice that ends at the window + # end. JRuby shares it for every slice. + return buf unless JRUBY || offset + size == window[1] + + String.new(buf, capacity: size) + end - raise InvalidDatabaseError, 'The MaxMind DB file contains bad data' if buf.nil? || buf.length != size + private + + def window_covering(offset, size) + return @mutex.synchronize { find_or_read_window(offset, size) } if @mutex + + find_or_read_window(offset, size) + end + + # Return a window, [start, end, bytes], that holds +size+ bytes at + # +offset+. Two windows are kept because a data pointer often leaves a + # map for another window, and the next read returns to the map. + # + # The file does not change, so a window does not go stale. Each window is + # frozen and set in a single assignment. On CRuby, a concurrent read can + # see either window or miss, and each result is correct. + # + # The range checks are inline to keep a method call off the hot path. + # rubocop:disable-next Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity + def find_or_read_window(offset, size) + window = @window + return window if window && offset >= window[0] && offset + size <= window[1] + + previous = @previous_window + if previous && offset >= previous[0] && offset + size <= previous[1] + @previous_window = window + @window = previous + return previous + end + + bytes = @fh.pread(WINDOW_SIZE, offset) + raise_bad_data if bytes.nil? || bytes.bytesize < size + + @previous_window = window + @window = [offset, offset + bytes.bytesize, bytes.freeze].freeze + end - buf + def raise_bad_data + raise InvalidDatabaseError, 'The MaxMind DB file contains bad data' end end end diff --git a/test/test_file_reader.rb b/test/test_file_reader.rb new file mode 100644 index 0000000..ec3fda8 --- /dev/null +++ b/test/test_file_reader.rb @@ -0,0 +1,114 @@ +# frozen_string_literal: true + +require 'maxmind/db/file_reader' +require 'minitest/autorun' +require 'tempfile' + +class FileReaderTest < Minitest::Test + def setup + # Larger than two 4 KiB windows, so reads can cross a window boundary. + @data = Random.new(1).bytes(10_000) + @file = Tempfile.new('file_reader', binmode: true) + @file.write(@data) + @file.flush + @reader = MaxMind::DB::FileReader.new(@file.path) + end + + def teardown + @reader.close + @file.close! + end + + def test_read_matches_file + rng = Random.new(2) + 2_000.times do + size = rng.rand(0..5_000) + offset = rng.rand(0..(@data.bytesize - size)) + + assert_equal(@data.byteslice(offset, size), @reader.read(offset, size), "read(#{offset}, #{size})") + end + end + + def test_getbyte_matches_file + [0, 4_095, 4_096, 9_999, 5_000, 4_095].each do |offset| + assert_equal(@data.getbyte(offset), @reader.getbyte(offset), "getbyte(#{offset})") + end + end + + def test_reads_reuse_the_two_windows + preads = 0 + fh = @reader.instance_variable_get(:@fh) + fh.define_singleton_method(:pread) do |*args| + preads += 1 + super(*args) + end + + reads = [ + # The first read fills a window. The next reads in it use it. + [[0, 1], [100, 1], [200, 50]], 1, + # A read in a new window keeps the old one as the previous window. + [[8_000, 10], [10, 5], [8_005, 2]], 2, + # A third window drops the least recently used one, which is at 0. The + # window at 8,000 is kept. 9,500 is only in that window. + [[5_000, 10], [9_500, 1], [5_001, 1]], 3, + [[0, 1]], 4, + ] + reads.each_slice(2) do |calls, expected| + calls.each do |offset, size| + assert_equal(@data.byteslice(offset, size), @reader.read(offset, size)) + assert_equal(@data.getbyte(offset), @reader.getbyte(offset)) + end + + assert_equal(expected, preads, calls.inspect) + end + end + + def test_read_returns_a_new_string + buf = @reader.read(0, 3) + buf.force_encoding(Encoding::UTF_8) + buf << 'x' + + assert_equal(@data.byteslice(0, 3), @reader.read(0, 3)) + assert_equal(Encoding::BINARY, @reader.read(0, 3).encoding) + end + + def test_read_does_not_share_the_window + skip 'ObjectSpace.dump is CRuby only' if defined?(JRUBY_VERSION) + + require 'objspace' + @reader.read(0, 1) + # This slice ends at the end of the window that starts at offset 0. + buf = @reader.read(3_000, 1_096) + + assert_equal(@data.byteslice(3_000, 1_096), buf) + refute_includes(ObjectSpace.dump(buf), '"shared":true') + end + + def test_concurrent_reads_match_file + threads = Array.new(8) do |i| + # rubocop:disable ThreadSafety/NewThread + Thread.new do + # rubocop:enable ThreadSafety/NewThread + rng = Random.new(i) + Array.new(2_000) do + size = rng.rand(1..100) + offset = rng.rand(0..(@data.bytesize - size)) + @reader.read(offset, size) == @data.byteslice(offset, size) + end + end + end + + assert_predicate(threads.flat_map(&:value), :all?) + end + + def test_inspect_does_not_show_the_window + @reader.read(0, 1) + + refute_includes(@reader.inspect, 'window') + end + + def test_read_requires_the_full_range + assert_raises(MaxMind::DB::InvalidDatabaseError) { @reader.read(9_998, 3) } + assert_raises(MaxMind::DB::InvalidDatabaseError) { @reader.read(5_000, 5_001) } + end +end From 3ffcd09784ce836a7eb895f809fd80f32786dc2d Mon Sep 17 00:00:00 2001 From: Gregory Oschwald Date: Fri, 6 Nov 2026 22:27:10 +0000 Subject: [PATCH 2/9] Raise InvalidDatabaseError for reads past the end of the file File#pread raises EOFError for an offset at or past the end of the file. A corrupt database whose data pointer targets that range made a lookup in MODE_FILE or MODE_AUTO raise EOFError. MemoryReader and the Windows fallback raise InvalidDatabaseError, as Reader#get documents. FileReader now does the same. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 3 +++ lib/maxmind/db/file_reader.rb | 12 ++++++++++-- test/test_file_reader.rb | 4 ++++ 3 files changed, 17 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6c35cae..768686a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,9 @@ last two blocks for the reads that follow. Before, it made one `pread` system call for each read. GeoLite City lookups are about 2x as fast on CRuby 4.0 and about 11% faster on JRuby 10.1. +* A read at or past the end of the database file in `MODE_FILE` or + `MODE_AUTO` now raises `InvalidDatabaseError`. Before, it raised + `EOFError`. ## 1.5.0 (2026-09-10) diff --git a/lib/maxmind/db/file_reader.rb b/lib/maxmind/db/file_reader.rb index b73dba3..4a87e8a 100644 --- a/lib/maxmind/db/file_reader.rb +++ b/lib/maxmind/db/file_reader.rb @@ -79,7 +79,7 @@ def read(offset, size) return ''.b if size == 0 if size > WINDOW_SIZE - buf = @fh.pread(size, offset) + buf = pread(size, offset) raise_bad_data if buf.nil? || buf.bytesize != size return buf @@ -125,13 +125,21 @@ def find_or_read_window(offset, size) return previous end - bytes = @fh.pread(WINDOW_SIZE, offset) + bytes = pread(WINDOW_SIZE, offset) raise_bad_data if bytes.nil? || bytes.bytesize < size @previous_window = window @window = [offset, offset + bytes.bytesize, bytes.freeze].freeze end + # File#pread raises EOFError for an offset at or past the end of the + # file, and the Windows fallback returns nil. Return nil for both. + def pread(size, offset) + @fh.pread(size, offset) + rescue EOFError + nil + end + def raise_bad_data raise InvalidDatabaseError, 'The MaxMind DB file contains bad data' end diff --git a/test/test_file_reader.rb b/test/test_file_reader.rb index ec3fda8..36fe35e 100644 --- a/test/test_file_reader.rb +++ b/test/test_file_reader.rb @@ -110,5 +110,9 @@ def test_inspect_does_not_show_the_window def test_read_requires_the_full_range assert_raises(MaxMind::DB::InvalidDatabaseError) { @reader.read(9_998, 3) } assert_raises(MaxMind::DB::InvalidDatabaseError) { @reader.read(5_000, 5_001) } + assert_raises(MaxMind::DB::InvalidDatabaseError) { @reader.getbyte(10_000) } + assert_raises(MaxMind::DB::InvalidDatabaseError) { @reader.read(10_000, 1) } + assert_raises(MaxMind::DB::InvalidDatabaseError) { @reader.read(10_001, 1) } + assert_raises(MaxMind::DB::InvalidDatabaseError) { @reader.read(10_000, 5_000) } end end From 703e938ba20bf999ec44231cd676e3823cfb471d Mon Sep 17 00:00:00 2001 From: Gregory Oschwald Date: Fri, 6 Nov 2026 22:56:20 +0000 Subject: [PATCH 3/9] Read search tree bits from the address as an Integer The tree walk took each bit from the packed address with String#[], which created a 1-byte String for each of up to 128 bits. Take the bit from the address's Integer value with Integer#[] instead. Integer#[] does not allocate, even for the Bignum of an IPv6 address, while a shift of a Bignum would. GeoLite2-City, CRuby 4.0.9, alternating runs: - 30,000 random IPv4 lookups, 4 runs: memory mode goes from 45.8-47.4 to 45.0-46.3 us per lookup, and file mode from 79.4-80.4 to 77.2-78.0 us. - 30,000 tree walks for random IPv6 addresses in 2000::/4: from 3.62-3.68 to 1.76-1.80 us per walk, and from 34.4 to 9.7 objects. Co-Authored-By: Claude Opus 5.5 --- lib/maxmind/db.rb | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/lib/maxmind/db.rb b/lib/maxmind/db.rb index 8e776bd..24c3881 100644 --- a/lib/maxmind/db.rb +++ b/lib/maxmind/db.rb @@ -203,7 +203,7 @@ def get_with_prefix_length(ip_address) private_constant :IP_VERSION_TO_BIT_COUNT def find_address_in_tree(ip_address, ip_version) - packed = ip_address.hton + ip = ip_address.to_i bit_count = IP_VERSION_TO_BIT_COUNT[ip_version] node = start_node(bit_count) @@ -211,12 +211,8 @@ def find_address_in_tree(ip_address, ip_version) node_count = @node_count depth = 0 - loop do - break if depth >= bit_count || node >= node_count - - c = packed[depth >> 3].ord - bit = 1 & (c >> (7 - (depth % 8))) - node = read_node(node, bit) + while depth < bit_count && node < node_count + node = read_node(node, ip[bit_count - 1 - depth]) depth += 1 end From e8ea4150d4801ebe92801520300b46ad9076b94a Mon Sep 17 00:00:00 2001 From: Gregory Oschwald Date: Fri, 6 Nov 2026 22:57:56 +0000 Subject: [PATCH 4/9] Check the current window first in FileReader reads Each getbyte and read went through window_covering and find_or_read_window, two method calls, even when the current window held the bytes. Most reads hit the current window, so check it in getbyte and read first. JRuby still takes the lock, because it must not read the window without it. GeoLite2-City, 30,000 random IPv4 lookups, CRuby 4.0.9, 3 alternating runs: file mode goes from 78.7-80.3 to 74.2-75.3 us per lookup. Co-Authored-By: Claude Opus 5.5 --- lib/maxmind/db/file_reader.rb | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/lib/maxmind/db/file_reader.rb b/lib/maxmind/db/file_reader.rb index 4a87e8a..5d9488c 100644 --- a/lib/maxmind/db/file_reader.rb +++ b/lib/maxmind/db/file_reader.rb @@ -66,7 +66,10 @@ def close # Return the byte at +offset+ as an Integer. def getbyte(offset) - window = window_covering(offset, 1) + # Most reads hit the current window, so check it here before the + # method calls of the full search. JRuby must take the lock to read it. + window = @window + window = window_covering(offset, 1) if JRUBY || !window || offset < window[0] || offset >= window[1] window[2].getbyte(offset - window[0]) end @@ -75,6 +78,7 @@ def getbyte(offset) # The decoder reads a record in many small, mostly adjacent reads. A pread # for each one costs a system call, so a read that misses reads # WINDOW_SIZE bytes and keeps them for the reads that follow. + # rubocop:disable-next Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity def read(offset, size) return ''.b if size == 0 @@ -85,7 +89,8 @@ def read(offset, size) return buf end - window = window_covering(offset, size) + window = @window + window = window_covering(offset, size) if JRUBY || !window || offset < window[0] || offset + size > window[1] buf = window[2].byteslice(offset - window[0], size) # The caller can keep the String, so it must not share the window's From 2cd536ab1019fd28351a0c25f5f8ce4c61620220 Mon Sep 17 00:00:00 2001 From: Gregory Oschwald Date: Fri, 6 Nov 2026 22:59:07 +0000 Subject: [PATCH 5/9] Keep the decode offset and budgets on a per-call decoder Each value decoder returned a new [value, offset] Array, and the budgets were a 3-element Array passed through every call. A City lookup decodes a few hundred values, so these Arrays were a large share of its allocations. decode now runs on a dup of the decoder. The copy holds the offset and the budgets as instance variables, and each helper returns only the value. The shared decoder never changes, so it stays safe for concurrent lookups. The limits and their checks do not change. GeoLite2-City, 30,000 random IPv4 lookups, CRuby 4.0.9, 3 alternating runs: memory mode goes from 45.3-45.8 to 40.8-42.1 us per lookup, and file mode from 73.5-74.3 to 69.6-70.5 us. Co-Authored-By: Claude Opus 5.5 --- lib/maxmind/db/decoder.rb | 195 ++++++++++++++++++++------------------ 1 file changed, 101 insertions(+), 94 deletions(-) diff --git a/lib/maxmind/db/decoder.rb b/lib/maxmind/db/decoder.rb index 6a6c0a7..d4daead 100644 --- a/lib/maxmind/db/decoder.rb +++ b/lib/maxmind/db/decoder.rb @@ -42,10 +42,7 @@ def initialize(io, pointer_base = 0, pointer_test = false, # Per-decode limits. The value and depth limits are the ones the MaxMind DB # specification recommends. The specification leaves the payload limit to - # the reader, and 2 MiB matches libmaxminddb. +budget+ is a three-element - # array, [values_remaining, depth, bytes_remaining], shared across the - # recursion so every count survives it. It is call-local, which keeps the - # decoder safe for concurrent reads. + # the reader, and 2 MiB matches libmaxminddb. # # The value limit stops a pointer fan-out. It follows the specification's # flat rule: the root is one value, each array reserves one value per @@ -74,11 +71,6 @@ def initialize(io, pointer_base = 0, pointer_test = false, MAX_DEPTH = 512 private_constant :MAX_DEPTH - BUDGET_VALUES = 0 - BUDGET_DEPTH = 1 - BUDGET_BYTES = 2 - private_constant :BUDGET_VALUES, :BUDGET_DEPTH, :BUDGET_BYTES - # JRuby can exhaust the stack before the depth limit is reached and raises # a Java StackOverflowError, which is not a SystemStackError. Catch both so # a pointer cycle always becomes an InvalidDatabaseError. @@ -112,37 +104,34 @@ def raise_bytes_exceeded 'The MaxMind DB file\'s data section exceeds the maximum number of bytes' end - def decode_array(size, offset, budget) - raise_values_exceeded if (budget[BUDGET_VALUES] -= size) < 0 - raise_depth_exceeded if (budget[BUDGET_DEPTH] += 1) > @max_depth + def decode_array(size, offset) + raise_values_exceeded if (@values_left -= size) < 0 + raise_depth_exceeded if (@depth += 1) > @max_depth + @offset = offset array = [] size.times do - value, offset = decode_with_budget(offset, budget) - array << value + array << decode_value end - budget[BUDGET_DEPTH] -= 1 - [array, offset] - end - - def decode_boolean(size, offset) - [size != 0, offset] + @depth -= 1 + array end - def decode_bytes(size, offset, budget) - raise_bytes_exceeded if (budget[BUDGET_BYTES] -= size) < 0 - [@io.read(offset, size), offset + size] + def decode_bytes(size, offset) + raise_bytes_exceeded if (@bytes_left -= size) < 0 + @offset = offset + size + @io.read(offset, size) end def decode_double(size, offset) verify_size(8, size) - buf = @io.read(offset, 8) - [buf.unpack1('G'), offset + 8] + @offset = offset + 8 + @io.read(offset, 8).unpack1('G') end def decode_float(size, offset) verify_size(4, size) - buf = @io.read(offset, 4) - [buf.unpack1('g'), offset + 4] + @offset = offset + 4 + @io.read(offset, 4).unpack1('g') end def verify_size(expected, actual) @@ -156,64 +145,63 @@ def raise_invalid_size 'The MaxMind DB file\'s data section contains bad data (unknown data type or corrupt data)' end - def decode_int32(size, offset, budget) - decode_int('l>', 4, size, offset, budget) + def decode_int32(size, offset) + decode_int('l>', 4, size, offset) end - def decode_uint16(size, offset, budget) - decode_int('n', 2, size, offset, budget) + def decode_uint16(size, offset) + decode_int('n', 2, size, offset) end - def decode_uint32(size, offset, budget) - decode_int('N', 4, size, offset, budget) + def decode_uint32(size, offset) + decode_int('N', 4, size, offset) end - def decode_uint64(size, offset, budget) - decode_int('Q>', 8, size, offset, budget) + def decode_uint64(size, offset) + decode_int('Q>', 8, size, offset) end - def decode_int(type_code, type_size, size, offset, budget) + def decode_int(type_code, type_size, size, offset) raise_invalid_size if size > type_size - return 0, offset if size == 0 + @offset = offset + size + return 0 if size == 0 - raise_bytes_exceeded if (budget[BUDGET_BYTES] -= size) < 0 + raise_bytes_exceeded if (@bytes_left -= size) < 0 buf = @io.read(offset, size) buf = buf.rjust(type_size, "\x00") if size != type_size - [buf.unpack1(type_code), offset + size] + buf.unpack1(type_code) end - def decode_uint128(size, offset, budget) + def decode_uint128(size, offset) raise_invalid_size if size > 16 - return 0, offset if size == 0 + @offset = offset + size + return 0 if size == 0 - raise_bytes_exceeded if (budget[BUDGET_BYTES] -= size) < 0 + raise_bytes_exceeded if (@bytes_left -= size) < 0 buf = @io.read(offset, size) - if size <= 8 - buf = buf.rjust(8, "\x00") - return buf.unpack1('Q>'), offset + size - end + return buf.rjust(8, "\x00").unpack1('Q>') if size <= 8 a_bytes = buf[0...-8].rjust(8, "\x00") b_bytes = buf[-8...buf.length] a = a_bytes.unpack1('Q>') b = b_bytes.unpack1('Q>') a <<= 64 - [a | b, offset + size] + a | b end - def decode_map(size, offset, budget) + def decode_map(size, offset) # A map entry decodes a key and a value, so it costs two values. - raise_values_exceeded if (budget[BUDGET_VALUES] -= size * 2) < 0 - raise_depth_exceeded if (budget[BUDGET_DEPTH] += 1) > @max_depth + raise_values_exceeded if (@values_left -= size * 2) < 0 + raise_depth_exceeded if (@depth += 1) > @max_depth + @offset = offset container = {} size.times do - key, offset = decode_with_budget(offset, budget) - value, offset = decode_with_budget(offset, budget) - container[key] = value + key = decode_value + container[key] = decode_value end - budget[BUDGET_DEPTH] -= 1 - [container, offset] + @depth -= 1 + container end def decode_pointer(size, offset) @@ -243,14 +231,14 @@ def decode_pointer(size, offset) [pointer, new_offset] end - def decode_utf8_string(size, offset, budget) - raise_bytes_exceeded if (budget[BUDGET_BYTES] -= size) < 0 - new_offset = offset + size + def decode_utf8_string(size, offset) + raise_bytes_exceeded if (@bytes_left -= size) < 0 + @offset = offset + size buf = @io.read(offset, size) buf.force_encoding(Encoding::UTF_8) # We could check it's valid UTF-8 with `valid_encoding?', but for # performance I do not. - [buf, new_offset] + buf end public @@ -264,27 +252,41 @@ def decode_utf8_string(size, offset, budget) # # Throws an exception if there is an error. def decode(offset) + # Each call decodes with its own copy of the decoder. The copy holds the + # offset and the remaining budgets, so the helpers return only the + # value, and this decoder never changes. That keeps it safe for + # concurrent reads. + dup.decode_root(offset) + end + + protected + + def decode_root(offset) # Bound the work per decode so a crafted database cannot exhaust CPU or - # memory. +budget+ carries the remaining value count, the current depth, - # and the remaining payload-byte allowance, and is call-local, which - # keeps the decoder safe for concurrent reads. The root value is charged - # here; containers charge their children. The depth limit catches a - # pointer cycle on MRI. JRuby can exhaust the stack before the limit is - # reached and raises a Java StackOverflowError, so catch that too and - # report the same error. - decode_with_budget(offset, [@max_values - 1, 0, @max_payload_bytes]) + # memory. The root value is charged here. Containers charge their + # children. The depth limit catches a pointer cycle on MRI. JRuby can + # exhaust the stack before the limit is reached and raises a Java + # StackOverflowError, so catch that too and report the same error. + @offset = offset + @values_left = @max_values - 1 + @depth = 0 + @bytes_left = @max_payload_bytes + value = decode_value + [value, @offset] rescue *STACK_ERRORS - raise InvalidDatabaseError, - 'The MaxMind DB file\'s data section exceeds the maximum depth' + raise_depth_exceeded end private + # Decode the value at @offset and move @offset past it. + # # The dispatch below is one branch per data type, so the method's # cyclomatic complexity is above the cop's default. It is inlined here # for speed and the branches are uniform. # rubocop:disable-next Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity - def decode_with_budget(offset, budget) + def decode_value + offset = @offset pointer_return_offset = nil new_offset = offset + 1 ctrl_byte = @io.getbyte(offset) @@ -294,13 +296,16 @@ def decode_with_budget(offset, budget) size, new_offset = size_from_ctrl_byte(ctrl_byte, new_offset, type_num) if type_num == 1 pointer, pointer_return_offset = decode_pointer(size, new_offset) - return [pointer, pointer_return_offset] if @pointer_test + if @pointer_test + @offset = pointer_return_offset + return pointer + end # The root or containing collection already charged the logical value # at the pointer's position. Following it adds depth but no separate # value. Its target cannot be another pointer, and decoding the target # still reserves container children and charges payload bytes. - raise_depth_exceeded if (budget[BUDGET_DEPTH] += 1) > @max_depth + raise_depth_exceeded if (@depth += 1) > @max_depth new_offset = pointer + 1 ctrl_byte = @io.getbyte(pointer) type_num = ctrl_byte >> 5 @@ -314,28 +319,30 @@ def decode_with_budget(offset, budget) # Direct case dispatch avoids looking the method up in a Hash and # calling it with send. - result = case type_num - when 2 then decode_utf8_string(size, new_offset, budget) - when 3 then decode_double(size, new_offset) - when 4 then decode_bytes(size, new_offset, budget) - when 5 then decode_uint16(size, new_offset, budget) - when 6 then decode_uint32(size, new_offset, budget) - when 7 then decode_map(size, new_offset, budget) - when 8 then decode_int32(size, new_offset, budget) - when 9 then decode_uint64(size, new_offset, budget) - when 10 then decode_uint128(size, new_offset, budget) - when 11 then decode_array(size, new_offset, budget) - when 14 then decode_boolean(size, new_offset) - when 15 then decode_float(size, new_offset) - else - raise InvalidDatabaseError, - "The MaxMind DB file's data section contains bad data (unknown data type #{type_num})" - end - return result unless pointer_return_offset - - budget[BUDGET_DEPTH] -= 1 - result[1] = pointer_return_offset - result + value = case type_num + when 2 then decode_utf8_string(size, new_offset) + when 3 then decode_double(size, new_offset) + when 4 then decode_bytes(size, new_offset) + when 5 then decode_uint16(size, new_offset) + when 6 then decode_uint32(size, new_offset) + when 7 then decode_map(size, new_offset) + when 8 then decode_int32(size, new_offset) + when 9 then decode_uint64(size, new_offset) + when 10 then decode_uint128(size, new_offset) + when 11 then decode_array(size, new_offset) + when 14 + @offset = new_offset + size != 0 + when 15 then decode_float(size, new_offset) + else + raise InvalidDatabaseError, + "The MaxMind DB file's data section contains bad data (unknown data type #{type_num})" + end + return value unless pointer_return_offset + + @depth -= 1 + @offset = pointer_return_offset + value end def read_extended(offset) From 5ae0d21da2b15e2a8f742ab2b5e4bffb6ba7a33a Mon Sep 17 00:00:00 2001 From: Gregory Oschwald Date: Fri, 6 Nov 2026 22:59:55 +0000 Subject: [PATCH 6/9] Parse the control byte without returning Arrays read_extended, size_from_ctrl_byte, and decode_pointer each returned a new 2-element Array, and size_from_ctrl_byte ran for every value. Read the extended type and the size inline in a new decode_data method. decode_pointer now follows the pointer itself and calls decode_data for the target. read_extended's check for a type below 7 is gone, because a byte plus 7 cannot be below 7. GeoLite2-City, 30,000 random IPv4 lookups, CRuby 4.0.9, 3 alternating runs: memory mode goes from 40.9-41.8 to 34.1-34.6 us per lookup, and file mode from 69.7-70.2 to 62.8-64.6 us. Co-Authored-By: Claude Opus 5.5 --- lib/maxmind/db/decoder.rb | 189 ++++++++++++++++++-------------------- 1 file changed, 87 insertions(+), 102 deletions(-) diff --git a/lib/maxmind/db/decoder.rb b/lib/maxmind/db/decoder.rb index d4daead..6a75128 100644 --- a/lib/maxmind/db/decoder.rb +++ b/lib/maxmind/db/decoder.rb @@ -204,33 +204,6 @@ def decode_map(size, offset) container end - def decode_pointer(size, offset) - pointer_size = size >> 3 - - # Build the pointer with integer arithmetic to avoid temporary strings - # when combining control bits with the payload bytes. - case pointer_size - when 0 - new_offset = offset + 1 - pointer = ((size & 0x7) << 8) | @io.getbyte(offset) - when 1 - new_offset = offset + 2 - pointer = ((size & 0x7) << 16) | @io.read(offset, 2).unpack1('n') - pointer += 2048 - when 2 - new_offset = offset + 3 - buf = @io.read(offset, 3) - pointer = ((size & 0x7) << 24) | (buf.getbyte(0) << 16) | - (buf.getbyte(1) << 8) | buf.getbyte(2) - pointer += 526_336 - else - new_offset = offset + 4 - pointer = @io.read(offset, 4).unpack1('N') - end - pointer += @pointer_base - [pointer, new_offset] - end - def decode_utf8_string(size, offset) raise_bytes_exceeded if (@bytes_left -= size) < 0 @offset = offset + size @@ -280,100 +253,112 @@ def decode_root(offset) private # Decode the value at @offset and move @offset past it. - # - # The dispatch below is one branch per data type, so the method's - # cyclomatic complexity is above the cop's default. It is inlined here - # for speed and the branches are uniform. - # rubocop:disable-next Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity def decode_value offset = @offset - pointer_return_offset = nil - new_offset = offset + 1 ctrl_byte = @io.getbyte(offset) type_num = ctrl_byte >> 5 - type_num, new_offset = read_extended(new_offset) if type_num == 0 + return decode_pointer(ctrl_byte, offset + 1) if type_num == 1 - size, new_offset = size_from_ctrl_byte(ctrl_byte, new_offset, type_num) - if type_num == 1 - pointer, pointer_return_offset = decode_pointer(size, new_offset) - if @pointer_test - @offset = pointer_return_offset - return pointer - end + decode_data(ctrl_byte, type_num, offset + 1) + end - # The root or containing collection already charged the logical value - # at the pointer's position. Following it adds depth but no separate - # value. Its target cannot be another pointer, and decoding the target - # still reserves container children and charges payload bytes. - raise_depth_exceeded if (@depth += 1) > @max_depth - new_offset = pointer + 1 - ctrl_byte = @io.getbyte(pointer) - type_num = ctrl_byte >> 5 - if type_num == 1 - raise InvalidDatabaseError, - 'The MaxMind DB file\'s data section contains bad data (pointer points to another pointer)' - end - type_num, new_offset = read_extended(new_offset) if type_num == 0 - size, new_offset = size_from_ctrl_byte(ctrl_byte, new_offset, type_num) + def decode_pointer(ctrl_byte, offset) + size = ctrl_byte & 0x1f + pointer_size = size >> 3 + + # Build the pointer with integer arithmetic to avoid temporary strings + # when combining control bits with the payload bytes. + case pointer_size + when 0 + return_offset = offset + 1 + pointer = ((size & 0x7) << 8) | @io.getbyte(offset) + when 1 + return_offset = offset + 2 + pointer = ((size & 0x7) << 16) | @io.read(offset, 2).unpack1('n') + pointer += 2048 + when 2 + return_offset = offset + 3 + buf = @io.read(offset, 3) + pointer = ((size & 0x7) << 24) | (buf.getbyte(0) << 16) | + (buf.getbyte(1) << 8) | buf.getbyte(2) + pointer += 526_336 + else + return_offset = offset + 4 + pointer = @io.read(offset, 4).unpack1('N') end + pointer += @pointer_base - # Direct case dispatch avoids looking the method up in a Hash and - # calling it with send. - value = case type_num - when 2 then decode_utf8_string(size, new_offset) - when 3 then decode_double(size, new_offset) - when 4 then decode_bytes(size, new_offset) - when 5 then decode_uint16(size, new_offset) - when 6 then decode_uint32(size, new_offset) - when 7 then decode_map(size, new_offset) - when 8 then decode_int32(size, new_offset) - when 9 then decode_uint64(size, new_offset) - when 10 then decode_uint128(size, new_offset) - when 11 then decode_array(size, new_offset) - when 14 - @offset = new_offset - size != 0 - when 15 then decode_float(size, new_offset) - else - raise InvalidDatabaseError, - "The MaxMind DB file's data section contains bad data (unknown data type #{type_num})" - end - return value unless pointer_return_offset + if @pointer_test + @offset = return_offset + return pointer + end + + # The root or containing collection already charged the logical value + # at the pointer's position. Following it adds depth but no separate + # value. Its target cannot be another pointer, and decoding the target + # still reserves container children and charges payload bytes. + raise_depth_exceeded if (@depth += 1) > @max_depth + ctrl_byte = @io.getbyte(pointer) + type_num = ctrl_byte >> 5 + if type_num == 1 + raise InvalidDatabaseError, + 'The MaxMind DB file\'s data section contains bad data (pointer points to another pointer)' + end + value = decode_data(ctrl_byte, type_num, pointer + 1) @depth -= 1 - @offset = pointer_return_offset + @offset = return_offset value end - def read_extended(offset) - next_byte = @io.getbyte(offset) - type_num = next_byte + 7 - if type_num < 7 - raise InvalidDatabaseError, - "Something went horribly wrong in the decoder. An extended type resolved to a type number < 8 (#{type_num})" + # Decode a value that is not a pointer. +offset+ is the byte after the + # control byte. + # + # The dispatch below is one branch per data type, so the method's + # cyclomatic complexity is above the cop's default. It is inlined here + # for speed and the branches are uniform. + # rubocop:disable-next Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity + def decode_data(ctrl_byte, type_num, offset) + if type_num == 0 + type_num = @io.getbyte(offset) + 7 + offset += 1 end - [type_num, offset + 1] - end - def size_from_ctrl_byte(ctrl_byte, offset, type_num) size = ctrl_byte & 0x1f - - return size, offset if type_num == 1 || size < 29 - - if size == 29 - size = 29 + @io.getbyte(offset) - return size, offset + 1 + if size >= 29 + if size == 29 + size = 29 + @io.getbyte(offset) + offset += 1 + elsif size == 30 + size = 285 + @io.read(offset, 2).unpack1('n') + offset += 2 + else + size = 65_821 + ("\x00".b << @io.read(offset, 3)).unpack1('N') + offset += 3 + end end - if size == 30 - size_bytes = @io.read(offset, 2) - size = 285 + size_bytes.unpack1('n') - return size, offset + 2 + # Direct case dispatch avoids looking the method up in a Hash and + # calling it with send. + case type_num + when 2 then decode_utf8_string(size, offset) + when 3 then decode_double(size, offset) + when 4 then decode_bytes(size, offset) + when 5 then decode_uint16(size, offset) + when 6 then decode_uint32(size, offset) + when 7 then decode_map(size, offset) + when 8 then decode_int32(size, offset) + when 9 then decode_uint64(size, offset) + when 10 then decode_uint128(size, offset) + when 11 then decode_array(size, offset) + when 14 + @offset = offset + size != 0 + when 15 then decode_float(size, offset) + else + raise InvalidDatabaseError, + "The MaxMind DB file's data section contains bad data (unknown data type #{type_num})" end - - size_bytes = "\x00".b << @io.read(offset, 3) - size = 65_821 + size_bytes.unpack1('N') - [size, offset + 3] end end end From a01c3d6d5eec04ca9abae7ff5a01379d6a79c1fb Mon Sep 17 00:00:00 2001 From: Gregory Oschwald Date: Fri, 6 Nov 2026 23:00:22 +0000 Subject: [PATCH 7/9] Use while loops to decode maps and arrays Integer#times calls a block for each entry. A while loop does not. The array still grows as its elements decode. Allocating it at the declared size up front was not faster, and a crafted header with a raised max_values limit could then allocate memory for elements that are not in the file. GeoLite2-City, 30,000 random IPv4 lookups, CRuby 4.0.9, 4 alternating runs: memory mode goes from 34.0-34.6 to 32.9-33.7 us per lookup, and file mode from 62.1-62.7 to 60.5-61.7 us. Co-Authored-By: Claude Opus 5.5 --- lib/maxmind/db/decoder.rb | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/maxmind/db/decoder.rb b/lib/maxmind/db/decoder.rb index 6a75128..c6fb56b 100644 --- a/lib/maxmind/db/decoder.rb +++ b/lib/maxmind/db/decoder.rb @@ -109,8 +109,10 @@ def decode_array(size, offset) raise_depth_exceeded if (@depth += 1) > @max_depth @offset = offset array = [] - size.times do + i = 0 + while i < size array << decode_value + i += 1 end @depth -= 1 array @@ -196,9 +198,11 @@ def decode_map(size, offset) raise_depth_exceeded if (@depth += 1) > @max_depth @offset = offset container = {} - size.times do + i = 0 + while i < size key = decode_value container[key] = decode_value + i += 1 end @depth -= 1 container From 29bd05f4f4d31ba4a7632770a1646173742de496 Mon Sep 17 00:00:00 2001 From: Gregory Oschwald Date: Fri, 6 Nov 2026 23:39:13 +0000 Subject: [PATCH 8/9] Describe the faster lookups in the changelog The entry described only the file reader. Say that lookups are faster in all modes. The measurements are in the commit messages and the PR. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 768686a..386c997 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,11 +2,7 @@ ## 1.5.1 -* Lookups that read from the file, in `MODE_FILE` and the default - `MODE_AUTO`, are faster. The reader now reads 4 KiB at a time and keeps the - last two blocks for the reads that follow. Before, it made one `pread` - system call for each read. GeoLite City lookups are about 2x as fast on - CRuby 4.0 and about 11% faster on JRuby 10.1. +* Lookups are faster, especially in `MODE_FILE` and the default `MODE_AUTO`. * A read at or past the end of the database file in `MODE_FILE` or `MODE_AUTO` now raises `InvalidDatabaseError`. Before, it raised `EOFError`. From 82e73f2cf3c1233b59c0141b0de26052fc247df8 Mon Sep 17 00:00:00 2001 From: Gregory Oschwald Date: Sat, 7 Nov 2026 00:24:02 +0000 Subject: [PATCH 9/9] Find the metadata marker with one read find_metadata_start read 14 bytes at each offset from the end of the file, up to 128 KiB back. In MODE_FILE, each read missed the file reader's windows, which start at the offset that missed, so each step read up to 4 KiB. Read the last 128 KiB once and search it with String#rindex. That finds the same marker, the last one in the tail. A new test checks that a buffer shorter than the marker, or one that ends with part of it, is rejected. GeoLite2-City in MODE_FILE, CRuby 4.0.9, alternating runs: opening the database goes from 0.203-0.207 to 0.072-0.077 ms. Rejecting a 300 KB file that is not a database goes from 111.7-116.8 to 0.0-0.1 ms. Lookups do not change. Co-Authored-By: Claude Opus 5.5 --- lib/maxmind/db.rb | 20 ++++++++------------ test/test_reader.rb | 12 ++++++++++++ 2 files changed, 20 insertions(+), 12 deletions(-) diff --git a/lib/maxmind/db.rb b/lib/maxmind/db.rb index 24c3881..ed18dbf 100644 --- a/lib/maxmind/db.rb +++ b/lib/maxmind/db.rb @@ -303,21 +303,17 @@ def decoder_limits(options) def find_metadata_start metadata_max_size = [@size, METADATA_MAX_SIZE].min + tail_start = @size - metadata_max_size - stop_index = @size - metadata_max_size - index = @size - METADATA_START_MARKER_LENGTH - while index >= stop_index - return index + METADATA_START_MARKER_LENGTH if at_metadata?(index) - - index -= 1 + # Read the tail once and search it from the end, so the last marker wins. + # One read is much faster than a read at each offset. + index = @io.read(tail_start, metadata_max_size).rindex(METADATA_START_MARKER) + unless index + raise InvalidDatabaseError, + 'Metadata section not found. Is this a valid MaxMind DB file?' end - raise InvalidDatabaseError, - 'Metadata section not found. Is this a valid MaxMind DB file?' - end - - def at_metadata?(index) - @io.read(index, METADATA_START_MARKER_LENGTH) == METADATA_START_MARKER + tail_start + index + METADATA_START_MARKER_LENGTH end public diff --git a/test/test_reader.rb b/test/test_reader.rb index 76543bf..70e1af9 100644 --- a/test/test_reader.rb +++ b/test/test_reader.rb @@ -396,6 +396,18 @@ def test_nondatabase ) end + def test_short_nondatabase + ['', "\xAB\xCD".b, "\xAB\xCD\xEFMaxMind.co".b].each do |buf| + e = assert_raises(MaxMind::DB::InvalidDatabaseError, buf.inspect) do + MaxMind::DB.new(buf, mode: MaxMind::DB::MODE_PARAM_IS_BUFFER) + end + assert_equal( + 'Metadata section not found. Is this a valid MaxMind DB file?', + e.message, + ) + end + end + def test_too_many_constructor_args e = assert_raises ArgumentError do MaxMind::DB.new('README.md', {}, 'blah')