code cleanup

Author: flexatone

README.md

@@ -5,4 +5,30 @@ arraymap
 The ArrayMap library provides dictionary-like lookup from NumPy array values to their integer positions. The hash table design and C implementation is based on [AutoMap](https://github.com/brandtbucher/automap), with extensive additions for direct support of NumPy arrays.
 
 
+Code: https://github.com/static-frame/arraymap
+
+Packages: https://pypi.org/project/arraymap
+
+
+
+Dependencies
+--------------
+
+ArrayKit requires the following:
+
+- Python >= 3.7
+- NumPy >= 1.18.5
+
+
+
+What is New in ArrayMap
+-------------------------
+
+
+
+
+0.1.0
+-------
+
+Initial release with NumPy integration.
 

arraymap.c

@@ -1,113 +1,8 @@
-// TODO: Rewrite performance tests using pyperf.
-// TODO: Group similar functionality.
-// TODO: Check refcounts when calling into hash and comparison functions.
-// TODO: Check allocation and cleanup.
-// TODO: Subinterpreter support.
-// TODO: Docstrings and stubs.
-// TODO: GC support.
 
+// For background on the hashtable design first implemented in AutoMap, see the following:
+// https://github.com/brandtbucher/automap/blob/b787199d38d6bfa1b55484e5ea1e89b31cc1fa72/automap.c#L12
 
-/*******************************************************************************
 
-Our use cases differ significantly from Python's general-purpose dict type, even
-when setting aside the whole immutable/grow-only and contiguous-integer-values
-stuff.
-
-What we don't care about:
-
-  - Memory usage. Python's dicts are used literally everywhere, so a tiny
-    reduction in the footprint of the average dict results in a significant gain
-    for *all* Python programs. We are happy to instead trade a few extra bytes
-    of RAM for a more cache-friendly hash table design. Since we don't store
-    values, we are still close to the same size on average!
-
-  - Worst-case performance. Again, Python's dicts are used for literally
-    everything, so they need to be able to gracefully handle lots of hash
-    collisions, whether resulting from bad hash algorithms, heterogeneous keys
-    with badly-combining hash algorithms, or maliciously-formed input. We can
-    safely assume that our use cases don't need to worry about these issues, and
-    instead choose lookup and collision resolution strategies that utilize cache
-    lines more effectively. This extends to the case of lookups for nonexistent
-    keys as well; we can assume that if our users are looking for something,
-    they know that it's probably there.
-
-What we do care about:
-
-  - Creation and update time. This is *by far* the most expensive operation you
-    do on a mapping. More on this below.
-
-  - The speed of lookups that result in hits. This is what the mapping is used
-    for, so it *must* be good. More on this below.
-
-  - Iteration order and speed. You really can't beat a Python list or tuple
-    here, so we can just store the keys in one of them to avoid reinventing the
-    wheel. We use a list since it allows us to grow more efficiently.
-
-So what we need is a hash table that's easy to insert into and easy to scan.
-
-Here's how it works. A vanilla Python dict of the form:
-
-{a: 0, b: 1, c: 2}
-
-...basically looks like this (assume the hashes are 3, 6, and 9):
-
-Indices: [-, 2, -, 0, -, -, 1,  -]
-
-Hashes:  [3, 6, 9, -, -]
-Keys:    [a, b, c, -, -]
-Values:  [0, 1, 2  -, -]
-
-It's pretty standard; keys, values, and cached hashes are stored in sequential
-order, and their offsets are placed in the Indices table at position
-HASH % TABLE_SIZE. Though it's not used here, collisions are resolved by jumping
-around the table according to the following recurrence:
-
-NEXT_INDEX = (5 * CURRENT_INDEX + 1 + (HASH >>= 5)) % TABLE_SIZE
-
-This is good in the face of bad hash algorithms, but is sorta expensive. It's
-also unable to utilize cache lines at all, since it's basically random (it's
-literally based on random number generation)!
-
-To contrast, the same table looks something like this for us:
-
-Indices: [-, -, -, 0, -, -, 1, -, -, 2, -, -, -, -, -, -, -, -, -]
-Hashes:  [-, -, -, 3, -, -, 6, -, -, 9, -, -, -, -, -, -, -, -, -]
-
-Keys:    [a,  b,  c]
-
-Right away you can see that we don't need to store the values, because they
-match the indices (by design).
-
-Notice that even though we allocated enough space in our table for 19 entries,
-we still insert them into initial position HASH % 4.  This leaves the whole
-15-element tail chunk of the table free for colliding keys. So, what's a good
-collision-resolution strategy?
-
-NEXT_INDEX = CURRENT_INDEX + 1
-
-It's just a sequential scan! That means *every* collision-resolution lookup is
-hot in L1 cache (and can even be predicted and speculatively executed). The
-indices and hashes are actually interleaved for better cache locality as well.
-
-We repeat this scan 15 times. We don't even have to worry about wrapping around
-the edge of the table during this part, since we've left enough free space
-(equal to the number of scans) to safely run over the end. It's wasteful for a
-small example like this, but for more realistic sizes it's just about perfect.
-
-We then jump to another spot in the table using a version of the recurrence
-above:
-
-NEXT_INDEX = (5 * (CURRENT_INDEX - 15) + 1 + (HASH >>= 1)) % TABLE_SIZE
-
-...and repeat the whole thing over again. This collision resolution strategy is
-similar to what Python's sets do, so we still handle some nasty collisions and
-missing keys well.
-
-There are a couple of other tricks that we use (like globally caching integer
-objects from value lookups), but the hardware-friendly hash table design is what
-really gives us our awesome performance.
-
-*******************************************************************************/
 # include <math.h>
 # define PY_SSIZE_T_CLEAN
 # include "Python.h"

tasks.py

@@ -8,15 +8,12 @@
 
 @invoke.task
 def install(context):
-    # type: (invoke.Context) -> None
     run(context, f"{sys.executable} -m pip install --upgrade pip")
     run(context, f"{sys.executable} -m pip install --upgrade -r requirements.txt")
 
 
 @invoke.task()
 def clean(context):
-    # type: (invoke.Context) -> None
-    # run(context, f"{sys.executable} setup.py develop --uninstall")
     run(context, f"{sys.executable} -m pip uninstall --yes arraymap")
 
     for artifact in ("*.egg-info", "*.so", "build", "dist"):
@@ -26,12 +23,9 @@ def clean(context):
 
 @invoke.task(clean)
 def build(context):
-    # type: (invoke.Context) -> None
-    # run(context, f"{sys.executable} setup.py develop")
     run(context, f"{sys.executable} -m pip -v install .")
 
 
 @invoke.task(build)
 def test(context):
-    # type: (invoke.Context) -> None
     run(context, f"{sys.executable} -m pytest -v")