ZEEK-ONLY: avoid initialization of every calloc'd int in fd_map

[ckreibich]

I keep running into the problem of Zeek gobbling up gigabytes of memory in Docker due to libkqueue's over-eager array allocations of the maximum number of file descriptors. This change is the smallest I could come up with to avoid that problem, simply by avoiding initialization of every int in a calloc()'d array.

Memory use of a regular Zeek in Docker, per htop (note resident memory):

    PID USER       PRI  NI  VIRT   RES   SHR S  CPU% MEM%▽  TIME+  Command (merged)
  42741 root        20   0 17.0G 4357M  176M S   0.0  6.8  0:01.26 zeek -i lo

Memory use of a build with this patch in Docker:

    PID USER       PRI  NI  VIRT   RES   SHR S  CPU% MEM%▽  TIME+  Command (merged)
  42460 root        20   0 17.0G  263M  177M S   0.0  0.4  0:00.21 zeek -i lo

Memory use outside of Docker, on the same system (with ulimit of 1024 fds):

    PID USER       PRI  NI  VIRT   RES   SHR S  CPU% MEM%▽  TIME+  Command (merged)
  47002 root        20   0 1083M  292M  202M S   0.0  0.5  0:00.63 zeek -i lo

See commit for details. @timwoj I wasn't quite sure where to point the merge at, does this work for you?

[timwoj]

I wasn't quite sure where to point the merge at, does this work for you?

It should just point at master and we'll make a new branch for where it gets mapped into zeek.

I'm confused by how this works though. It's still calling calloc on the massive array. Is it simply the fact that you're not setting them all to a non-zero value the part that makes it not think it's using the memory? Also, what's the need for the addition/subtraction in the get/set methods?

[ckreibich]

Will do, Tim.

The calloc() alone doesn't actually use the allocated memory. At least on Linux calloc() is optimized to return a single, zero-filled page on read access, and copy-on-write (and thus allocate) only when you write other values. The initialization to -1 that the code did thus ensured that the full 4GB were really actually using up memory.

The +1/-1 is just so the surrounding checks for -1 (there's only one) keep working.

I think one could go to town on all of this code a bunch more (given there's a separate array implementing a use count, but thankfully keeping the default to 0), and I actually started out implementing an abstraction that initially allocates a small chunk and then reallocs as needed as larger fds come in. But this had me thinking it's not actually needed.

[ckreibich]

Closing in favor of #3 so it's an isolated change on top of master.