Segfault while building the Rust toolchain with a toolchain that uses mimalloc

So I'm trying to build a Rust toolchain with mimalloc that works here: rust-lang/rust#62340 , such that we can use the Rust toolchain compiler benchmark suite to test whether it would be worth it switching to mimalloc from jemalloc.

I think I've managed so far to link mimalloc properly, and built a toolchain that's statically linked with it and uses it. When that toolchain is used to build another toolchain, after building a chunk of it, it segfaults with: signal: 11, SIGSEGV: invalid memory reference. Target is x86_64-unknown-linux-gnu , and I'm using the master branch of mimalloc, via the mimalloc-sys bindings, which are here: https://github.com/gnzlbg/mimallocator/tree/master/mimalloc-sys

I'm building mimalloc following the steps from this script: https://github.com/gnzlbg/mimallocator/blob/master/mimalloc-sys/build.rs

Very nice you are working on integrating this with Rust -- let me know of issues or performance.
With regard to the crash, when integrating with a language, it is best to:

if the runtime uses plain malloc/free, then first use LD_PRELOAD to see if those programs still work. This lets you know if the bug is in mimalloc or in the integration.
for a language runtime, it is best to link statically, and have the runtime consistently use the mi_ prefixed API and never mix up pointers gotten through some other API. However, that means special care must be taken that pointers obtained through some C library do not get freed with the new Rust free (which uses mi_free). In particular, watch for uses of strdup, strndup, and realpath ! and use the mi_ prefixed versions instead. Same for posix_align etc. (this will get easier once I get to an update for issue Expose standard C and POSIX APIs even without override #75 )

Do you have a backtrace of the segfault? Was it using the debug version of mimalloc with -DMI_CHECK_FULL=ON enabled?

I will try help out and try to look at the Rust thread tomorrow.

edit: I looked a little and saw rust already works with jemalloc so the above issues should already be handled well. Don't use MI_OVERRIDE and statically link seems the best approach (which you are already doing).

Just a note, ensure the aligment and size arguments are in the right order for the aligned allocation functions in:

        static _F2: unsafe extern fn(*mut *mut c_void, usize, usize) -> c_int =
            mimalloc_sys::posix_memalign;
        static _F3: unsafe extern fn(usize, usize) -> *mut c_void =
            mimalloc_sys::aligned_alloc;

since mimalloc lets alignment always come after the size but posix_memalign and memalign do it the other way around!

let me know of issues or performance.

First step is to get it to work correctly, but that's the next step :)

if the runtime uses plain malloc/free, then first use LD_PRELOAD to see if those programs still work. This lets you know if the bug is in mimalloc or in the integration.

So in the mimalloc-sys crate, I do two things:

check that mimalloc is actually used: with the MI_OVERRIDE setting, mimalloc replaces the standard symbols, but still provides the mi_ symbols. This was great to be able to test that the library was properly linked, e.g., by calling mi_malloc+free and malloc+mi_free, and making sure that works. (When I wrote the tests the first time, they failed, because I didn't had MI_INTERPOSE enabled on macos, so ~~hurd~~ the system allocator was trying to free memory from mimalloc and vice-versa, which crashed).
c-ffi-test: for all functions, types, etc. exposed by the Rust bindings to a C API, tests are generated that check that the size, alignment, field offsets, etc. of types are the same, that the addresses of the rust function wrapper and the C functions is the same, that the functions take the same arguments, return types, etc. Essentially, that the ABI of the wrapper matches that of the library being wrapped. This is done by generating C code and Rust code, to compute these things at each side of the FFI, and then collecting the results and compare them.

if the runtime uses plain malloc/free, then first use LD_PRELOAD to see if those programs still work. This lets you know if the bug is in mimalloc or in the integration.

The Rust compiler uses "plain" malloc/free (the C and POSIX apis), but right now jemalloc is statically linked into the binary, overriding these symbols. I know LD_PRELOAD works if the malloc library is linked dynamically, but I'm not sure if it works in this situation.

By default, Rust binaries use the system allocator. I've tried DYLD_LIBRARY_INSERT=.. with mimalloc on some small binary and that just worked (I saw the statistics output), but the Rust compiler is a beast in code size, and number of tests and benchmarks, and other linked libraries, so it

for a language runtime, it is best to link statically, and have the runtime consistently use the mi_ prefixed API and never mix up pointers gotten through some other API. However, that means special care must be taken that pointers obtained through some C library do not get freed with the new Rust free (which uses mi_free). In particular, watch for uses of strdup, strndup, and realpath ! and use the mi_ prefixed versions instead. Same for posix_align etc. (this will get easier once I get to an update for issue #75 )

So I'm statically linking mimalloc into the binary, and hopefully correctly overriding all symbols. That's what we do for jemalloc and it works. So I would hope that this would override strdup and friends as well.

One thing we are doing is compiling LLVM and somehow linking it into rustc. We don't do anything special for jemalloc here, so I don't think we need to do anything special for mimalloc either.

Just a note, ensure the aligment and size arguments are in the right order for the aligned allocation functions in:

Those are wrapped here: https://github.com/gnzlbg/mimallocator/blob/master/mimalloc-sys/src/lib.rs#L49

We just provide the C and POSIX API. I suppose the ones mimalloc defines have the arguments in the standard orders for these. From looking at the code here, they appear to match: https://github.com/microsoft/mimalloc/blob/master/src/alloc-override.c#L138

That is, we are only using the overriden functions from mimalloc, we don't use any of the mi_... APIs directly in the compiler (yet).

Do you have a backtrace of the segfault? Was it using the debug version of mimalloc with -DMI_CHECK_FULL=ON enabled?

AFAICT it was using a release build, but I will enable full checks and try again.

Those are wrapped here: https://github.com/gnzlbg/mimallocator/blob/master/mimalloc-sys/src/lib.rs#L49

We just provide the C and POSIX API. I suppose the ones mimalloc defines have the arguments in the standard orders for these. From looking at the code here, they appear to match: https://github.com/microsoft/mimalloc/blob/master/src/alloc-override.c#L138

Mmm, I mean the mi_malloc_aligned and friends don't match with posix_memalign: the arguments are swapped:

int posix_memalign(void** p, size_t alignment, size_t size);
void* memalign(size_t alignment, size_t size);

versus

void* mi_malloc_aligned(size_t size, size_t alignment);

Be very careful when binding to Rust that it matches what Rust defined.

Mmm, I mean the mi_malloc_aligned and friends don't match with posix_memalign:

Ah ok. Yes, we don't use mi_... and friends, only malloc/posix_memalign, etc. and the APIs of those that we use appear correct.

I understand you want to link statically with mimalloc and override the standard malloc/free. If so, that will still be tricky general. Static + override:

works on Linux/BSD with weak symbols -- just link carefully to ensure mimalloc comes first
should work on macOS -- but currently there are issues (OSX debugging #50) but I am confident it will be fixed in the future as other allocators make this work too. More testing is needed here and I need to study how jemalloc does this, tcmalloc uses MacOS zones which I am currently testing with.
does not work on Windows with msvc! (needs dynamic loading)

Static but not overriding is the safest option but you need to control the runtime system enough to ensure no pointers are mixed -- but that may well be the case for rust.

For now, I would at least try your current approach on Linux first to ensure it all works before moving on to other platforms.

I forgot to mention that, to be able to run Rust compiler benchmarks, we only need to be able to link mimalloc on Linux.

Static but not overriding is the safest option but you need to control the runtime system enough to ensure no pointers are mixed -- but that may well be the case for rust.

I don't think this is an option. We link against LLVM, which uses the malloc APIs to allocate memory. I can't really assume that there aren't any places in the code where memory from one is passed to the other, which afterwards deallocates it. I hope this isn't the case, but making sure that it isn't would be a lot of work at this point (EDIT: the code interfacing with LLVM is compartmentalized in a single place, so while doable, it is still a lot of work to do this).

For now, I would at least try your current approach on Linux first to ensure it all works before moving on to other platforms.

That's the plan. Maybe I wasn't clear. Our current goal isn't to switch from jemalloc to mimalloc in the Rust toolchain, but rather to gather data about the performance of the Rust toolchain when mimalloc is its allocator. Performance is just one of the many data-points that would need to be considered in a discussion about doing a switch, but that discussion hasn't started yet. This is purely exploratory work.

If you want to try my PR branch on Linux, the only thing you have to do is clone it, and execute ./x.py build --stage 1. It should work out-of-the box.

Thanks for explaining :-) I thought that you were working on macOS somehow. Anyway, in that case there should not be segfaults so let's see if we can track down the cause. If can provide me with a test case, or a debug backtrace from the faulty build that will help. If I try your branch, will it cause the error for me investigate?

If I try your branch, will it cause the error for me investigate?

Yes. Briefly, the Rust toolchain bootstraps itself. So the first "stage" is stage0, and here, an older compiler is used to compile the toolchain, producing a stage0 compiler. This stage0 compiler that gets produced here is linked statically with mimalloc in my PR branch. Then, in stage1, this compiler is used to compile the toolchain again, producing a stage1 compiler. This is where the compiler using mimalloc gets exercised, and where the error happens.

The ./x.py script handles everything for you. To build a stage1 compiler, one can write ./x.py build --stage 1. Unless you want to modify the stage0 compiler, e.g., to link mimalloc differently, you probably don't want to re-build stage0 all the time. You can prevent that by doing ./x.py build --stage 1 --keep-stage 0. When compilation fails, you get an error message, providing the CLI invocation that failed to execute. So you can just copy-paste it into your terminal, and re-try, use gdb, etc. Please let me know if you have any questions or need any help. If you start doing this, and happen to have Discord or Zulip installed, you can join the Rust channels there (search for Rust discord and Rust zulip for the links). There are compiler channels in both, where you can ask any questions.

This is still on my todo list -- let me know of further issues but I will get to it soonish :-)

Daan (@daanx)

Daan (@daanx) let me know if you need help setting anything up or feel free to drop by Rust's discord or Zulip chat rooms.

Building rustc seems to work fine for me, am I doing something wrong?

git clone git@github.com:gnzlbg/rust.git rust-mimalloc
cd rust-mimalloc
git checkout mimalloc
./x.py build

Results in:

Build completed successfully in 0:37:10

If I do ./x.py build --stage 1 --keep-stage 0, I get the same.

The issue is fairly old and in the meantime various bugs were addressed. Perhaps it is just working now? :-) If so, please confirm as it would be nice if Rust is able to easily use mimalloc as an allocator. I read somewhere that Rust now uses no default allocator but instead let's the user opt-in to another allocator -- it would be great if mimalloc can be one of those options.

Steps from #77 (comment) give you 2 months old Rust and Mimalloc. There are no fixes included so it shouldn't work meaning something went wrong.

I read somewhere that Rust now uses no default allocator but instead let's the user opt-in to another

Rust compiler uses jemalloc but users can choose which allocator they will use in their binaries (defaults to system allocator).

Mateusz Mikuła (@mati865)

Mateusz Mikuła (@mati865) actually, the latest commit in that PR was never "try"ed (some other PR landed before that that made it unmergeable), so... maybe it worked ? I'll rebase and we should schedule a try-build.

Daan (@daanx)

Daan (@daanx) I don't know how to do that, but we should document how to do that: there is a tracking issue for documenting this here (rust-lang/rustc-dev-guide#443), and a solution for a different case which sadly I don't think will work here.

Daan (@daanx)

Daan (@daanx) what you can do is add the --on-fail=bash flag to the x.py invocation. Once something fails, it will create an interactive shell with the environment properly set up. In that shell you can then spawn a debugger to run the command that failed.

Daan (@daanx)

Daan (@daanx) if --on-fail=bash does not work, you can use --on-fail=env to print the environment that's used by x.py. And then set it up yourself before trying to reproduce by executing the command that failed.

Has there been any progress on this?

I would like to try this again with the latest dev (or dev-abandon) mimalloc branches. However, it would be great if I can control the mimalloc sources; as I understand it is now pulled from https://github.com/gnzlbg/mimallocator. What is the best way to change this so I can change the source or (cmake) compilation flags? Can it point to a local folder?

Try this patch:

diff --git a/src/rustc/Cargo.toml b/src/rustc/Cargo.toml
index 825e3655ac..7a4f81ce59 100644
--- a/src/rustc/Cargo.toml
+++ b/src/rustc/Cargo.toml
@@ -22,7 +22,7 @@ rustc_codegen_ssa = { path = "../librustc_codegen_ssa" }
 #features = ['unprefixed_malloc_on_supported_platforms']

 [dependencies.mimalloc-sys]
-version = '=0.1.6-dev'
+path="../../../mimallocator/mimalloc-sys"
 optional = true

 [features]

After something like:

git clone git@github.com:gnzlbg/rust.git
cd rust
git checkout mimalloc
cd ..
git clone git@github.com:gnzlbg/mimallocator.git
cd mimallocator/mimalloc-sys
rmdir mimalloc
git clone git@github.com:microsoft/mimalloc.git
cd mimalloc
git checkout dev

Remember to enable jemalloc in config.toml or using the configure script.

For what it's worth, I got Rust to compile with only these messages from mimalloc:

   Compiling rustc_resolve v0.0.0 (/sdb/src/rust/src/librustc_resolve)
   Compiling rustc_privacy v0.0.0 (/sdb/src/rust/src/librustc_privacy)
   Compiling rustc_plugin v0.0.0 (/sdb/src/rust/src/librustc_plugin/deprecated)
   Compiling rustc_codegen_ssa v0.0.0 (/sdb/src/rust/src/librustc_codegen_ssa)
   Compiling rustc_save_analysis v0.0.0 (/sdb/src/rust/src/librustc_save_analysis)
mimalloc: warning: possibly trying to free a pointer that does not point to a valid heap region: 0x0x40030000100
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: (yes, the previous pointer 0x0x40030000100 was valid after all)
mimalloc: warning: possibly trying to free a pointer that does not point to a valid heap region: 0x0x40040000100
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: (yes, the previous pointer 0x0x40040000100 was valid after all)
mimalloc: warning: possibly trying to free a pointer that does not point to a valid heap region: 0x0x40074400100
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: (yes, the previous pointer 0x0x40074400100 was valid after all)
   Compiling rustc-main v0.0.0 (/sdb/src/rust/src/rustc)
    Finished release [optimized] target(s) in 9m 48s
Copying stage1 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage1 codegen artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu, llvm)
   Compiling cc v1.0.35

Árni Dagur (@ArniDagur)

Thank you Árni Dagur (@ArniDagur), gnzlbg (@gnzlbg): with the above instructions I successfully built using the latest dev-abandon branch (on its way to become version 2.0 with dev to 1.3). However, when enabling the secure build it still fails! The error was:

mimalloc: warning: mprotect error: start: 0x0x4ab9e75f000, csize: 0x1000, err: 12

(actually, the real error occurred later with a panic message a thread could not start due to resource exhaustion)

Some careful reading shows errno 12 is ENOMEM, and the man page says that that might be caused by creating too many mappings. Well, of course, full secure mode creates guard pages between every mimalloc "page" thus creating many mappings, and I guessed the biggest compiled sources may exceed the default (64k mappings) so I increased it to 256k:

sudo sysctl -w vm.max_map_count=262144

and that was it! After that it built in secure mode with full debug checking. :-)
Great. I will add a better warning message and add documentation; and perhaps a secure mode that only uses guard pages around full segments instead of every mimalloc page.

Pfff, glad this got sorted out -- would love to see some benchmarks with rust.

add mprotect error when the mmap limit might be reached in secure mod…

I just opened a new test try with another crate: rust-lang/rust#81782. Let's see do we get benchmarks at some point.

Edit: It looks good at the moment!

And here are the results: https://perf.rust-lang.org/compare.html?start=f9435f4c92651d67d5dbaba13c5606c4c4fc1327&end=86f3a470dd7fa102be01dc6d84ecc6920f8af32a

Pretty good and only a few regressions. Props for mimalloc!

I think this issue can be closed now.

Daan (@daanx)

I tried also the MacOS rustc-build and there the override did not work as expected, statistics show zeroes.

heap stats:     peak      total      freed    current       unit      count  
    normal:       0 b        0 b        0 b        0 b        1 b              ok
      huge:       0 b        0 b        0 b        0 b        1 b              ok
     giant:       0 b        0 b        0 b        0 b        1 b              ok
     total:       0 b        0 b        0 b        0 b        1 b              ok
malloc req:       0 b        0 b        0 b        0 b        1 b              ok

  reserved:       0 b        0 b        0 b        0 b        1 b              ok
 committed:       0 b        0 b        0 b        0 b        1 b              ok
     reset:       0 b        0 b        0 b        0 b        1 b              ok
   touched:       0 b        0 b        0 b        0 b        1 b              ok
  segments:       0          0          0          0                           ok
-abandoned:       0          0          0          0                           ok
   -cached:       0          0          0          0                           ok
     pages:       0          0          0          0                           ok
-abandoned:       0          0          0          0                           ok
 -extended:       0   
 -noretire:       0   
     mmaps:       0   
   commits:       0   
   threads:       0          0          0          0                           ok
  searches:     0.0 avg
numa nodes:       1
   elapsed:      10.628 s
   process: user: 29.280 s, system: 0.574 s, faults: 679, rss: 154.8 mb

I did enable debugs and ran it with env MIMALLOC_SHOW_STATS=1 ./x.py build

Daan (@daanx) Anything more to try here?

Seems that MacOS support for rustc is difficult to achieve in practice. And jemalloc does not support it either. So this is a known issue.

Jemalloc for macOS was just broken and can be fixed via simple changes provided in rust-lang/rust#82423.

Daan (@daanx)

Daan (@daanx) said:

Static + override:
does not work on Windows with msvc! (needs dynamic loading)

Is this still true today? To be able to static link and override the libc symbols is still preferable.

Navigation Menu

Repository navigation

Segfault while building the Rust toolchain with a toolchain that uses mimalloc #77

34 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Segfault while building the Rust toolchain with a toolchain that uses mimalloc #77

Description

Activity

daanx commented on Jul 4, 2019

daanx commented on Jul 4, 2019

gnzlbg commented on Jul 4, 2019

daanx commented on Jul 4, 2019

gnzlbg commented on Jul 4, 2019

daanx commented on Jul 4, 2019

gnzlbg commented on Jul 4, 2019

daanx commented on Jul 4, 2019

gnzlbg commented on Jul 4, 2019

daanx commented on Jul 11, 2019

gnzlbg commented on Jul 24, 2019

ArniDagur commented on Sep 12, 2019

daanx commented on Sep 12, 2019

mati865 commented on Sep 12, 2019

gnzlbg commented on Sep 12, 2019

34 remaining items

gnzlbg commented on Sep 18, 2019

gnzlbg commented on Sep 18, 2019

gnzlbg commented on Sep 18, 2019

gnzlbg commented on Dec 4, 2019

daanx commented on Jan 3, 2020

ArniDagur commented on Jan 3, 2020

ArniDagur commented on Jan 3, 2020

daanx commented on Jan 9, 2020

jq-rs commented on Feb 5, 2021

jq-rs commented on Feb 6, 2021

jq-rs commented on Feb 6, 2021

jq-rs commented on Feb 8, 2021

jq-rs commented on Feb 23, 2021

fee1-dead commented on Dec 26, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions