OOM installing rustc-dev with 1GB of memory

Problem

I tried installing a rust toolchain in a VM I assigned 1GB of ram. (100MB used by the minimal debian install) When installing rustc-dev it got an OOM however.

Steps

Create a VM
Assign 1GB of ram
Install a minimal debian system in the VM. When running it only uses 100MB of ram.
Attempt to install a rust toolchain with the rustc-dev component in the VM.
OOM

Possible Solution(s)

I thought rustup 1.24 enabled installation on systems with little ram. Is this a regression? Would it be possible to stream the component rather than attempt to load it all at once in memory?

Notes

For now I will be increasing the assigned amount of memory, but IMO it should be possible to install on systems with this little ram, hence this issue.

Rustup version

rustup 1.25.1 (bb60b1e89 2022-07-12)

Installed toolchains

Default host: x86_64-unknown-linux-gnu
rustup home:  /home/bjorn/.rustup

no active toolchain

I attempted to install nightly-2022-12-13-x86_64-unknown-linux-gnu.

Assigned 2GB and now rustup maxed at 1.55GB.

Sounds like the general overheads on that platform are higher than we expect. If you reduce RUSTUP_UNPACK_RAM does that help? You can set it to something tiny (e.g. 10 bytes) to get rustup to tell you the minimum it thinks it needs for your system. You can also set it to something huge (10s of gigabytes) to see if Rustup is mis-detecting the RAM available in your system.

Currently we estimate 200MB for rustup, IO logic, etc. and then we use detected max ram (https://docs.rs/effective-limits/latest/effective_limits/fn.memory_limit.html) to limit our buffer use to leave at least that estimate free.

@bjorn3

@bjorn3 ok, so 1GB does not work now. Have you tried @kinnison 's suggestions - in particular setting RUSTUP_UNPACK_RAM very low?

I don't recall any changes to the memory buffer management stuff recently, so not anticipating regressions. Note that the cached component dist files are not memory regulated - but we do only only one in memory cache at a time, and its pull-parsed, so I wouldn't expect a huge footprint.

Perhaps running under memgrind or massif or some such would help too.

I am currently trying to install rustup on a t4g.nano instance running Ubuntu 22.04 and the unpacking is running out of memory.

info: profile set to 'minimal'
info: setting default host triple to aarch64-unknown-linux-gnu
info: syncing channel updates for 'stable-aarch64-unknown-linux-gnu'
info: latest update on 2023-10-05, rust version 1.73.0 (cc66ad468 2023-10-03)
info: downloading component 'cargo'
info: downloading component 'rust-std'
info: downloading component 'rustc'
 76.5 MiB /  76.5 MiB (100 %)  72.4 MiB/s in  1s ETA:  0s
info: installing component 'cargo'
warning: Ignoring RUSTUP_UNPACK_RAM (10) less than minimum of 33554432.
info: installing component 'rust-std'
warning: Ignoring RUSTUP_UNPACK_RAM (10) less than minimum of 33554432.
Killed

free -h looks like this:

               total        used        free      shared  buff/cache   available
Mem:           419Mi       126Mi       252Mi       0.0Ki        40Mi       274Mi
Swap:             0B          0B          0B

@saethlin

@saethlin Right. Per the linked closed issue installing the Rust tool chain requires at least 1 GB.

I don't think that will do anything. Allocation failures abort, as far as I can tell re-landing the patch that makes OOM call the panic handler is stuck in limbo with a bunch of concerns about the idea: rust-lang/rust#112331

fix(diskio): fall back to single-threaded unpacking when `ram_budget`…

Navigation Menu

Repository navigation

OOM installing rustc-dev with 1GB of memory #3125

Problem

Steps

Possible Solution(s)

Notes

Rustup version

Installed toolchains

13 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

OOM installing rustc-dev with 1GB of memory #3125

Description

Problem

Steps

Possible Solution(s)

Notes

Rustup version

Installed toolchains

Activity

bjorn3 commented on Dec 25, 2022

kinnison commented on Jan 12, 2023

guest271314 commented on Feb 19, 2023

guest271314 commented on Feb 19, 2023

bjorn3 commented on Feb 19, 2023

guest271314 commented on Feb 19, 2023

guest271314 commented on Feb 19, 2023

bjorn3 commented on Feb 19, 2023

guest271314 commented on Feb 19, 2023

bjorn3 commented on Feb 19, 2023

rbtcollins commented on Feb 22, 2023

rbtcollins commented on Feb 22, 2023

saethlin commented on Nov 13, 2023

guest271314 commented on Nov 13, 2023

13 remaining items

saethlin commented on Oct 8, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions