Installing docs with x.py install is very slow.

On today's git (0cd459f) x.py install seems to get stuck if you are building docs.

In config.toml:

[install]
prefix = "build/ykrustc-stage2-latest"
sysconfdir = "etc"

x.py install will get stuck at:

    Finished release [optimized] target(s) in 8.38s
Dist rust-docs-1.51.0-dev-x86_64-unknown-linux-gnu
        finished in 17.222 seconds
Install docs stage2 (Some(TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }))
install: creating uninstall script at /home/vext01/source/rust/build/tmp/empty_dir/build/ykrustc-stage2-latest/lib/rustlib/uninstall.sh
install: installing component 'rust-docs'

The underlying command is: sh /home/vext01/source/rust/build/tmp/tarball/rust-docs/x86_64-unknown-linux-gnu/rust-docs-1.51.0-dev-x86_64-unknown-linux-gnu/install.sh --prefix=/home/vext01/source/rust/build/ykrustc-stage2-latest --sysconfdir=build/ykrustc-stage2-latest/etc --datadir=build/ykrustc-stage2-latest/share --docdir=build/ykrustc-stage2-latest/share/doc/rust --bindir=build/ykrustc-stage2-latest/bin --libdir=build/ykrustc-stage2-latest/lib --mandir=build/ykrustc-stage2-latest/share/man --disable-ldconfig

Adding this to the config seems to work around the hang:

[build]
docs = false

Thanks!

How long did you let the command run before killing it? I remember installing the docs taking a long time last time I did it.

I left it for half an hour or so IIRC. I'll run it for longer and see if it goes away.

How long do you think it should take to install on a modern SSD?

I don't remember exactly how long it takes, but it doesn't depend on the speed of the disk. install.sh is just really inefficient. You can see if it's actually hanging by checking the contents of lib/rustlib/install.log this command while it's installing:

tail -f build/ykrustc-stage2-latest/lib/rustlib/install.log

The log file is empty, but I can see new files appearing on the disk (using find and diff) so progress must be occurring. Slowly.

It's been about 40 minutes now. It seems to be adding files to build/tmp/empty_dir/ is that right?

Build completed successfully in 0:57:44

So it's just very slow.

@Mark-Simulacrum

I took a look into this. The main issue is that the generated install.sh is very inefficient; it takes about 5 minutes to copy 2500 files (there are more than 2500, but I got bored and cancelled it). @pietroalbini @Mark-Simulacrum what do you think about rewriting this inner loop in Rust so it's not launching so many external processes?

https://github.com/rust-lang/rust-installer/blob/5254dbfd25d5284728ab624dca1969d61427a0db/install-template.sh#L548

This issue is compounded on macOS, where it seems to be even more inefficient to launch lots of processes from a shell. Sorry I don't have timings, but it takes tens of minutes at least for this step [edit: over an hour]; long enough that I would expect most users who are trying to build our software from source in macOS (it's Zcash, which installs a pinned version of Rust in order to build) are likely to think it's hanging and give up.

I think the problem may largely be that the manifest.in generated for rust-docs component (at least) is a file: list instead of using the directory: command; directory commands should be much more efficient. This change may be a recent(ish) regression; I suspect that fixing that is likely the right solution here.

what do you think about rewriting this inner loop in Rust so it's not launching so many external processes?

Oops I forgot to link back here: we discussed this briefly on Zulip and it was difficult to do in a cross-platform way.

We solved our problem for Zcash by just not installing rust-docs (passing --without=rust-docs to install.sh). This reduced our total build time on macOS from 95 minutes to 27 minutes! Obviously it doesn't solve the underlying issue.

rust-installer/install-template.sh: improve efficiency, step 1.

Auto merge of #145809 - he32:installer-perf-fix-1, r=Mark-Simulacrum

Auto merge of rust-lang#145809 - he32:installer-perf-fix-1, r=Mark-Si…

Auto merge of #145809 - he32:installer-perf-fix-1, r=Mark-Simulacrum

Auto merge of rust-lang#145809 - he32:installer-perf-fix-1, r=Mark-Si…

Auto merge of #145809 - he32:installer-perf-fix-1, r=Mark-Simulacrum

Navigation Menu

Repository navigation

Installing docs with `x.py install` is very slow. #80684

12 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Installing docs with x.py install is very slow. #80684

Description

Activity

emilyalbini commented on Jan 4, 2021

vext01 commented on Jan 4, 2021

emilyalbini commented on Jan 4, 2021

vext01 commented on Jan 4, 2021

vext01 commented on Jan 4, 2021

vext01 commented on Jan 4, 2021

jyn514 commented on Feb 14, 2021

daira commented on Mar 31, 2021

Mark-Simulacrum commented on Mar 31, 2021

jyn514 commented on Mar 31, 2021

daira commented on Apr 1, 2021

12 remaining items

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Installing docs with `x.py install` is very slow. #80684