Repository navigation
rustdoc: adjust static file names for better cache configuration #98413
Description
rustdoc has a number of static files that should really be long-cached by the browser for best loading performance: the fonts, the CSS, storage.js, main.js, and so on. We should add a hash to their names so that services can be more confident in setting long cache headers for them.
Right now we categorize things into Unversioned, ToolchainSpecific, and InvocationSpecific:
rust/src/librustdoc/html/render/write_shared.rs
Lines 44 to 57 in 10f4ce3
Unversioned is used just for the font files. ToolchainSpecific is used for the CSS, the images, and most of the JS. InvocationSpecific is used for search-indexN.NN.N.js, source-filesN.NN.N.js, cratesN.NN.N.js, the JS that contains the list of implementors on trait pages, and the JS that contains the list of additional sidebar items (siblings in a module).
Unversioned gets no infix. ToolchainSpecific gets a version suffix, like main1.63.0.js (from main.js). InvocationSpecific gets the same version suffix.
Unversioned and ToolchainSpecific files should be infinitely cacheable. Right now, that's not the case for ToolchainSpecific, because multiple toolchains have the same version infix. For instance, every nightly build right now creates a main1.63.0.js, but it's potentially different each night. That means https://doc.rust-lang.org/nightly/main1.63.0.js potentially changes every night, and can't be long-cached. Since docs.rs uses the nightly toolchain, the main1.63.0.js it produces for a crate today may be different than the one it produces for a crate it builds tomorrow.
docs.rs has special code to recognize the ToolchainSpecific files and rename them to contain a date and a hash, like https://docs.rs/main-20220517-1.63.0-nightly-4c5f6e627.js. But doc.rust-lang.org doesn't have that code, and as a result is less able to cache things that should be cached. And anyone who self-hosts docs is on their own.
I propose that we change our file naming scheme. All Unversioned and ToolchainSpecific files should be emitted to a subdirectory s/<hash>/, where <hash> is calculated over the contents of all of those files together. This makes it easy to configure a web server to set Cache-Control headers for everything under that subdirectory.
Advantage: this makes calculating URLs for such resources easy, especially when the calculation is done in JS. Disadvantage: if one file changes, the whole hash changes, potentially requiring the user to load more files when navigating between crates generated with different rustdoc versions.
Alternately, we could add a hash of each individual file to that file's name. That makes calculating URLs harder, but means better reuse of cached data across different nightly versions.
/cc @rust-lang/rustdoc @rust-lang/docs-rs
Activity
jyn514 commentedon Jun 23, 2022
That's not quite right - docs.rs passes --resource-suffix to rustdoc to make it automatically include the suffix; the alternative would be parsing and rewriting thousands of html files at build time. It should be possible for doc.rust-lang.org to do the same pretty easily.
I'm not opposed to adding a default for --resource-suffix on nightly, but I also don't think it's a particularly pressing problem.
GuillaumeGomez commentedon Jun 23, 2022
I guess adding
-{VERSION}-{DATE}as--resource-suffixwould fix the issue.jsha commentedon Jun 26, 2022
Ah, thanks for the correction! I had taken the hex bytes in those filenames to be a hash, but I see now they are a commit id.
Also, for some reason I had assumed that self-built docs had the version suffix by default (e.g. 1.61.0), but I see now they don't. That means people self-hosting their docs can't set caching headers for static files.
My main priority here is that http://doc.rust-lang.org/std (i.e. stable) doesn't have caching headers for the JS and CSS, which adds some delay to page loads. To add those I'd like to make it things so that when content changes, filenames change (as suggested in rust-lang/simpleinfra#108 (comment)).
I think the straightforward fix is to add a default value for
--resource-suffixon all builds - nightly and stable. If we choose a default that meets the needs of docs.rs, would that allow us to deprecate the flag itself? I note that docs.rs uses<date>-<version>-<channel>-<commit hash>, which seems like more than needed. We could get by with just the commit hash, right? The original commit in docs.rs doesn't mention the reason for the current scheme.jyn514 commentedon Jun 26, 2022
Just the hash would make it a little harder to tell when the docs were published, but I guess we can find that out from our database records anyway. I don't think there's any special logic behind the current format other than being unambiguous.
Do note though that the hash will have to be longer if you omit the channel and date - docs.rs stores enough years of docs that hash collisions are a real risk (especially since git is still using SHA1).
jyn514 commentedon Jun 26, 2022
Well, they can, they just have to know about resource-suffix.
Mark-Simulacrum commentedon Jun 27, 2022
FWIW, it's not clear to me why we should be versioning these files along rustc's commit hash or similar: a hash of the file contents is relatively easy to compute and avoids any questions around needing git (e.g., when building from tarballs with patches applied) to prevent problems. It also gives us longer-lived caches on a more granular basis, and should work for all files that aren't user visible in links. (Even without resource suffix passed).
Is there a reason we're not doing that by default?
jsha commentedon Jun 27, 2022
We do some loading of resource files from JS:
rust/src/librustdoc/html/static/js/main.js
Lines 52 to 56 in 3b0d481
Still, the benefits of using hashes might be worth it.
jsha commentedon Aug 30, 2022
Thinking about it some more, I'm convinced hashes are the right approach. My goal is to enable faster page loads in general. One was is by turning on caching of static files on doc.rust-lang.org. But another nice improvement would be to improve cache hit rates on docs.rs. Right now it's unlikely that any two crates have the same URL for, e.g. normalize.css, because it will always be specific to the rustdoc version, e.g. https://docs.rs/normalize-20220709-1.64.0-nightly-6dba4ed21.css. But that file basically never changes.
If we used a hash as part of the URL, in theory docs.rs could stop setting --resource-suffix, and most page navigations would load a cached normalize.css. Even static files like main.js that change more frequently change less often than nightly and would have a better cache hit rate.
jyn514 commentedon Aug 30, 2022
👍 for not having to keep track of this in docs.rs - ideally we could get rid of --print=invocation-specific at the same time
jsha commentedon Sep 2, 2022
I have a branch in progress and would love some eyes on it just to check I'm going in the right direction:
https://github.com/rust-lang/rust/compare/master...jsha:rust:static-files?expand=1
https://rustdoc.crud.net/jsha/static-files/std/io/trait.Read.html
Basically everything static (Unversioned + InvocationSpecific) now gets written to static.files/, which should make it simpler for docs.rs to figure out what should be copied out. The InvocationSpecific files get a hash in the filename based on their contents. I can also collapse the two categories so the fonts (Unversioned) get the hash treatment as well.
Still to be done:
GuillaumeGomez commentedon Sep 2, 2022
Your approach seems a bit complex. I would have done it as follows: generate static files and their hash first, then save this value into a struct stored somewhere and then use it when generating the template files. Like that it wouldn't require too much changes and would still work out easily.
The downside with the hash approach is that it's tricky to know what the new files are since you need to look at the file date and it makes impossible to use the resource suffix. I'd say we lose in predictability what we gain in simplicity for the source code. Not sure if it's important or not though. Also, changing/removing cli options is a breaking change I think. So maybe we need a transition period in-between.
notriddle commentedon Sep 2, 2022
@GuillaumeGomez
The complexity comes from the dependency graph between them. The
main.jsscript is supposed to be able to lazily loadsearch.js, which means it needs to know somehow what the filename is. If that's embedded in the file itself, then the hash ofmain.jsdepends on the hash ofsearch.js.You can work around this by generating a separate "file manifest", either inlined into all of the HTML files or stored as another JS file, but that causes its own complexity (at least, it adds more boilerplate to rustdoc's already heavy HTML output).
This is an unstable option anyway. A deprecation period might be a good idea, but it doesn't have to be very long.
GuillaumeGomez commentedon Sep 2, 2022
The order of files is known ahead of time so there shouldn't be too much complexity normally? But maybe I'm missing something obvious.
Also, do we know if there are users relying on the replaced cli options apart from docs.rs?
I guess it can be short if no ones uses this option indeed.
jyn514 commentedon Sep 2, 2022
Docs.rs uses this option. It's ok if rustdoc removes it, but we need advance notice (cc @rust-lang/docs-rs) and to know what we should replace it with.
jsha commentedon Sep 2, 2022
The
--emitoption, right? That should continue to work with my branch.To make sure I understand how it works, is this summary accurate?
The first time a new toolchain is installed, rustdoc is called on a dummy crate with
--emit=unversioned-shared-resources,toolchain-shared-resources, which gets the fonts and the static JS/CSS. These are stored somewhere such that they can be served under/and shared by all crates.For regular crate builds, rustdoc is called with
--emit=invocation-specific, to avoid emitting the fonts and static JS/CSS, which saves storage space. The resulting directory can be directly packaged up and won't contain unnecessary files.With the proposal in my branch, everything emitted by
--emit=unversioned-shared-resources,toolchain-shared-resourceswould be guaranteed to go understatic.files/. So we could either keep using the flags as they are, or deprecate them. If we deprecate them, docs.rs would rely onrm -r'ingstatic.files/in regular crate builds, and copyingstatic.files/on new toolchain installs. I don't have a strong opinion on which is better. Getting rid of flags is nice; at the same time, if the flag approach is working why mess with it. :-)This is effectively what we do here, though instead of "a struct", I use a series of statics. However, having a single top-level static with fields would indeed make it easier to make these values available to templates. I could give that a shot.
GuillaumeGomez commentedon Sep 2, 2022
I was talking about the
--resource-suffix. If you were not talking about this option, I completely misunderstood you. ^^'👍
jsha commentedon Sep 2, 2022
@GuillaumeGomez said:
Indeed, earlier in this thread I talked about potentially deprecating
--resource-suffix. @notriddle also mentioned--emit. Both are relevant, though after poking at it some more I have concluded my branch cannot completely deprecate--resource-suffix. The "dynamic" JS still (probably) needs it: sidebar-items, implementors, crates, and search-index.We could eventually talk about changing the naming scheme for our dynamic JS: instead of being named based on
--resource-suffix, the format of object provided by those files could be versioned independently, like we version the rustdoc JSON output today. But that's outside the scope of this issue IMO.jsha commentedon Sep 11, 2022
PR is up: #101702
Rollup merge of rust-lang#101702 - jsha:static-files2, r=notriddle,Gu…
Rollup merge of rust-lang#101702 - jsha:static-files2, r=notriddle,Gu…