diff --git a/README.md b/README.md index 31d366ce..38d663d4 100644 --- a/README.md +++ b/README.md @@ -53,6 +53,9 @@ speedscope is designed to ingest profiles from a variety of different profilers ](https://github.com/jlfwong/speedscope/wiki/Importing-from-async%E2%80%90profiler-(Java)) - [Importing from Java Flight Recorder](https://github.com/jlfwong/speedscope/wiki/Importing-from-Java-Flight-Recorder) - [Importing Java thread dumps (`jstack` / `Thread.print`)](docs/JAVA_THREAD_DUMPS.md) + - [Importing ZIP archives of Java thread-dump snapshots](docs/JAVA_THREAD_DUMP_ZIP.md) +- Android + - [Classic Android method traces (versions 1–3)](docs/IMPORT_FORMATS.md#classic-android-method-traces) - Erlang/Elixir - [eflambe](https://github.com/Stratus3D/eflambe) - Native code @@ -63,8 +66,9 @@ speedscope is designed to ingest profiles from a variety of different profilers - [Importing from custom sources](https://github.com/jlfwong/speedscope/wiki/Importing-from-custom-sources) Contributions to add support for additional formats are welcome! Open an issue in -[this fork](https://github.com/node-projects/speedscope/issues). The importer guides -above are maintained upstream and remain useful for the supported file formats. +[this fork](https://github.com/node-projects/speedscope/issues). Upstream importer +guides remain useful for their formats; [fork-specific import behavior](docs/IMPORT_FORMATS.md) +documents the Android, archive, pprof and background-import policies. # Usage Visit [the hosted app](https://node-projects.github.io/speedscope/), then browse for a @@ -74,8 +78,10 @@ local recording. See [offline use and update behavior](docs/OFFLINE.md). Repository builds can also produce a self-contained HTML document with `npm run build:single-file`; see [single-file builds and CSP](docs/SINGLE_FILE.md). -Large JSON and trace captures import in the background with progress and a -**Cancel import** button. This improves responsiveness; very large captures can +Recordings of at least 1 MiB use a background importer when workers are available, +including perf, folded stacks, pprof, JFR, Android traces and renamed captures. +Progress and **Cancel import** keep the app responsive; generic formats still +materialize their input and profile model. Very large captures can still exceed browser memory limits. Search offers Contains, Whole word and Exact matching for function names and source paths. Graph-search navigation retains at most 10,000 results and clearly marks larger diff --git a/docs/IMPORT_FORMATS.md b/docs/IMPORT_FORMATS.md new file mode 100644 index 00000000..d97b3847 --- /dev/null +++ b/docs/IMPORT_FORMATS.md @@ -0,0 +1,99 @@ +# Import formats and background processing + +The [README](../README.md#supported-file-formats) lists supported sources. +This guide describes additional fork-specific behavior and its limits. + +## Classic Android method traces + +Open the classic concatenated key/data format emitted by Android method tracing: +a text key section followed by a little-endian `SLOW` binary section. Versions +1, 2 and 3 are supported, including version-3 record padding. The file can be +named `.trace` or renamed; detection uses the format signature. + +The importer preserves method signatures, thread identities and available Java +source locations. Clock metadata is required. `thread-cpu` and `wall` clocks +remain distinct; version-3 `dual` input creates separate labeled CPU and wall +profiles for each thread. Version-1 `global` means wall time. All values use +microseconds. CPU clocks are local to their thread; wall offsets remain relative +to the trace origin, so leading unobserved time can appear as idle weight. +The absolute date in the binary header is not added to those offsets. + +Initial method exits reconstruct calls already active at trace start. Calls +still open at EOF close at that thread's last observed event. These are clipped +observation windows and do not invent work before or after the recording. + +Fixed records are scanned without retaining a second parsed-record array. +Temporary state includes key dictionaries and per-thread stacks, alongside the +output builders. Key text is decoded line by line with a 16 MiB section limit +and 1 MiB line limit. The input buffer and final profiles still occupy memory. + +Raw binary-only/streaming traces, newer ART packet formats, Perfetto protobuf, +Android systrace HTML and wrapped timestamp recovery are unsupported. Unknown +clocks, partial records, invalid method actions and decreasing per-thread clocks +produce explicit errors. See the +[synthetic fixture, generation script and AOSP references](../sample/profiles/android-method-trace/README.md) +for the exact supported layouts and clock policy. + +## Java thread-dump archives + +Traditional HotSpot `jstack` / `Thread.print` text can be opened directly or as +an archive of captures from one JVM. Observed stacks contribute raw samples; +they do not measure elapsed time. The ZIP importer checks declared sizes and +actual expansion, feeds bounded UTF-8 lines into the parser and supports +cancellation without returning partial results. + +See [text snapshot semantics](JAVA_THREAD_DUMPS.md) and the +[ZIP format and resource policy](JAVA_THREAD_DUMP_ZIP.md). Archive support is +limited to classic ZIP32 STORE/DEFLATE; it is not a generic archive extractor. + +## pprof metric and source identity + +pprof's `default_sample_type` selects a sample metric by its string-table +identity. Protobuf integer objects are compared by value. If the preferred +metric is absent or cannot be found, the existing last-metric fallback applies. +For example, a recording can prefer allocation counts even when a later sample +column measures bytes; the displayed unit follows the chosen metric. + +Function IDs and location IDs retain their complete unsigned 64-bit identity. +They are not truncated to their low 32 bits or rounded through a JavaScript +number, so IDs that share a low word remain distinct. String-table positions +must fit safe nonnegative integers. Source lines must fit safe positive integers. + +Each location retains its own source line, falling back to the function's +definition line when needed. One location no longer overwrites the shared +function metadata used by another location. The current importer retains the +outermost entry of an inline-location chain; expanding all inlined functions +is outside this change. + +This does not add signed/differential flamegraphs, exact arithmetic for weights +larger than JavaScript's safe-integer range, or a metric chooser. Negative Go +heap delta weights remain a separate unsupported case. See the +[pprof schema used by the importer](../src/import/profile.proto). + +## Background import for large recordings + +Inputs of at least **1 MiB** use a Web Worker when available, regardless of +filename extension. This includes perf, folded stacks, pprof, JFR, Android +method traces, supported ZIP archives and renamed captures. Format detection, +generic parsing, demangling and profile snapshot preparation happen away from +the main UI thread. Cancelling the task terminates its worker. + +The worker first attempts the existing duration-trace streaming path, then +uses the generic importer when that format does not apply. Generic formats +still read/materialize their input and retain their final model; routing them +through a worker does not make them streaming or out-of-core. Snapshot transfer, +geometry and final view construction still have memory costs proportional to +the recording. [Large-trace guidance](LARGE_TRACE_PERFORMANCE.md) describes the +duration-trace limits and measurements separately. + +If workers are unavailable or blocked by a host's CSP, the app falls back to +the main-thread importer. DOM-dependent formats can also fall back after worker +detection. A transferred input buffer is returned before this fallback, so the +main thread does not attempt to parse an empty detached buffer. Synchronous +generic fallback parsing can block the UI until it returns; the asynchronous +ZIP importer also yields and checks cancellation on the main thread. + +Oversized supported ZIP files receive a four-byte signature preflight before +their complete body is read, both in the main importer and worker path. Other +formats retain their existing input/resource policies; the ZIP compressed-size +limit is not a blanket cap on every supported recording. diff --git a/docs/IMPORT_VALIDATION_2026-11-21.md b/docs/IMPORT_VALIDATION_2026-11-21.md new file mode 100644 index 00000000..f004c9c7 --- /dev/null +++ b/docs/IMPORT_VALIDATION_2026-11-21.md @@ -0,0 +1,41 @@ +# Import-format validation, 2026-11-21 + +This batch adds classic Android method traces, bounded Java thread-dump ZIP +archives, broader large-file worker routing, and pprof metadata corrections. +See [format coverage and limitations](IMPORT_FORMATS.md) and +[ZIP limits](JAVA_THREAD_DUMP_ZIP.md). It does not make generic formats streaming +or the final profile model bounded in size. + +## PLC regression + +The private PLC gzip recording imported in 3,004 ms in one Node run on this +Windows ARM64 machine. This is a regression observation, not a controlled speedup +comparison or a browser timing guarantee. The export retained four profiles, +17,510 frames and 1,039,656 call events. Its baseline hash remains +`522de77afbfc4dd94f1f30c3a9551979d12db506305ffef999db9c910ae8eddb`. +The capture and benchmark artifacts remain uncommitted. + +Browser checks used the built single-HTML app served over loopback HTTP with +external connections blocked by CSP. The PLC imported successfully, exposed +four profiles and rendered its call tree without console errors. The synthetic +Android dual-clock capture exposed CPU/wall totals of 20/40 microseconds for +the main thread and 7/20 microseconds for its worker. The two-snapshot Java ZIP +produced two raw samples and correctly disabled chronological Time Order. +These checks do not validate file:// loading or a real Android device capture. + +## Automated verification + +The final clean-cache Jest run passed 458 tests across 75 suites with 307 snapshots +using the official local Node 22.23.5 ARM64 runtime. System Node 24.19.0 ARM64 +passed all five CLI/release-helper Node tests. Previous native crashes of full +Jest on this system's Node 24 ARM64 remain a local runtime limitation; no tests +were weakened to hide them. The CI matrix covers Node 22 and 24 on Windows and +Linux x64. + +Type checking, linting, schema/backlog checks and Pages/standalone/single-HTML builds +passed. npm dry-pack includes the new format and ZIP resource-policy guides. +Targeted regressions cover ZIP expansion ceilings, cancellation, corrupt archives, +300 snapshots, Android record/clock validation, generic worker ownership and pprof +source identities. Pprof's existing fixture totals were reviewed after separating +source positions; frame cardinality can increase, while aggregate weights remain +unchanged. Signed negative pprof weights still require separate model support. diff --git a/docs/ISSUE_BACKLOG.md b/docs/ISSUE_BACKLOG.md index 928677dd..a12861ac 100644 --- a/docs/ISSUE_BACKLOG.md +++ b/docs/ISSUE_BACKLOG.md @@ -9,7 +9,7 @@ Source: [upstream issues](https://github.com/jlfwong/speedscope/issues). [Machin | Kind | Implemented | Already supported | Partial | Planned | Needs reproduction | Declined | Obsolete | Total | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | bug | 20 | 9 | 14 | 7 | 5 | 0 | 1 | 56 | -| feature | 27 | 9 | 7 | 23 | 0 | 4 | 0 | 70 | +| feature | 29 | 9 | 6 | 22 | 0 | 4 | 0 | 70 | | maintenance | 4 | 1 | 2 | 0 | 0 | 1 | 0 | 8 | **Implemented** means the reviewed scope has an implementation. **Supported** means it already works or the established format is supported. **Partial** never means fully fixed. **Needs reproduction** means insufficient current evidence. A useful proposal can remain planned because it needs substantial design, fixtures or validation. @@ -20,19 +20,19 @@ P1: correctness/large-profile failures. P2: valuable product work. P3: lower-pri | Issue | Kind | Useful? | Status | Priority | Work / evidence | | --- | --- | --- | --- | --- | --- | -| [#519](https://github.com/jlfwong/speedscope/issues/519) [Bug] Chrome crashes with "Out of Memory" error when viewing large profiles (~375MB) | bug | yes | partial | P1 | Final profile models, worker snapshots, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer. | +| [#519](https://github.com/jlfwong/speedscope/issues/519) [Bug] Chrome crashes with "Out of Memory" error when viewing large profiles (~375MB) | bug | yes | partial | P1 | Final profile models, worker snapshots, generic full-input parsing, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Broader streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer. | | [#501](https://github.com/jlfwong/speedscope/issues/501) Nothing Rendered (Firefox 134.0.2) | bug | yes | partial | P1 | Public 25MB trace imports (19 profiles); GL resize re-entry guarded and Call Tree handles initial WebGL failure. Firefox framebuffer allocation/driver behavior is not reproduced or proven fixed. | -| [#478](https://github.com/jlfwong/speedscope/issues/478) Cannot open large size JSON | bug | yes | partial | P1 | Final profile models, worker snapshots, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer. | +| [#478](https://github.com/jlfwong/speedscope/issues/478) Cannot open large size JSON | bug | yes | partial | P1 | Final profile models, worker snapshots, generic full-input parsing, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Broader streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer. | | [#393](https://github.com/jlfwong/speedscope/issues/393) perf data idle time | bug | yes | planned | P1 | Off-CPU attribution cannot be corrected using sample timestamps alone. Need sched-switch/period-aware recording support and tests separating blocked time from CPU work; do not label idle intervals as execution. | -| [#386](https://github.com/jlfwong/speedscope/issues/386) json file with more size is not loading | bug | yes | partial | P1 | Final profile models, worker snapshots, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer. | +| [#386](https://github.com/jlfwong/speedscope/issues/386) json file with more size is not loading | bug | yes | partial | P1 | Final profile models, worker snapshots, generic full-input parsing, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Broader streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer. | | [#370](https://github.com/jlfwong/speedscope/issues/370) Not working for Golang's memory profiling | bug | yes | planned | P1 | Negative Go heap delta weights require signed/differential representation. Silently dropping negative samples would produce incorrect allocation totals. | -| [#347](https://github.com/jlfwong/speedscope/issues/347) Chrome crashes when viewing large profiles | bug | yes | partial | P1 | Final profile models, worker snapshots, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer. | +| [#347](https://github.com/jlfwong/speedscope/issues/347) Chrome crashes when viewing large profiles | bug | yes | partial | P1 | Final profile models, worker snapshots, generic full-input parsing, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Broader streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer. | | [#508](https://github.com/jlfwong/speedscope/issues/508) RuntimeError: memory access out of bounds | bug | yes | partial | P2 | Native demangler exceptions preserve/cache original symbols instead of aborting loading. The public #507 compressed reproduction imports and completes demangling with warnings. This does not fix the native decoder itself. | | [#507](https://github.com/jlfwong/speedscope/issues/507) Index out of bounds when demangling rust symbol | bug | yes | partial | P2 | Native demangler exceptions preserve/cache original symbols instead of aborting loading. The public #507 compressed reproduction imports and completes demangling with warnings. This does not fix the native decoder itself. | | [#467](https://github.com/jlfwong/speedscope/issues/467) Incorrect rendering with long traces (thousands of seconds) | bug | conditional | partial | P2 | Compare the unavailable original upstream capture/zoom location before claiming every rendering symptom in that report fixed. Model/GPU storage still scales with capture size; pixel-regression coverage is tracked under #125. | | [#384](https://github.com/jlfwong/speedscope/issues/384) Some Instruments files don't load | bug | yes | planned | P2 | New Instruments .symbols binary storage needs a versioned decoder, not plist assumptions. Public archive was collected; no speculative reinterpretation of binary structures applied. | | [#446](https://github.com/jlfwong/speedscope/issues/446) souremaps arent being applied | bug | yes | partial | P3 | Automatic linked-map discovery/fetching and CORS handling remain unimplemented. Logs without source positions and ambiguous relative paths containing spaces still need producer-specific handling; the original Node/esbuild automatic-loading case is not claimed fixed. | -| [#404](https://github.com/jlfwong/speedscope/issues/404) Use fetch instead of uint8array-json-parser for parsing JSON balls larger than the string limit | bug | yes | partial | P3 | Final profile models, worker snapshots, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer. | +| [#404](https://github.com/jlfwong/speedscope/issues/404) Use fetch instead of uint8array-json-parser for parsing JSON balls larger than the string limit | bug | yes | partial | P3 | Final profile models, worker snapshots, generic full-input parsing, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Broader streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer. | | [#402](https://github.com/jlfwong/speedscope/issues/402) format: Chrome Trace with instant samples | bug | yes | partial | P3 | Trailing commas and B/E trace events work. Instant events and custom stack-bearing event fields require a defined point-event presentation; inventing positive duration would misrepresent them. | | [#381](https://github.com/jlfwong/speedscope/issues/381) Rendering bugs | bug | yes | partial | P3 | Public TimeProfiler trace imports quickly, but emits mismatched-end warnings. Epoch-rounding handling fixes the tested adjacency case; the remaining non-laminar intervals need an explicit overlap/track policy. | | [#361](https://github.com/jlfwong/speedscope/issues/361) Stackprof mode: Garbage collection frames should not be stacked on their previous frame | bug | yes | planned | P3 | Stackprof GC overlay versus separate samples has competing interpretations in the discussion. Preserve current totals until a configurable policy is specified and tested against Ruby versions. | @@ -42,7 +42,7 @@ P1: correctness/large-profile failures. P2: valuable product work. P3: lower-pri | [#226](https://github.com/jlfwong/speedscope/issues/226) Seeing addresses instead of names for Instruments profile | bug | yes | planned | P3 | Instruments 10 schema/symbolication compatibility requires macOS-era fixtures and decoder validation. Existing newer binary .symbols issue is separately tracked in #384. | | [#196](https://github.com/jlfwong/speedscope/issues/196) Importing from XCode 10.1 Instruments fails | bug | yes | planned | P3 | Instruments 10 schema/symbolication compatibility requires macOS-era fixtures and decoder validation. Existing newer binary .symbols issue is separately tracked in #384. | -## Remaining useful features (30) +## Remaining useful features (28) | Issue | Kind | Useful? | Status | Priority | Work / evidence | | --- | --- | --- | --- | --- | --- | @@ -69,12 +69,10 @@ P1: correctness/large-profile failures. P2: valuable product work. P3: lower-pri | [#270](https://github.com/jlfwong/speedscope/issues/270) Support import from React devtools performance tab | feature | yes | planned | P3 | React DevTools export is a versioned renderer-specific format; require representative exports and a committed mapping of render/commit durations to profiles. | | [#245](https://github.com/jlfwong/speedscope/issues/245) FeatureRequest: memory chart | feature | yes | planned | P3 | Memory/counter time series are non-stack tracks requiring sampling/interpolation, independent axes and synchronized timeline support. | | [#236](https://github.com/jlfwong/speedscope/issues/236) custom counters / graphs | feature | yes | planned | P3 | Memory/counter time series are non-stack tracks requiring sampling/interpolation, independent axes and synchronized timeline support. | -| [#173](https://github.com/jlfwong/speedscope/issues/173) Format Support: concatenated output from many calls to jstack | feature | yes | partial | P3 | Direct ZIP archive ingestion requested upstream remains unsupported; establish archive entry, decompression and total resource limits before implementing it. | | [#145](https://github.com/jlfwong/speedscope/issues/145) Import & display frame boundaries from Chrome | feature | yes | planned | P3 | Annotations and arbitrary per-block metadata require schema, serialization, selection and presentation design. Avoid putting metadata into frame identity because it can explode frame cardinality. | | [#118](https://github.com/jlfwong/speedscope/issues/118) Support non-call-stack event visualization | feature | yes | planned | P3 | Arbitrary asynchronous events need separate tracks/links because overlapping spans are not a synchronous stack. Coordinate this with OTEL and multitrack support. | | [#108](https://github.com/jlfwong/speedscope/issues/108) Add a single file (or single JS file at least) build option | feature | yes | partial | P3 | Strict policies forbidding dynamic/inline styles still need style-engine integration. File-protocol startup could not be tested by the browser tool (HTTP/S only). This is an optional CI artifact, not automatic npm/release inclusion or an embeddable instance API. | | [#97](https://github.com/jlfwong/speedscope/issues/97) Trace files from Instruments can only be dragged, not opened | feature | yes | planned | P3 | Instruments .trace is a directory package on macOS and cannot be read through a regular single-file input. Need directory/archive import UI and hierarchy-aware decoder support. | -| [#40](https://github.com/jlfwong/speedscope/issues/40) Format Support: Android trace file | feature | yes | planned | P3 | Android method tracing binary format needs versioned parsing and thread-clock semantics, distinct from Chrome trace JSON. No fixture-backed implementation available in this pass. | | [#16](https://github.com/jlfwong/speedscope/issues/16) Allow speedscope to be used as a library instead of a standalone application | feature | yes | planned | P3 | A supported embeddable instance API requires isolating global atoms, lifecycle/disposal and multiple viewers. CLI --serve and profileURL cover launch integrations; publishing internal classes now would freeze an unsuitable API. | ## Remaining maintenance (2) @@ -94,7 +92,7 @@ P1: correctness/large-profile failures. P2: valuable product work. P3: lower-pri | [#269](https://github.com/jlfwong/speedscope/issues/269) Table view doesn't update at 60fps during scrolling | bug | conditional | needs-reproduction | P3 | Original slow scrolling report references a 2013 Mac. Iterative traversals and bounded Call Tree improve scale; no evidence that the exact older scrolling problem is fixed. | | [#42](https://github.com/jlfwong/speedscope/issues/42) Low framerate in Chrome 66 on MBP | bug | conditional | needs-reproduction | P3 | Report targets Chrome 66 compositor behavior. Current supported browsers differ; need a current profile and rendering trace before applying historical GPU workarounds. | -## Implemented or already supported (70) +## Implemented or already supported (72) | Issue | Kind | Useful? | Status | Priority | Work / evidence | | --- | --- | --- | --- | --- | --- | @@ -122,7 +120,7 @@ P1: correctness/large-profile failures. P2: valuable product work. P3: lower-pri | [#444](https://github.com/jlfwong/speedscope/issues/444) Tooltip partially hidden under mouse | bug | yes | supported | none | PR #555 port adds OS-dependent tooltip cursor offsets. No new independent fix needed. | | [#441](https://github.com/jlfwong/speedscope/issues/441) when i use this, how can i get release when i change it? | maintenance | yes | implemented | none | Current project uses esbuild/tsx; portable prepack now produces dist/release and a standalone ZIP on Windows too. See README and CONTRIBUTING for dev/build paths. | | [#426](https://github.com/jlfwong/speedscope/issues/426) Maximum call stack size exceeded | bug | yes | implemented | none | GLCanvas ignores nested resize calls and clears the guard even after errors. Regression exercises recursive resize and subsequent recovery. | -| [#398](https://github.com/jlfwong/speedscope/issues/398) Need to support Java thread dump | feature | yes | implemented | none | Traditional HotSpot jstack/Thread.print text imports single or concatenated snapshots into aggregate per-thread raw-sample profiles. Lazy line iteration retains only the current stack and snapshot identity set beyond output builders. Eight regression tests cover ordering, source locations, identity, malformed input and deadlock repeats; browser import verified. See docs/JAVA_THREAD_DUMPS.md. | +| [#398](https://github.com/jlfwong/speedscope/issues/398) Need to support Java thread dump | feature | yes | implemented | none | Traditional HotSpot jstack/Thread.print text imports single or concatenated snapshots into aggregate per-thread raw-sample profiles. Lazy line iteration retains only the current stack and snapshot identity set beyond output builders. Eight regression tests cover ordering, source locations, identity, malformed input and deadlock repeats; browser import verified. See docs/JAVA_THREAD_DUMPS.md. Same-JVM snapshots can also be imported from bounded ZIP32 STORE/DEFLATE archives; see docs/JAVA_THREAD_DUMP_ZIP.md. | | [#389](https://github.com/jlfwong/speedscope/issues/389) node12 inspector data doesn't render idle time correctly | bug | yes | supported | none | PR #383 port separates CPU program samples from JS frames rather than extending the previous JavaScript leaf through idle/native work. | | [#380](https://github.com/jlfwong/speedscope/issues/380) Better IFrame and/or Jupyter support | bug | yes | implemented | none | Storage read/write failures no longer crash theme initialization or interaction in sandboxed iframes. Preferences remain in memory when localStorage is unavailable. | | [#378](https://github.com/jlfwong/speedscope/issues/378) Web flamegraph sometimes crashes with "startsWith of undefined" (rack-mini-profiler) | bug | yes | supported | none | Stackprof fallback profile name is already present from upstream #419 in this baseline. | @@ -140,6 +138,7 @@ P1: correctness/large-profile failures. P2: valuable product work. P3: lower-pri | [#193](https://github.com/jlfwong/speedscope/issues/193) `perf script --hide-call-graph` output is not accepted | bug | yes | implemented | none | perf script --hide-call-graph samples with inline IP, symbol and DSO now import as single-frame stacks, including consecutive padded headers and CRLF. Real callchains take precedence; existing timestamp/counter-period weighting remains unchanged. Five new regressions and the existing issue/perf suites passed (22 tests, 26 snapshots). | | [#189](https://github.com/jlfwong/speedscope/issues/189) Optimize fallback path when TextDecoder is not available | bug | yes | implemented | none | Legacy ASCII decoder builds bounded pieces and joins per chunk instead of flattening the entire growing string for every byte. Chunk-boundary regression added. | | [#185](https://github.com/jlfwong/speedscope/issues/185) Hide the time ordered view for profiles known to contain only aggregate information | feature | yes | implemented | none | Known aggregate importers and derived aggregates disable Time Order, including keyboard/profile-switch paths. Optional chronological/derived metadata survives export/reimport; legacy files retain their previous default. See the performance report for covered importers. | +| [#173](https://github.com/jlfwong/speedscope/issues/173) Format Support: concatenated output from many calls to jstack | feature | yes | implemented | none | Concatenated traditional jstack text and classic ZIP32 archives of same-JVM snapshots import into aggregate per-thread raw-sample profiles. ZIP STORE/DEFLATE input validates local/central metadata, CRC, paths and actual expansion; bounded UTF-8 lines feed the shared incremental parser. Limits are 32 MiB compressed, 512 entries, 16 MiB per entry and 64 MiB total. Large File preflight reads only the signature before enforcing the cap; decompression yields and supports cancellation. See docs/JAVA_THREAD_DUMP_ZIP.md. | | [#168](https://github.com/jlfwong/speedscope/issues/168) Doesn't open browser in Windows | bug | yes | supported | none | PR ports add --serve, a localhost binary profile endpoint, temporary subdirectories and correct platform file URLs. | | [#166](https://github.com/jlfwong/speedscope/issues/166) Add support for color scheme controls | feature | yes | supported | none | Current viewer has system/dark/light themes and symbol/file color assignment; further alternative palettes need a concrete requested scheme and contrast validation. | | [#157](https://github.com/jlfwong/speedscope/issues/157) Support trailing whitespace & floating point values with Brendan Gregg collapsed stack format | bug | yes | implemented | none | Folded weights accept decimals, exponents and trailing whitespace; reject negative/nonfinite values with a clear error. Regression preserves 21.75 total. | @@ -152,6 +151,7 @@ P1: correctness/large-profile failures. P2: valuable product work. P3: lower-pri | [#89](https://github.com/jlfwong/speedscope/issues/89) Display progress bar on export | feature | yes | implemented | none | Interactive exports serialize lazy call events into Blob chunks with progress, event-loop yields and cancellation. No full duplicate event tree or trace-sized JS string is built. Completed downloads revoke their object URLs. Native output bytes and source profiles remain in memory. | | [#62](https://github.com/jlfwong/speedscope/issues/62) Feature request: see the duration of a run in the UI | feature | yes | implemented | none | Active profile total duration/weight appears beside its name and in the Call Tree caption. | | [#58](https://github.com/jlfwong/speedscope/issues/58) Make www.speedscope.app work offline | feature | yes | implemented | none | Hosted HTTP builds precache an allowlisted app shell with bounded generations, safe waiting updates and no recording/runtime cache insertion. Six service-worker VM tests cover lifecycle, failure, scope and exclusions. Browser QA reopened the app with its server stopped, imported a local dump, and confirmed uncached profileURL fetch failure. Standalone and development builds do not register a worker. See docs/OFFLINE.md. | +| [#40](https://github.com/jlfwong/speedscope/issues/40) Format Support: Android trace file | feature | yes | implemented | none | Classic Android concatenated key + SLOW method traces (versions 1-3) import with explicit CPU/wall/dual-clock policy, per-thread stack reconstruction, source/signature identity and clipped observation windows. Fixed records are scanned without a parsed-record array; key section/line sizes are bounded. Independently authored fixtures and malformed-input coverage follow AOSP format references. See docs/IMPORT_FORMATS.md and sample/profiles/android-method-trace/README.md. | | [#39](https://github.com/jlfwong/speedscope/issues/39) Add support for crop | feature | yes | implemented | none | Crop to current time view creates a new rebased profile, clips nested calls at both boundaries and retains idle range duration. Original profiles remain unchanged. | | [#410](https://github.com/jlfwong/speedscope/issues/410) Link display to source code | feature | yes | implemented | P2 | Profile actions configures browser-local VS Code/Insiders or fixed-host HTTP(S) source links with path-prefix mapping and encoded file/line/column values. Call Tree and selected-stack source locations open only on explicit clicks; no source discovery or per-profile URL index. See docs/SOURCE_AND_LABELS.md. | | [#246](https://github.com/jlfwong/speedscope/issues/246) FeatureRequest: only "my code" | feature | yes | implemented | P2 | Profile actions now creates non-destructive aggregate frame filters with include/exclude literal rules for file paths, names or either. Exclusions win; empty stacks/idle are omitted and retained-stack weights/self reattribution are documented. Originals, rule history and derived flags survive export. Iterative grouped traversal yields and cancels; tests cover deep trees and UI cancellation/paste isolation. | diff --git a/docs/JAVA_THREAD_DUMPS.md b/docs/JAVA_THREAD_DUMPS.md index 2123e870..f3f4b585 100644 --- a/docs/JAVA_THREAD_DUMPS.md +++ b/docs/JAVA_THREAD_DUMPS.md @@ -3,7 +3,9 @@ Import traditional HotSpot text output from `jstack PID` or `jcmd PID Thread.print` by opening a `.jstack` file or a text file containing the `Full thread dump …` header. Concatenate captures from the same JVM to -compare observed stacks across repeated snapshots. Extract ZIP archives first. +compare observed stacks across repeated snapshots. You can also open a ZIP of +traditional text captures from the same JVM; see [archive limits and supported +ZIP formats](JAVA_THREAD_DUMP_ZIP.md). Each observed Java stack contributes one raw sample. The importer creates an aggregate profile per thread name and Java thread ID, falling back to `tid` @@ -23,8 +25,11 @@ The parser iterates lines and retains the current stack and current snapshot's thread identities, rather than collecting every line or observation. The text input and resulting profile model still occupy memory; this format does not use the duration-trace streaming importer. Prefer captures from one JVM per file. -ZIP input, structured virtual-thread JSON, native address stacks and lock/state -visualization are outside the supported format. +The ZIP importer instead feeds bounded UTF-8 lines into the same parser and +checks both declared and actual decompressed sizes. Structured virtual-thread +JSON, native address stacks and lock/state visualization remain outside the +supported format. Unsupported ZIP variants can be extracted before importing +their text captures. The checked-in [synthetic example](../sample/profiles/java-thread-dump/synthetic.jstack) contains repeated snapshots and source locations. Format references are diff --git a/docs/JAVA_THREAD_DUMP_ZIP.md b/docs/JAVA_THREAD_DUMP_ZIP.md new file mode 100644 index 00000000..619c49f5 --- /dev/null +++ b/docs/JAVA_THREAD_DUMP_ZIP.md @@ -0,0 +1,65 @@ +# Java thread-dump ZIP archives + +Open a ZIP containing traditional HotSpot `jstack` / `jcmd Thread.print` text +captures from **one JVM**. The same thread name and Java thread ID identify a +thread across snapshots; renamed threads remain separate. Each observed Java +stack contributes one raw sample, not measured elapsed time. Missing threads +and entries without Java stacks contribute no samples. + +Entries are processed in deterministic, case-sensitive filename order, not ZIP +insertion order or file modification time. For example, `1.jstack`, `10.jstack`, +then `2.jstack` is the lexical order. The result is aggregate, so this ordering +does not claim a timeline or attach elapsed durations to snapshot intervals. +Every `.jstack` entry must contain its own `Full thread dump` header near the +start. Each entry boundary closes the previous snapshot. + +The importer recognizes `.jstack`, `.txt`, `.log`, `.dump` and `.tdump` entries +with a `Full thread dump` header within the first 64 KiB. Other regular files +and ordinary notes are ignored as profile input, but still undergo bounded +decompression and checksum validation. Directories are skipped. Text is strict +UTF-8; UTF-16 dumps should be converted or imported as separate text files. + +## Resource policy + +| Resource | Maximum | +| --- | ---: | +| Compressed archive | 32 MiB | +| Archive entries, including directories | 512 | +| Decompressed bytes per entry | 16 MiB | +| Total decompressed bytes, including ignored files | 64 MiB | +| UTF-8 filename bytes | 512 | +| Path depth | 8 components | +| Decoded line length | 65,536 UTF-16 code units | + +Declared sizes are checked before decompression. Actual expanded bytes are +checked on every output chunk before decoding or parsing, so lying ZIP headers +cannot bypass the limits. The inflater emits chunks of at most 16 KiB; a limit +violation can allocate that one output chunk before it is rejected. It never +accumulates the whole expanded entry. The line decoder retains at most one +bounded line, while the parser retains the current stack, current snapshot's +thread identities and the resulting profile builders. The compressed input and +final profile model still occupy memory; this is not an out-of-core viewer. +The importer yields during work and supports cancellation without returning a +partially imported group. + +## Supported ZIP subset + +Single-disk ZIP32 archives using **STORE** or **DEFLATE** are supported, including +ordinary data descriptors. Local/central headers, entry extents, decompressed +sizes and CRC32 checksums must agree. Paths are validated without extracting +anything to disk. Absolute/traversal paths, duplicate or case-colliding names, +file/directory conflicts, symlinks, special files and non-normalized Unicode +paths are rejected. Names must be ASCII or explicitly UTF-8. + +Encrypted archives, ZIP64, split archives, self-extracting prefixes, archive +padding, alternate Unicode-path extra fields and other compression methods are +unsupported. Extract an unsupported archive yourself and open its text captures +individually or concatenate them into one `.jstack` file. + +The checked-in [synthetic archive](../sample/profiles/java-thread-dump/synthetic.zip) +contains two short snapshots of the same thread. This is an independently +implemented ZIP32 reader based on +[PKWARE's ZIP specification](https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT), +using the existing +[pako incremental inflater](https://github.com/nodeca/pako/blob/1.0.6/lib/inflate.js). +It does not use JSZip's whole-entry `async('string')` allocation path. diff --git a/docs/issue-backlog.json b/docs/issue-backlog.json index ddd8e802..ebb875a6 100644 --- a/docs/issue-backlog.json +++ b/docs/issue-backlog.json @@ -168,8 +168,8 @@ "status": "partial", "priority": "P1", "workstream": "Large profiles and responsiveness", - "evidence": "Cancellable streaming duration imports now frame one bounded event at a time for native JSON parsing, keeping byte/event/depth guards and format fallbacks. Three fresh-process PLC comparisons reduced median import time from 4336 to 2493 ms and median peak RSS from 613 to 579 MiB. Export equality verified; general out-of-core storage/rendering and original OOM cases remain open. Complete-event queues now reuse original begin events and retain compact end records; already chronological queues avoid sort workspace. Selected-stack details render at most 100 ancestors per page. Graph search now retains at most 10,000 navigation matches, 4,096 boolean frame results and 512 labels with at most 64 ranges each; truncation is explicit and matching continues outside the navigation window. Sandwich now uses fixed-height count/size geometry instead of per-frame size objects or a frame-index Map. Viewport arithmetic is constant time; repeated/adjacent/wrap selection lookup is cached, with one scan for unrelated selections. Resize/filter clamping is tested.", - "remaining": "Final profile models, worker snapshots, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer." + "evidence": "Cancellable streaming duration imports now frame one bounded event at a time for native JSON parsing, keeping byte/event/depth guards and format fallbacks. Three fresh-process PLC comparisons reduced median import time from 4336 to 2493 ms and median peak RSS from 613 to 579 MiB. Export equality verified; general out-of-core storage/rendering and original OOM cases remain open. Complete-event queues now reuse original begin events and retain compact end records; already chronological queues avoid sort workspace. Selected-stack details render at most 100 ancestors per page. Graph search now retains at most 10,000 navigation matches, 4,096 boolean frame results and 512 labels with at most 64 ranges each; truncation is explicit and matching continues outside the navigation window. Sandwich now uses fixed-height count/size geometry instead of per-frame size objects or a frame-index Map. Viewport arithmetic is constant time; repeated/adjacent/wrap selection lookup is cached, with one scan for unrelated selections. Resize/filter clamping is tested. Inputs of at least 1 MiB now use generic worker routing regardless of filename extension, covering perf/folded/pprof/JFR/Android/ZIP and renamed captures when workers are available. DOM-dependent/CSP-blocked fallback can still run on the main thread; generic imports are not made streaming by this routing.", + "remaining": "Final profile models, worker snapshots, generic full-input parsing, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Broader streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer." }, { "id": 518, @@ -384,8 +384,8 @@ "status": "partial", "priority": "P1", "workstream": "Large profiles and responsiveness", - "evidence": "Cancellable streaming duration imports now frame one bounded event at a time for native JSON parsing, keeping byte/event/depth guards and format fallbacks. Three fresh-process PLC comparisons reduced median import time from 4336 to 2493 ms and median peak RSS from 613 to 579 MiB. Export equality verified; general out-of-core storage/rendering and original OOM cases remain open. Complete-event queues now reuse original begin events and retain compact end records; already chronological queues avoid sort workspace. Selected-stack details render at most 100 ancestors per page. Graph search now retains at most 10,000 navigation matches, 4,096 boolean frame results and 512 labels with at most 64 ranges each; truncation is explicit and matching continues outside the navigation window. Sandwich now uses fixed-height count/size geometry instead of per-frame size objects or a frame-index Map. Viewport arithmetic is constant time; repeated/adjacent/wrap selection lookup is cached, with one scan for unrelated selections. Resize/filter clamping is tested.", - "remaining": "Final profile models, worker snapshots, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer." + "evidence": "Cancellable streaming duration imports now frame one bounded event at a time for native JSON parsing, keeping byte/event/depth guards and format fallbacks. Three fresh-process PLC comparisons reduced median import time from 4336 to 2493 ms and median peak RSS from 613 to 579 MiB. Export equality verified; general out-of-core storage/rendering and original OOM cases remain open. Complete-event queues now reuse original begin events and retain compact end records; already chronological queues avoid sort workspace. Selected-stack details render at most 100 ancestors per page. Graph search now retains at most 10,000 navigation matches, 4,096 boolean frame results and 512 labels with at most 64 ranges each; truncation is explicit and matching continues outside the navigation window. Sandwich now uses fixed-height count/size geometry instead of per-frame size objects or a frame-index Map. Viewport arithmetic is constant time; repeated/adjacent/wrap selection lookup is cached, with one scan for unrelated selections. Resize/filter clamping is tested. Inputs of at least 1 MiB now use generic worker routing regardless of filename extension, covering perf/folded/pprof/JFR/Android/ZIP and renamed captures when workers are available. DOM-dependent/CSP-blocked fallback can still run on the main thread; generic imports are not made streaming by this routing.", + "remaining": "Final profile models, worker snapshots, generic full-input parsing, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Broader streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer." }, { "id": 477, @@ -624,8 +624,8 @@ "status": "partial", "priority": "P3", "workstream": "Large profiles and responsiveness", - "evidence": "Cancellable streaming duration imports now frame one bounded event at a time for native JSON parsing, keeping byte/event/depth guards and format fallbacks. Three fresh-process PLC comparisons reduced median import time from 4336 to 2493 ms and median peak RSS from 613 to 579 MiB. Export equality verified; general out-of-core storage/rendering and original OOM cases remain open. Complete-event queues now reuse original begin events and retain compact end records; already chronological queues avoid sort workspace. Selected-stack details render at most 100 ancestors per page. Graph search now retains at most 10,000 navigation matches, 4,096 boolean frame results and 512 labels with at most 64 ranges each; truncation is explicit and matching continues outside the navigation window. Sandwich now uses fixed-height count/size geometry instead of per-frame size objects or a frame-index Map. Viewport arithmetic is constant time; repeated/adjacent/wrap selection lookup is cached, with one scan for unrelated selections. Resize/filter clamping is tested.", - "remaining": "Final profile models, worker snapshots, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer." + "evidence": "Cancellable streaming duration imports now frame one bounded event at a time for native JSON parsing, keeping byte/event/depth guards and format fallbacks. Three fresh-process PLC comparisons reduced median import time from 4336 to 2493 ms and median peak RSS from 613 to 579 MiB. Export equality verified; general out-of-core storage/rendering and original OOM cases remain open. Complete-event queues now reuse original begin events and retain compact end records; already chronological queues avoid sort workspace. Selected-stack details render at most 100 ancestors per page. Graph search now retains at most 10,000 navigation matches, 4,096 boolean frame results and 512 labels with at most 64 ranges each; truncation is explicit and matching continues outside the navigation window. Sandwich now uses fixed-height count/size geometry instead of per-frame size objects or a frame-index Map. Viewport arithmetic is constant time; repeated/adjacent/wrap selection lookup is cached, with one scan for unrelated selections. Resize/filter clamping is tested. Inputs of at least 1 MiB now use generic worker routing regardless of filename extension, covering perf/folded/pprof/JFR/Android/ZIP and renamed captures when workers are available. DOM-dependent/CSP-blocked fallback can still run on the main thread; generic imports are not made streaming by this routing.", + "remaining": "Final profile models, worker snapshots, generic full-input parsing, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Broader streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer." }, { "id": 402, @@ -648,7 +648,7 @@ "status": "implemented", "priority": "none", "workstream": "Import formats and correctness", - "evidence": "Traditional HotSpot jstack/Thread.print text imports single or concatenated snapshots into aggregate per-thread raw-sample profiles. Lazy line iteration retains only the current stack and snapshot identity set beyond output builders. Eight regression tests cover ordering, source locations, identity, malformed input and deadlock repeats; browser import verified. See docs/JAVA_THREAD_DUMPS.md.", + "evidence": "Traditional HotSpot jstack/Thread.print text imports single or concatenated snapshots into aggregate per-thread raw-sample profiles. Lazy line iteration retains only the current stack and snapshot identity set beyond output builders. Eight regression tests cover ordering, source locations, identity, malformed input and deadlock repeats; browser import verified. See docs/JAVA_THREAD_DUMPS.md. Same-JVM snapshots can also be imported from bounded ZIP32 STORE/DEFLATE archives; see docs/JAVA_THREAD_DUMP_ZIP.md.", "remaining": "No additional work selected for traditional text thread dumps. Structured virtual-thread JSON and lock/state visualization remain outside this format." }, { @@ -696,8 +696,8 @@ "status": "partial", "priority": "P1", "workstream": "Large profiles and responsiveness", - "evidence": "Cancellable streaming duration imports now frame one bounded event at a time for native JSON parsing, keeping byte/event/depth guards and format fallbacks. Three fresh-process PLC comparisons reduced median import time from 4336 to 2493 ms and median peak RSS from 613 to 579 MiB. Export equality verified; general out-of-core storage/rendering and original OOM cases remain open. Complete-event queues now reuse original begin events and retain compact end records; already chronological queues avoid sort workspace. Selected-stack details render at most 100 ancestors per page. Graph search now retains at most 10,000 navigation matches, 4,096 boolean frame results and 512 labels with at most 64 ranges each; truncation is explicit and matching continues outside the navigation window. Sandwich now uses fixed-height count/size geometry instead of per-frame size objects or a frame-index Map. Viewport arithmetic is constant time; repeated/adjacent/wrap selection lookup is cached, with one scan for unrelated selections. Resize/filter clamping is tested.", - "remaining": "Final profile models, worker snapshots, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer." + "evidence": "Cancellable streaming duration imports now frame one bounded event at a time for native JSON parsing, keeping byte/event/depth guards and format fallbacks. Three fresh-process PLC comparisons reduced median import time from 4336 to 2493 ms and median peak RSS from 613 to 579 MiB. Export equality verified; general out-of-core storage/rendering and original OOM cases remain open. Complete-event queues now reuse original begin events and retain compact end records; already chronological queues avoid sort workspace. Selected-stack details render at most 100 ancestors per page. Graph search now retains at most 10,000 navigation matches, 4,096 boolean frame results and 512 labels with at most 64 ranges each; truncation is explicit and matching continues outside the navigation window. Sandwich now uses fixed-height count/size geometry instead of per-frame size objects or a frame-index Map. Viewport arithmetic is constant time; repeated/adjacent/wrap selection lookup is cached, with one scan for unrelated selections. Resize/filter clamping is tested. Inputs of at least 1 MiB now use generic worker routing regardless of filename extension, covering perf/folded/pprof/JFR/Android/ZIP and renamed captures when workers are available. DOM-dependent/CSP-blocked fallback can still run on the main thread; generic imports are not made streaming by this routing.", + "remaining": "Final profile models, worker snapshots, generic full-input parsing, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Broader streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer." }, { "id": 384, @@ -912,8 +912,8 @@ "status": "partial", "priority": "P1", "workstream": "Large profiles and responsiveness", - "evidence": "Cancellable streaming duration imports now frame one bounded event at a time for native JSON parsing, keeping byte/event/depth guards and format fallbacks. Three fresh-process PLC comparisons reduced median import time from 4336 to 2493 ms and median peak RSS from 613 to 579 MiB. Export equality verified; general out-of-core storage/rendering and original OOM cases remain open. Complete-event queues now reuse original begin events and retain compact end records; already chronological queues avoid sort workspace. Selected-stack details render at most 100 ancestors per page. Graph search now retains at most 10,000 navigation matches, 4,096 boolean frame results and 512 labels with at most 64 ranges each; truncation is explicit and matching continues outside the navigation window. Sandwich now uses fixed-height count/size geometry instead of per-frame size objects or a frame-index Map. Viewport arithmetic is constant time; repeated/adjacent/wrap selection lookup is cached, with one scan for unrelated selections. Resize/filter clamping is tested.", - "remaining": "Final profile models, worker snapshots, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer." + "evidence": "Cancellable streaming duration imports now frame one bounded event at a time for native JSON parsing, keeping byte/event/depth guards and format fallbacks. Three fresh-process PLC comparisons reduced median import time from 4336 to 2493 ms and median peak RSS from 613 to 579 MiB. Export equality verified; general out-of-core storage/rendering and original OOM cases remain open. Complete-event queues now reuse original begin events and retain compact end records; already chronological queues avoid sort workspace. Selected-stack details render at most 100 ancestors per page. Graph search now retains at most 10,000 navigation matches, 4,096 boolean frame results and 512 labels with at most 64 ranges each; truncation is explicit and matching continues outside the navigation window. Sandwich now uses fixed-height count/size geometry instead of per-frame size objects or a frame-index Map. Viewport arithmetic is constant time; repeated/adjacent/wrap selection lookup is cached, with one scan for unrelated selections. Resize/filter clamping is tested. Inputs of at least 1 MiB now use generic worker routing regardless of filename extension, covering perf/folded/pprof/JFR/Android/ZIP and renamed captures when workers are available. DOM-dependent/CSP-blocked fallback can still run on the main thread; generic imports are not made streaming by this routing.", + "remaining": "Final profile models, worker snapshots, generic full-input parsing, sorting and full frame arrays, graph/color geometry and browser scroll-height limits still scale with the capture. Broader streaming coverage and original OOM reproductions remain incomplete; this is not an out-of-core viewer." }, { "id": 346, @@ -1293,11 +1293,11 @@ "url": "https://github.com/jlfwong/speedscope/issues/173", "kind": "feature", "usefulness": "yes", - "status": "partial", - "priority": "P3", + "status": "implemented", + "priority": "none", "workstream": "Import formats and correctness", - "evidence": "Concatenated traditional jstack text is implemented with one raw sample per observed Java stack, aggregate per-thread profiles, source locations and malformed-input checks. ZIP archives must be extracted before import. See docs/JAVA_THREAD_DUMPS.md.", - "remaining": "Direct ZIP archive ingestion requested upstream remains unsupported; establish archive entry, decompression and total resource limits before implementing it." + "evidence": "Concatenated traditional jstack text and classic ZIP32 archives of same-JVM snapshots import into aggregate per-thread raw-sample profiles. ZIP STORE/DEFLATE input validates local/central metadata, CRC, paths and actual expansion; bounded UTF-8 lines feed the shared incremental parser. Limits are 32 MiB compressed, 512 entries, 16 MiB per entry and 64 MiB total. Large File preflight reads only the signature before enforcing the cap; decompression yields and supports cancellation. See docs/JAVA_THREAD_DUMP_ZIP.md.", + "remaining": "Archive support intentionally excludes ZIP64, encryption, split archives, special files, prefixed executables, padding, alternate Unicode-path extra fields and other compression methods. Final profile builders still scale with retained stacks; archives must concern one JVM." }, { "id": 168, @@ -1569,11 +1569,11 @@ "url": "https://github.com/jlfwong/speedscope/issues/40", "kind": "feature", "usefulness": "yes", - "status": "planned", - "priority": "P3", + "status": "implemented", + "priority": "none", "workstream": "Import formats and correctness", - "evidence": "Android method tracing binary format needs versioned parsing and thread-clock semantics, distinct from Chrome trace JSON. No fixture-backed implementation available in this pass.", - "remaining": "Android method tracing binary format needs versioned parsing and thread-clock semantics, distinct from Chrome trace JSON. No fixture-backed implementation available in this pass." + "evidence": "Classic Android concatenated key + SLOW method traces (versions 1-3) import with explicit CPU/wall/dual-clock policy, per-thread stack reconstruction, source/signature identity and clipped observation windows. Fixed records are scanned without a parsed-record array; key section/line sizes are bounded. Independently authored fixtures and malformed-input coverage follow AOSP format references. See docs/IMPORT_FORMATS.md and sample/profiles/android-method-trace/README.md.", + "remaining": "Reviewed scope is classic v1-v3 method tracing. Raw binary-only/streaming/new ART packet formats, Perfetto protobuf, systrace HTML, timestamp-wrap recovery and deobfuscation remain unsupported." }, { "id": 39, diff --git a/package.json b/package.json index 5e37b457..856fbf52 100644 --- a/package.json +++ b/package.json @@ -47,6 +47,8 @@ "docs/SEARCH_AND_LAYOUT.md", "docs/SOURCE_AND_LABELS.md", "docs/JAVA_THREAD_DUMPS.md", + "docs/JAVA_THREAD_DUMP_ZIP.md", + "docs/IMPORT_FORMATS.md", "docs/OFFLINE.md", "docs/SINGLE_FILE.md", "docs/PROPERTY_TESTS.md", diff --git a/sample/profiles/android-method-trace/README.md b/sample/profiles/android-method-trace/README.md new file mode 100644 index 00000000..8a4a43de --- /dev/null +++ b/sample/profiles/android-method-trace/README.md @@ -0,0 +1,62 @@ +# Android method-trace fixture + +`synthetic-dual.trace` is independently authored synthetic data, not a recording +from an Android device. Regenerate it with: + +```sh +node sample/profiles/android-method-trace/generate-synthetic.mjs +``` + +It contains a UTF-8 key section followed by a little-endian `SLOW` version-3 +header and six 14-byte dual-clock records. Two thread identities and overloaded +method signatures exercise source metadata, per-thread ordering and separate +CPU/wall totals. Interleaved wall records need not be globally sorted. + +## Format and clock policy + +The importer supports the classic concatenated key/data format: version 1 with +one-byte thread IDs, version 2 with two-byte thread IDs, and version 3 with an +explicit record size (including padding). `clock=thread-cpu`, `clock=wall`, and +version-3 `clock=dual` are recognized; legacy version-1 `clock=global` means wall +time. Missing/unknown clock metadata is rejected instead of guessed. + +All profile values use microseconds. Dual-clock input creates separate labeled +CPU and wall profiles for each observed thread, so CPU consumption is never +substituted for elapsed time. CPU clocks are local to their thread. Wall offsets +remain relative to the trace origin; leading unobserved intervals can therefore +appear as idle weight. The header's absolute start date is not added to offsets. + +Method entry, exit and exception-unwind actions maintain stack context. +Calls already active at trace start are reconstructed from initial exits, +wrapping their observed children from offset zero. Calls still open at EOF close +at their own thread's last observed event; no unobserved tail time is added. +These are clipped observation windows, not claims about complete method duration. + +The importer scans fixed records twice without building a parsed-record array. +Temporary state consists of the key dictionaries and per-thread stacks, plus +the output builders. The input buffer and resulting profiles still occupy memory. +Key text is decoded one line at a time, with explicit limits of 16 MiB per key +section and 1 MiB per line. The binary length, record count when declared, +method IDs, actions, per-thread timestamp monotonicity and stack transitions +are validated before building the output. + +## Sources and unsupported variants + +Primary references checked on 2026-11-21: + +- [AOSP VmTraceParser](https://android.googlesource.com/platform/tools/base/+/studio-master-dev/perflib/src/main/java/com/android/tools/perflib/vmtrace/VmTraceParser.java) + (blob `a01e92ff88f03b57dcdb160015c11c2193c9d238`): versions, key metadata, + clock interpretation, record layouts and actions. +- [AOSP classic format documentation](https://android.googlesource.com/platform/frameworks/base/+/b5a1ce2f5daa20b5c77113b8887e12dcc05590fc/docs/html/tools/debugging/debugging-tracing.jd): + concatenated files, little-endian values and version-1 timestamps. +- [AOSP CallStackReconstructor](https://android.googlesource.com/platform/tools/base/+/refs/tags/android-4.4.3_r1.1/perflib/src/main/java/com/android/tools/perflib/vmtrace/CallStackReconstructor.java) + and [dmtracedump](https://android.googlesource.com/platform/dalvik/+/c423f55f64f430d0e56a1809db49129f03760230/tools/dmtracedump/TraceDump.c): + initial unmatched exits and final open-call clipping. +- [Current ART writer](https://android.googlesource.com/platform/art/+/master/runtime/trace.cc): + explicit clocks, per-thread CPU origin and distinct streaming/new-format modes. + +Raw binary-only/streaming traces (including their version-mask extension), newer +ART packet formats, Perfetto protobuf, Android systrace HTML and timestamp-wrap +recovery are not supported by this importer. Extract/archive conversion and +deobfuscation are separate concerns. Corrupt partial records and decreasing or +wrapped per-thread timestamps produce explicit errors. diff --git a/sample/profiles/android-method-trace/generate-synthetic.mjs b/sample/profiles/android-method-trace/generate-synthetic.mjs new file mode 100644 index 00000000..d8caaa56 --- /dev/null +++ b/sample/profiles/android-method-trace/generate-synthetic.mjs @@ -0,0 +1,42 @@ +// Independently authored fixture in AOSP's classic key + SLOW v3 format. +import {writeFileSync} from 'node:fs' +const key = `*version +3 +clock=dual +data-file-overflow=false +num-method-calls=6 +*threads +1\tmain +513\tworker +*methods +0x4\tcom/example/Main\trun\t()V\tMain.java\t12 +0x8\tcom/example/Main\twork\t(I)V\tMain.java\t24 +0xc\tcom/example/Main\twork\t()V\tMain.java\t25 +*end +` +// [thread, method/action, CPU microseconds, wall microseconds]. +const records = [ + [1, 4, 0, 10], + [513, 12, 0, 5], + [1, 8, 5, 20], + [1, 9, 10, 35], + [513, 13, 7, 25], + [1, 5, 20, 50], +] +const binary = Buffer.alloc(32 + records.length * 14) +binary.write('SLOW') +binary.writeUInt16LE(3, 4) +binary.writeUInt16LE(32, 6) +binary.writeUInt16LE(14, 16) +for (let i = 0; i < records.length; i++) { + const [thread, method, cpu, wall] = records[i], + offset = 32 + i * 14 + binary.writeUInt16LE(thread, offset) + binary.writeUInt32LE(method, offset + 2) + binary.writeUInt32LE(cpu, offset + 6) + binary.writeUInt32LE(wall, offset + 10) +} +writeFileSync( + new URL('./synthetic-dual.trace', import.meta.url), + Buffer.concat([Buffer.from(key), binary]), +) diff --git a/sample/profiles/android-method-trace/synthetic-dual.trace b/sample/profiles/android-method-trace/synthetic-dual.trace new file mode 100644 index 00000000..1c0a490b Binary files /dev/null and b/sample/profiles/android-method-trace/synthetic-dual.trace differ diff --git a/sample/profiles/java-thread-dump/1.jstack b/sample/profiles/java-thread-dump/1.jstack new file mode 100644 index 00000000..d6aebdad --- /dev/null +++ b/sample/profiles/java-thread-dump/1.jstack @@ -0,0 +1,3 @@ +Full thread dump OpenJDK VM: +"main" #1 prio=5 tid=0x1 nid=1 runnable + at app.Main.first(Main.java:20) diff --git a/sample/profiles/java-thread-dump/2.jstack b/sample/profiles/java-thread-dump/2.jstack new file mode 100644 index 00000000..c65d26c2 --- /dev/null +++ b/sample/profiles/java-thread-dump/2.jstack @@ -0,0 +1,3 @@ +Full thread dump OpenJDK VM: +"main" #1 prio=5 tid=0x1 nid=1 runnable + at app.Main.second(Main.java:20) diff --git a/sample/profiles/java-thread-dump/synthetic.zip b/sample/profiles/java-thread-dump/synthetic.zip new file mode 100644 index 00000000..b6bae8d7 Binary files /dev/null and b/sample/profiles/java-thread-dump/synthetic.zip differ diff --git a/src/import/__snapshots__/pprof.test.ts.snap b/src/import/__snapshots__/pprof.test.ts.snap index ac505a09..a9f00f40 100644 --- a/src/import/__snapshots__/pprof.test.ts.snap +++ b/src/import/__snapshots__/pprof.test.ts.snap @@ -6,7 +6,7 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go", - "key": "runtime.main:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go:0", + "key": "["runtime.main","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go",198]", "line": 198, "name": "runtime.main", "selfWeight": 0, @@ -15,25 +15,25 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/Users/jlfwong/code/speedscope/sample/programs/go/simple.go", - "key": "main.main:/Users/jlfwong/code/speedscope/sample/programs/go/simple.go:0", - "line": 49, + "key": "["main.main","/Users/jlfwong/code/speedscope/sample/programs/go/simple.go",50]", + "line": 50, "name": "main.main", "selfWeight": 0, - "totalWeight": 1360000000, + "totalWeight": 580000000, }, Frame { "col": undefined, "file": "/Users/jlfwong/code/speedscope/sample/programs/go/simple.go", - "key": "main.delta:/Users/jlfwong/code/speedscope/sample/programs/go/simple.go:0", - "line": 28, + "key": "["main.delta","/Users/jlfwong/code/speedscope/sample/programs/go/simple.go",29]", + "line": 29, "name": "main.delta", - "selfWeight": 220000000, - "totalWeight": 580000000, + "selfWeight": 0, + "totalWeight": 140000000, }, Frame { "col": undefined, "file": "/Users/jlfwong/code/speedscope/sample/programs/go/simple.go", - "key": "main.beta:/Users/jlfwong/code/speedscope/sample/programs/go/simple.go:0", + "key": "["main.beta","/Users/jlfwong/code/speedscope/sample/programs/go/simple.go",18]", "line": 18, "name": "main.beta", "selfWeight": 390000000, @@ -42,7 +42,16 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/Users/jlfwong/code/speedscope/sample/programs/go/simple.go", - "key": "main.alpha:/Users/jlfwong/code/speedscope/sample/programs/go/simple.go:0", + "key": "["main.main","/Users/jlfwong/code/speedscope/sample/programs/go/simple.go",48]", + "line": 48, + "name": "main.main", + "selfWeight": 0, + "totalWeight": 260000000, + }, + Frame { + "col": undefined, + "file": "/Users/jlfwong/code/speedscope/sample/programs/go/simple.go", + "key": "["main.alpha","/Users/jlfwong/code/speedscope/sample/programs/go/simple.go",11]", "line": 11, "name": "main.alpha", "selfWeight": 480000000, @@ -51,7 +60,16 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/Users/jlfwong/code/speedscope/sample/programs/go/simple.go", - "key": "main.gamma:/Users/jlfwong/code/speedscope/sample/programs/go/simple.go:0", + "key": "["main.main","/Users/jlfwong/code/speedscope/sample/programs/go/simple.go",51]", + "line": 51, + "name": "main.main", + "selfWeight": 0, + "totalWeight": 270000000, + }, + Frame { + "col": undefined, + "file": "/Users/jlfwong/code/speedscope/sample/programs/go/simple.go", + "key": "["main.gamma","/Users/jlfwong/code/speedscope/sample/programs/go/simple.go",34]", "line": 34, "name": "main.gamma", "selfWeight": 270000000, @@ -60,7 +78,7 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/asm_amd64.s", - "key": "runtime.morestack:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/asm_amd64.s:0", + "key": "["runtime.morestack","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/asm_amd64.s",480]", "line": 480, "name": "runtime.morestack", "selfWeight": 0, @@ -69,7 +87,7 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/stack.go", - "key": "runtime.newstack:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/stack.go:0", + "key": "["runtime.newstack","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/stack.go",957]", "line": 957, "name": "runtime.newstack", "selfWeight": 0, @@ -78,16 +96,43 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/duff_amd64.s", - "key": "runtime.duffcopy:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/duff_amd64.s:0", + "key": "["runtime.duffcopy","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/duff_amd64.s",412]", "line": 412, "name": "runtime.duffcopy", "selfWeight": 110000000, "totalWeight": 110000000, }, + Frame { + "col": undefined, + "file": "/Users/jlfwong/code/speedscope/sample/programs/go/simple.go", + "key": "["main.main","/Users/jlfwong/code/speedscope/sample/programs/go/simple.go",49]", + "line": 49, + "name": "main.main", + "selfWeight": 0, + "totalWeight": 250000000, + }, + Frame { + "col": undefined, + "file": "/Users/jlfwong/code/speedscope/sample/programs/go/simple.go", + "key": "["main.delta","/Users/jlfwong/code/speedscope/sample/programs/go/simple.go",25]", + "line": 25, + "name": "main.delta", + "selfWeight": 220000000, + "totalWeight": 220000000, + }, + Frame { + "col": undefined, + "file": "/Users/jlfwong/code/speedscope/sample/programs/go/simple.go", + "key": "["main.delta","/Users/jlfwong/code/speedscope/sample/programs/go/simple.go",28]", + "line": 28, + "name": "main.delta", + "selfWeight": 0, + "totalWeight": 220000000, + }, Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/time.go", - "key": "runtime.timerproc:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/time.go:0", + "key": "["runtime.timerproc","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/time.go",247]", "line": 247, "name": "runtime.timerproc", "selfWeight": 0, @@ -96,7 +141,7 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/time.go", - "key": "runtime.goroutineReady:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/time.go:0", + "key": "["runtime.goroutineReady","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/time.go",125]", "line": 125, "name": "runtime.goroutineReady", "selfWeight": 0, @@ -105,7 +150,7 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go", - "key": "runtime.goready:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go:0", + "key": "["runtime.goready","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go",301]", "line": 301, "name": "runtime.goready", "selfWeight": 0, @@ -114,7 +159,7 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/asm_amd64.s", - "key": "runtime.systemstack:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/asm_amd64.s:0", + "key": "["runtime.systemstack","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/asm_amd64.s",409]", "line": 409, "name": "runtime.systemstack", "selfWeight": 0, @@ -123,7 +168,7 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go", - "key": "runtime.goready.func1:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go:0", + "key": "["runtime.goready.func1","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go",302]", "line": 302, "name": "runtime.goready.func1", "selfWeight": 0, @@ -132,7 +177,7 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go", - "key": "runtime.ready:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go:0", + "key": "["runtime.ready","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go",608]", "line": 608, "name": "runtime.ready", "selfWeight": 0, @@ -141,7 +186,7 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go", - "key": "runtime.wakep:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go:0", + "key": "["runtime.wakep","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go",2083]", "line": 2083, "name": "runtime.wakep", "selfWeight": 0, @@ -150,7 +195,7 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go", - "key": "runtime.startm:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go:0", + "key": "["runtime.startm","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go",2017]", "line": 2017, "name": "runtime.startm", "selfWeight": 0, @@ -159,7 +204,7 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/lock_sema.go", - "key": "runtime.notewakeup:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/lock_sema.go:0", + "key": "["runtime.notewakeup","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/lock_sema.go",147]", "line": 147, "name": "runtime.notewakeup", "selfWeight": 0, @@ -168,7 +213,7 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/os_darwin.go", - "key": "runtime.semawakeup:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/os_darwin.go:0", + "key": "["runtime.semawakeup","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/os_darwin.go",36]", "line": 36, "name": "runtime.semawakeup", "selfWeight": 0, @@ -177,7 +222,7 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/os_darwin.go", - "key": "runtime.mach_semrelease:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/os_darwin.go:0", + "key": "["runtime.mach_semrelease","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/os_darwin.go",465]", "line": 465, "name": "runtime.mach_semrelease", "selfWeight": 0, @@ -186,7 +231,7 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/sys_darwin_amd64.s", - "key": "runtime.mach_semaphore_signal:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/sys_darwin_amd64.s:0", + "key": "["runtime.mach_semaphore_signal","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/sys_darwin_amd64.s",558]", "line": 558, "name": "runtime.mach_semaphore_signal", "selfWeight": 10000000, @@ -195,7 +240,7 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go", - "key": "runtime.mstart:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go:0", + "key": "["runtime.mstart","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go",1193]", "line": 1193, "name": "runtime.mstart", "selfWeight": 0, @@ -204,7 +249,7 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go", - "key": "runtime.mstart1:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go:0", + "key": "["runtime.mstart1","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go",1227]", "line": 1227, "name": "runtime.mstart1", "selfWeight": 0, @@ -213,7 +258,7 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go", - "key": "runtime.sysmon:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go:0", + "key": "["runtime.sysmon","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/proc.go",4221]", "line": 4221, "name": "runtime.sysmon", "selfWeight": 0, @@ -222,7 +267,7 @@ exports[`importAsPprofProfile 1`] = ` Frame { "col": undefined, "file": "/usr/local/Cellar/go/1.10.1/libexec/src/runtime/sys_darwin_amd64.s", - "key": "runtime.usleep:/usr/local/Cellar/go/1.10.1/libexec/src/runtime/sys_darwin_amd64.s:0", + "key": "["runtime.usleep","/usr/local/Cellar/go/1.10.1/libexec/src/runtime/sys_darwin_amd64.s",418]", "line": 418, "name": "runtime.usleep", "selfWeight": 20000000, diff --git a/src/import/android-method-trace.test.ts b/src/import/android-method-trace.test.ts new file mode 100644 index 00000000..bb404cb7 --- /dev/null +++ b/src/import/android-method-trace.test.ts @@ -0,0 +1,303 @@ +import * as fs from 'fs' +import {importFromAndroidMethodTrace, isAndroidMethodTrace} from './android-method-trace' +import {exportProfileGroup, importSpeedscopeProfiles} from '../lib/file-format' +import {importProfilesFromArrayBuffer} from '.' + +type Record = [number, number, number, number?] +function capture(options: { + version?: number + clock?: string + records?: Record[] + methods?: string + recordSize?: number + keyVersion?: number +}) { + const version = options.version ?? 3, + clock = options.clock ?? 'wall' + const dual = clock === 'dual' + const records = options.records || [ + [1, 4, 0], + [1, 5, 20], + ] + const recordSize = options.recordSize ?? (version === 1 ? 9 : dual ? 14 : 10) + const key = `*version\n${ + options.keyVersion ?? version + }\nclock=${clock}\n*threads\n1\tmain\n*methods\n${ + options.methods || + '0x4\tapp/Main\trun\t()V\tMain.java\t12\n0x8\tapp/Main\twork\t()V\tMain.java\t24' + }\n*end\n` + const header = Buffer.alloc(32) + header.write('SLOW') + header.writeUInt16LE(version, 4) + header.writeUInt16LE(32, 6) + header.writeUInt16LE(recordSize, 16) + const data = Buffer.alloc(records.length * recordSize) + for (let i = 0; i < records.length; i++) { + const offset = i * recordSize, + [thread, method, time, wall = time] = records[i] + const width = version === 1 ? 1 : 2 + if (width === 1) data.writeUInt8(thread, offset) + else data.writeUInt16LE(thread, offset) + data.writeUInt32LE(method, offset + width) + data.writeUInt32LE(time, offset + width + 4) + if (dual) data.writeUInt32LE(wall, offset + width + 8) + } + return Buffer.concat([Buffer.from(key), header, data]) +} +function input(buffer: Buffer): ArrayBuffer { + return buffer.buffer.slice( + buffer.byteOffset, + buffer.byteOffset + buffer.byteLength, + ) as ArrayBuffer +} +const read = (buffer: Buffer) => importFromAndroidMethodTrace('capture.trace', input(buffer))! +const fixture = fs.readFileSync('./sample/profiles/android-method-trace/synthetic-dual.trace') + +test('classic dual-clock Android traces preserve thread identities, clock units and independent timing', () => { + const group = read(fixture) + expect(group.profiles.map(profile => profile.getName())).toEqual([ + 'main (#1) — thread CPU time', + 'main (#1) — wall time', + 'worker (#513) — thread CPU time', + 'worker (#513) — wall time', + ]) + expect(group.profiles.map(profile => profile.getTotalWeight())).toEqual([20, 50, 7, 25]) + for (const profile of group.profiles) expect(profile.getWeightUnit()).toBe('microseconds') + expect(group.profiles[0].getFrame(4)!.getTotalWeight()).toBe(20) + expect(group.profiles[0].getFrame(8)!.getSelfWeight()).toBe(5) + expect(group.profiles[1].getFrame(4)!.getTotalWeight()).toBe(40) + expect(group.profiles[1].getFrame(8)!.getTotalWeight()).toBe(15) + expect(group.profiles[0].getFrame(8)!.file).toBe('com/example/Main.java') + expect(group.profiles[0].getFrame(8)!.line).toBe(24) + expect(group.profiles[0].getFrame(8)!.name).toBe('com.example.Main.work(I)V') + expect(group.profiles[2].getFrame(12)!.name).toBe('com.example.Main.work()V') + const exported = exportProfileGroup(group) + expect(exportProfileGroup(importSpeedscopeProfiles(exported))).toEqual(exported) +}) + +test('v1 uses one-byte thread IDs/global wall clock; v2 uses two-byte IDs/thread CPU clock', () => { + const first = read( + capture({ + version: 1, + clock: 'global', + records: [ + [255, 4, 3], + [255, 5, 14], + ], + }), + ).profiles[0] + expect(first.getName()).toBe('Thread (#255) — wall time') + expect(first.getFrame(4)!.getTotalWeight()).toBe(11) + const second = read( + capture({ + version: 2, + clock: 'thread-cpu', + records: [ + [65535, 4, 0], + [65535, 5, 9], + ], + }), + ).profiles[0] + expect(second.getName()).toBe('Thread (#65535) — thread CPU time') + expect(second.getTotalWeight()).toBe(9) +}) + +test('capture-boundary exits reconstruct outer scopes and unfinished calls close only at observed end', () => { + const profile = read( + capture({ + records: [ + [1, 8, 2], + [1, 9, 5], + [1, 5, 10], + [1, 8, 14], + ], + }), + ).profiles[0] + expect(profile.getFrame(4)!.getTotalWeight()).toBe(10) + expect(profile.getFrame(8)!.getTotalWeight()).toBe(3) + const root = profile.getAppendOrderCalltreeRoot() + expect(root.children[0].frame.key).toBe(4) + expect(root.children[0].children[0].frame.key).toBe(8) + expect(profile.getTotalWeight()).toBe(14) +}) + +test('unwind actions exit each method and reconstruct a pre-capture ancestor', () => { + const profile = read( + capture({ + records: [ + [1, 8, 2], + [1, 10, 9], + [1, 6, 9], + ], + }), + ).profiles[0] + expect(profile.getFrame(4)!.getTotalWeight()).toBe(9) + expect(profile.getFrame(8)!.getTotalWeight()).toBe(7) + const known = read( + capture({ + records: [ + [1, 4, 1], + [1, 8, 3], + [1, 10, 7], + [1, 6, 7], + ], + }), + ).profiles[0] + expect(known.getFrame(4)!.getTotalWeight()).toBe(6) + expect(known.getFrame(8)!.getTotalWeight()).toBe(4) + expect(() => + read( + capture({ + records: [ + [1, 4, 1], + [1, 8, 3], + [1, 6, 7], + ], + }), + ), + ).toThrow('does not match its stack') +}) + +test('unsigned high method IDs and v3 padded records retain method identity', () => { + const profile = read( + capture({ + methods: '0x80000004\tapp/Main\trun\t()V', + records: [ + [1, 0x80000004, 0], + [1, 0x80000005, 9], + ], + recordSize: 18, + }), + ).profiles[0] + expect(profile.getFrame(0x80000004)!.getTotalWeight()).toBe(9) +}) + +test('invalid record actions, methods, stack transitions and decreasing clocks fail explicitly', () => { + for (const [records, error] of [ + [[[1, 7, 0]], 'reserved method action'], + [[[1, 12, 0]], 'unknown method ID'], + [ + [ + [1, 4, 0], + [1, 9, 2], + ], + 'does not match its stack', + ], + [ + [ + [1, 4, 10], + [1, 5, 9], + ], + 'wall timestamps decrease', + ], + ] as [Record[], string][]) + expect(() => read(capture({records}))).toThrow(error) + expect(() => + read( + capture({ + clock: 'thread-cpu', + records: [ + [1, 4, 10], + [1, 5, 9], + ], + }), + ), + ).toThrow('CPU timestamps decrease') + expect(() => + read( + capture({ + clock: 'dual', + records: [ + [1, 4, 1, 20], + [1, 5, 2, 19], + ], + }), + ), + ).toThrow('wall timestamps decrease') +}) + +test('truncated records/headers and incompatible version/clock metadata are rejected', () => { + expect(() => read(fixture.subarray(0, fixture.length - 1))).toThrow('truncated final record') + expect(() => read(Buffer.from('*version\n3\nclock=wall\n'))).toThrow('missing *end') + expect(() => read(capture({version: 4}))).toThrow('unsupported version 4') + expect(() => read(capture({keyVersion: 2}))).toThrow('version mismatch') + expect(() => read(capture({version: 2, clock: 'dual'}))).toThrow('dual clock requires') + expect(() => read(capture({clock: 'mystery'}))).toThrow('unsupported clock') + const tooSmall = Buffer.from(fixture), + binary = tooSmall.indexOf('SLOW') + tooSmall.writeUInt16LE(10, binary + 16) + expect(() => read(tooSmall)).toThrow('too small') + const offset = Buffer.from(fixture) + offset.writeUInt16LE(17, binary + 6) + expect(() => read(offset)).toThrow('invalid data offset') +}) + +test('binary-only/streaming variants are detected but rejected without guessing a clock', () => { + expect(isAndroidMethodTrace(input(Buffer.from('SLOW\u0000\u0000')))).toBe(true) + expect(() => read(Buffer.from('SLOW\u0000\u0000'))).toThrow('streaming/binary-only') + expect(isAndroidMethodTrace(input(Buffer.from('{"traceEvents":[]}')))).toBe(false) + expect(importFromAndroidMethodTrace('other', input(Buffer.from('hello')))).toBe(null) + const noClock = Buffer.from(capture({}).toString('binary').replace('clock=wall\n', ''), 'binary') + expect(() => read(noClock)).toThrow('missing explicit clock') +}) + +test('metadata preserves class-only and source-only methods and rejects duplicates/malformed IDs', () => { + const profile = read(capture({methods: '0x4\tapp/Main\tMain.java\t0'})).profiles[0] + expect(profile.getFrame(4)!.name).toBe('app.Main') + expect(profile.getFrame(4)!.file).toBe('app/Main.java') + expect(profile.getFrame(4)!.line).toBeUndefined() + expect(read(capture({methods: '0x4\tapp/Main'})).profiles[0].getFrame(4)!.name).toBe('app.Main') + expect(() => read(capture({methods: '0x5\tapp/Main'}))).toThrow('action bits clear') + expect(() => read(capture({methods: '0x4\tapp/Main\n0x4\tapp/Main'}))).toThrow('duplicate method') + expect(() => read(capture({methods: '0x4\tapp/Main\trun'}))).toThrow('invalid method metadata') +}) + +test('declared record counts detect whole-record truncation, while Unicode/CRLF keys preserve offsets', () => { + const binary = fs.readFileSync('./sample/profiles/android-method-trace/synthetic-dual.trace') + expect(() => readBytes(binary.subarray(0, binary.length - 14))).toThrow('record count mismatch') + const offset = binary.indexOf('SLOW') + const key = binary + .subarray(0, offset) + .toString('utf8') + .replace('main', 'méthod λ') + .replace(/\n/g, '\r\n') + const combined = Buffer.concat([Buffer.from('\uFEFF' + key), binary.subarray(offset)]) + expect(readBytes(combined).profiles[0].getName()).toBe('méthod λ (#1) — thread CPU time') +}) + +function readBytes(bytes: Buffer) { + return importFromAndroidMethodTrace( + 'capture.trace', + bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength) as ArrayBuffer, + )! +} + +test('fixed records are consumed directly for long traces without a parsed event list', () => { + const count = 10000, + prefix = capture({records: []}) + const events = Buffer.alloc(count * 20) + for (let i = 0; i < count; i++) { + for (const [position, action, time] of [ + [i * 20, 4, i * 2], + [i * 20 + 10, 5, i * 2 + 1], + ]) { + events.writeUInt16LE(1, position) + events.writeUInt32LE(action, position + 2) + events.writeUInt32LE(time, position + 6) + } + } + const profile = readBytes(Buffer.concat([prefix, events])).profiles[0] + expect(profile.getFrame(4)!.getTotalWeight()).toBe(count) + expect(profile.getTotalWeight()).toBe(count * 2 - 1) +}) + +test('ordinary importer detects an Android trace by signature, not filename', async () => { + const bytes = fs.readFileSync('./sample/profiles/android-method-trace/synthetic-dual.trace') + const group = await importProfilesFromArrayBuffer( + 'renamed.bin', + bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength) as ArrayBuffer, + ) + expect(group!.profiles).toHaveLength(4) + expect(group!.profiles.map(profile => profile.getTotalWeight())).toEqual([20, 50, 7, 25]) + expect(group!.profiles.every(profile => profile.getWeightUnit() === 'microseconds')).toBe(true) +}) diff --git a/src/import/android-method-trace.ts b/src/import/android-method-trace.ts new file mode 100644 index 00000000..1c153271 --- /dev/null +++ b/src/import/android-method-trace.ts @@ -0,0 +1,278 @@ +import {CallTreeProfileBuilder, FrameInfo, ProfileGroup} from '../lib/profile' +import {TimeFormatter} from '../lib/value-formatters' + +// Independently implemented from AOSP VmTraceParser and ART trace.cc, not +// copied parser code. Only the classic concatenated key + SLOW v1-v3 format. +// https://android.googlesource.com/platform/tools/base/+/studio-master-dev/perflib/src/main/java/com/android/tools/perflib/vmtrace/VmTraceParser.java +type Clock = 'thread-cpu' | 'wall' | 'dual' +const MAX_KEY_BYTES = 16 * 1024 * 1024 +const MAX_KEY_LINE_BYTES = 1024 * 1024 + +function fail(message: string): never { + throw new Error(`Android method trace: ${message}`) +} + +function startsWith(bytes: Uint8Array, offset: number, text: string): boolean { + for (let i = 0; i < text.length; i++) if (bytes[offset + i] !== text.charCodeAt(i)) return false + return true +} + +export function isAndroidMethodTrace(buffer: ArrayBuffer): boolean { + const bytes = new Uint8Array(buffer) + const offset = bytes[0] === 0xef && bytes[1] === 0xbb && bytes[2] === 0xbf ? 3 : 0 + return ( + startsWith(bytes, 0, 'SLOW') || + (startsWith(bytes, offset, '*version') && [10, 13].includes(bytes[offset + 8])) + ) +} + +function unsignedId(text: string, label: string, maximum = 0xffffffff): number { + if (!/^(?:0x[\da-f]+|\d+)$/i.test(text)) fail(`invalid ${label}`) + const value = Number(text) + if (!Number.isSafeInteger(value) || value < 0 || value > maximum) fail(`invalid ${label}`) + return value +} + +function methodInfo(line: string): FrameInfo { + const fields = line.includes('\t') ? line.split('\t') : line.split(/\s+/) + if (![2, 4, 6].includes(fields.length) || fields.some(field => !field)) + fail('invalid method metadata') + const id = unsignedId(fields[0], 'method ID') + if ((id & 3) !== 0) fail('method ID must leave its action bits clear') + const className = fields[1] + let name = className.replace(/\//g, '.') + let file: string | undefined, lineNumber: number | undefined + if (fields.length === 6 || (fields.length === 4 && fields[3].startsWith('('))) { + if (!fields[3].startsWith('(')) fail('invalid method signature') + name += '.' + fields[2] + fields[3] + } + if (fields.length === 6 || (fields.length === 4 && !fields[3].startsWith('('))) { + const source = fields[fields.length - 2], + location = fields[fields.length - 1] + if (!/^-?\d+$/.test(location) || !Number.isSafeInteger(Number(location))) + fail('invalid source line') + const number = Number(location) + if (number > 0) lineNumber = number + if (source !== 'Unknown Source' && source !== '(null)' && !/^<.*>$/.test(source)) { + const slash = className.lastIndexOf('/') + file = + slash > 0 && source.endsWith('.java') && !source.includes('/') + ? className.slice(0, slash + 1) + source + : source + } + } + return {key: id, name, file, line: file ? lineNumber : undefined} +} + +interface Header { + version: number + clock: Clock + threads: Map + methods: Map + dataOffset: number + recordSize: number + recordCount?: number +} + +function readHeader(buffer: ArrayBuffer): Header { + const bytes = new Uint8Array(buffer) + if (startsWith(bytes, 0, 'SLOW')) + fail('streaming/binary-only traces are not supported; use a classic key + data trace') + const decoder = new TextDecoder('utf-8', {fatal: true}) + let offset = bytes[0] === 0xef && bytes[1] === 0xbb && bytes[2] === 0xbf ? 3 : 0 + const nextLine = () => { + const start = offset + while (offset < bytes.length && bytes[offset] !== 10) { + if (offset >= MAX_KEY_BYTES) fail('key section exceeds 16 MiB') + if (offset - start >= MAX_KEY_LINE_BYTES) fail('key line exceeds 1 MiB') + offset++ + } + if (offset === bytes.length) fail('truncated key section or missing *end marker') + let text: string + try { + text = decoder.decode(bytes.subarray(start, offset)) + } catch { + fail('key section is not valid UTF-8') + } + offset++ + return text.replace(/\r$/, '') + } + if (nextLine() !== '*version') fail('missing *version marker') + const versionText = nextLine() + if (!/^[123]$/.test(versionText)) fail(`unsupported version ${versionText}`) + const version = Number(versionText) + let mode = 'options', + clock: Clock | undefined + let recordCount: number | undefined + const threads = new Map(), + methods = new Map() + while (true) { + const line = nextLine() + if (line === '*end') break + if (line === '*threads' || line === '*methods') { + mode = line.slice(1) + continue + } + if (!line) continue + if (line.startsWith('*')) fail('unsupported key section ' + line) + if (mode === 'options') { + if (line.startsWith('clock=')) { + if (clock) fail('duplicate clock metadata') + const value = line.slice(6) + if (value === 'global' && version === 1) clock = 'wall' + else if (value === 'wall' || value === 'thread-cpu' || value === 'dual') clock = value + else fail('unsupported clock ' + value) + } else if (line.startsWith('num-method-calls=')) { + if (recordCount !== undefined) fail('duplicate record count metadata') + const value = line.slice('num-method-calls='.length) + if (!/^\d+$/.test(value) || !Number.isSafeInteger(Number(value))) + fail('invalid record count') + recordCount = Number(value) + } else if (!line.includes('=')) fail('invalid trace option') + } else if (mode === 'threads') { + const match = /^(\S+)[\t ]+(.+)$/.exec(line) + if (!match) fail('invalid thread metadata') + const id = unsignedId(match[1], 'thread ID', version === 1 ? 255 : 65535) + if (threads.has(id)) fail('duplicate thread ID') + threads.set(id, match[2]) + } else { + const frame = methodInfo(line), + id = frame.key as number + if (methods.has(id)) fail('duplicate method ID') + methods.set(id, frame) + } + } + if (!clock) fail('missing explicit clock metadata') + if (clock === 'dual' && version !== 3) fail('dual clock requires version 3') + const binaryOffset = offset + if (bytes.length - binaryOffset < 16 || !startsWith(bytes, binaryOffset, 'SLOW')) + fail('truncated or invalid SLOW header') + const view = new DataView(buffer) + if (view.getUint16(binaryOffset + 4, true) !== version) fail('key/data version mismatch') + const headerSize = view.getUint16(binaryOffset + 6, true) + const minimumHeader = version === 3 ? 18 : 16 + if (headerSize < minimumHeader || binaryOffset + headerSize > bytes.length) + fail('invalid data offset or truncated header') + const recordSize = + version === 1 ? 9 : version === 2 ? 10 : view.getUint16(binaryOffset + 16, true) + const expectedSize = (version === 1 ? 1 : 2) + 8 + (clock === 'dual' ? 4 : 0) + if (recordSize < expectedSize || recordSize > 65535) + fail('record size is too small for the selected clock') + const dataOffset = binaryOffset + headerSize + if ((bytes.length - dataOffset) % recordSize !== 0) fail('truncated final record') + if (recordCount !== undefined && (bytes.length - dataOffset) / recordSize !== recordCount) + fail('record count mismatch or truncated trace') + return {version, clock, threads, methods, dataOffset, recordSize, recordCount} +} + +interface ThreadState { + initial: number[] + stack: number[] + lastCPU?: number + lastWall?: number + builders: CallTreeProfileBuilder[] +} + +export function importFromAndroidMethodTrace( + name: string, + buffer: ArrayBuffer, +): ProfileGroup | null { + if (!isAndroidMethodTrace(buffer)) return null + const header = readHeader(buffer), + view = new DataView(buffer) + const states = new Map() + const records = ( + consume: (thread: number, method: number, action: number, cpu: number, wall: number) => void, + ) => { + for (let offset = header.dataOffset; offset < buffer.byteLength; offset += header.recordSize) { + const thread = header.version === 1 ? view.getUint8(offset) : view.getUint16(offset, true) + const methodOffset = offset + (header.version === 1 ? 1 : 2) + const packed = view.getUint32(methodOffset, true), + action = packed & 3, + method = (packed & ~3) >>> 0 + const first = view.getUint32(methodOffset + 4, true) + const cpu = header.clock === 'wall' ? 0 : first + const wall = + header.clock === 'dual' + ? view.getUint32(methodOffset + 8, true) + : header.clock === 'wall' + ? first + : 0 + consume(thread, method, action, cpu, wall) + } + } + // Discover calls already active when tracing started without buffering events. + // Initial exits arrive from inner to outer; store in that order, then replay + // the prefix in reverse. The second pass preserves all observed child calls. + records((thread, method, action, cpu, wall) => { + if (action === 3) fail('reserved method action 3') + if (!header.methods.has(method)) fail(`unknown method ID 0x${method.toString(16)}`) + let state = states.get(thread) + if (!state) { + state = {initial: [], stack: [], builders: []} + states.set(thread, state) + } + if (state.lastCPU !== undefined && cpu < state.lastCPU) + fail('thread CPU timestamps decrease or wrap') + if (state.lastWall !== undefined && wall < state.lastWall) + fail('wall timestamps decrease or wrap within a thread') + state.lastCPU = cpu + state.lastWall = wall + if (action === 0) state.stack.push(method) + else { + // ART reports one unwind action for each exited method. It is not an + // instruction to discard arbitrary intervening calls. + if (!state.stack.length) state.initial.push(method) + else if (state.stack.pop() !== method) fail('method exit does not match its stack') + } + }) + const clocks: ('thread-cpu' | 'wall')[] = + header.clock === 'dual' ? ['thread-cpu', 'wall'] : [header.clock] + for (const [thread, state] of states) { + state.stack = [] + state.builders = clocks.map(clock => { + const builder = new CallTreeProfileBuilder() + builder.setName( + `${header.threads.get(thread) || 'Thread'} (#${thread}) — ${ + clock === 'wall' ? 'wall time' : 'thread CPU time' + }`, + ) + builder.setValueFormatter(new TimeFormatter('microseconds')) + for (let i = state.initial.length - 1; i >= 0; i--) + builder.enterFrame(header.methods.get(state.initial[i])!, 0) + return builder + }) + for (let i = state.initial.length - 1; i >= 0; i--) state.stack.push(state.initial[i]) + state.initial = [] + } + records((thread, method, action, cpu, wall) => { + const state = states.get(thread)! + const update = (enter: boolean, id: number) => { + for (let i = 0; i < clocks.length; i++) { + const time = clocks[i] === 'wall' ? wall : cpu + const frame = header.methods.get(id)! + if (enter) state.builders[i].enterFrame(frame, time) + else state.builders[i].leaveFrame(frame, time) + } + } + if (action === 0) { + state.stack.push(method) + update(true, method) + } else { + const exiting = state.stack.pop() + if (exiting === undefined) fail('method exit cannot be reconstructed') + if (exiting !== method) fail('method exit does not match its reconstructed stack') + update(false, exiting) + } + }) + const profiles = [] + for (const state of states.values()) { + while (state.stack.length) { + const frame = header.methods.get(state.stack.pop()!)! + for (let i = 0; i < clocks.length; i++) + state.builders[i].leaveFrame(frame, clocks[i] === 'wall' ? state.lastWall! : state.lastCPU!) + } + for (const builder of state.builders) profiles.push(builder.build()) + } + return {name, indexToView: 0, profiles} +} diff --git a/src/import/index.ts b/src/import/index.ts index f5fb0545..72802905 100644 --- a/src/import/index.ts +++ b/src/import/index.ts @@ -28,6 +28,22 @@ import {importFromPMCStatCallGraph} from './pmcstat-callgraph' import {importFromJfr, isJfrRecording} from './java-flight-recorder' import {importFromJSSelfProfiling, isJSSelfProfiling} from './js-self-profiling' import {importFromJavaThreadDump, isJavaThreadDump} from './java-thread-dump' +import type {ImportProgress} from './trace-worker-protocol' +import {importFromAndroidMethodTrace, isAndroidMethodTrace} from './android-method-trace' +import { + checkJavaThreadDumpZipFileSize, + importFromJavaThreadDumpZip, + isJavaThreadDumpZip, +} from './java-thread-dump-zip' + +export interface ImportReadOptions { + signal?: AbortSignal + onProgress?: (progress: ImportProgress) => void +} + +function checkImportAbort(options: ImportReadOptions) { + if (options.signal?.aborted) throw new DOMException('Import cancelled', 'AbortError') +} export async function importProfileGroupFromText( fileName: string, @@ -48,21 +64,34 @@ export async function importProfileGroupFromBase64( ) } -export async function importProfilesFromFile(file: File): Promise { - return importProfileGroup(MaybeCompressedDataReader.fromFile(file)) +export async function importProfilesFromFile( + file: File, + options: ImportReadOptions = {}, +): Promise { + checkImportAbort(options) + await checkJavaThreadDumpZipFileSize(file, options.signal) + return importProfileGroup(MaybeCompressedDataReader.fromFile(file), options) } export async function importProfilesFromArrayBuffer( fileName: string, buffer: ArrayBuffer, + options: ImportReadOptions = {}, ): Promise { - return importProfileGroup(MaybeCompressedDataReader.fromArrayBuffer(fileName, buffer)) + checkImportAbort(options) + return importProfileGroup(MaybeCompressedDataReader.fromArrayBuffer(fileName, buffer), options) } -async function importProfileGroup(dataSource: ProfileDataSource): Promise { +async function importProfileGroup( + dataSource: ProfileDataSource, + options: ImportReadOptions = {}, +): Promise { + checkImportAbort(options) const fileName = await dataSource.name() - const profileGroup = await _importProfileGroup(dataSource) + checkImportAbort(options) + const profileGroup = await _importProfileGroup(dataSource, options) + checkImportAbort(options) if (profileGroup) { profileGroup.sourceName = sourceFileName(fileName) if (!profileGroup.name) { @@ -83,10 +112,27 @@ function toGroup(profile: Profile | null): ProfileGroup | null { return {name: profile.getName(), indexToView: 0, profiles: [profile]} } -async function _importProfileGroup(dataSource: ProfileDataSource): Promise { +async function _importProfileGroup( + dataSource: ProfileDataSource, + options: ImportReadOptions, +): Promise { const fileName = await dataSource.name() const buffer = await dataSource.readAsArrayBuffer() + checkImportAbort(options) + + // Container and binary signatures take precedence over text/protobuf sniffing. + if (isJavaThreadDumpZip(new Uint8Array(buffer))) { + console.log('Importing Java thread dump ZIP') + return importFromJavaThreadDumpZip(buffer, fileName, { + signal: options.signal, + onProgress: options.onProgress, + }) + } + if (isAndroidMethodTrace(buffer)) { + console.log('Importing Android method trace') + return importFromAndroidMethodTrace(fileName, buffer) + } { const profile = importAsPprofProfile(buffer) diff --git a/src/import/java-thread-dump-zip.test.ts b/src/import/java-thread-dump-zip.test.ts new file mode 100644 index 00000000..5b72da52 --- /dev/null +++ b/src/import/java-thread-dump-zip.test.ts @@ -0,0 +1,266 @@ +import JSZip from 'jszip' +import * as pako from 'pako' +import fs from 'fs' +import { + importFromJavaThreadDumpZip, + checkJavaThreadDumpZipFileSize, + JAVA_DUMP_ZIP_LIMITS, +} from './java-thread-dump-zip' +import {exportProfileGroup} from '../lib/file-format' + +const dump = (method: string) => + `Full thread dump OpenJDK VM:\n"main" #1 prio=5 tid=0x1 nid=1 runnable\n at app.Main.${method}(Main.java:20)\n` +async function archive( + files: Record, + compression: 'STORE' | 'DEFLATE' = 'DEFLATE', + streamFiles = false, +) { + const zip = new JSZip() + for (const name of Object.keys(files)) zip.file(name, files[name], {createFolders: false}) + return zip.generateAsync({type: 'arraybuffer', compression, streamFiles}) +} +function central(buffer: ArrayBuffer) { + const view = new DataView(buffer) + const end = buffer.byteLength - 22 + return {view, offset: view.getUint32(end + 16, true)} +} +function setSize(buffer: ArrayBuffer, size: number) { + const {view, offset} = central(buffer) + view.setUint32(22, size, true) + view.setUint32(offset + 24, size, true) +} + +test('STORE, DEFLATE and descriptors combine snapshots in deterministic filename order', async () => { + for (const compression of ['STORE', 'DEFLATE'] as const) + for (const descriptors of [false, true]) { + const input = await archive( + {'2.jstack': dump('second'), '1.jstack': dump('first'), 'notes.txt': 'not a profile'}, + compression, + descriptors, + ) + const group = (await importFromJavaThreadDumpZip(input, 'capture.zip'))! + expect(group.profiles.length).toBe(1) + const profile = group.profiles[0] + expect(profile.getName()).toBe('main (#1)') + expect(profile.getTotalWeight()).toBe(2) + expect(profile.isChronological()).toBe(false) + expect(profile.getWeightUnit()).toBe('none') + expect([...profile.getFrames()].map(f => f.name)).toEqual([ + 'app.Main.first', + 'app.Main.second', + ]) + expect(exportProfileGroup(group).profiles.length).toBe(1) + } +}) + +test('300 jstack snapshots from the upstream shell example fit the bounded entry policy', async () => { + const files: Record = {} + for (let i = 1; i <= 300; i++) files[`${i}.jstack`] = dump('run') + const group = await importFromJavaThreadDumpZip(await archive(files), '300-snapshots.zip') + expect(group!.profiles[0].getTotalWeight()).toBe(300) +}) + +test('unsigned descriptors work even when their CRC bytes equal the optional signature', async () => { + const signed = await archive({'a.jstack': dump('a')}, 'DEFLATE', true) + const c = central(signed) + const descriptor = 30 + 'a.jstack'.length + c.view.getUint32(c.offset + 20, true) + const bytes = new Uint8Array(signed.byteLength - 4) + bytes.set(new Uint8Array(signed, 0, descriptor)) + bytes.set(new Uint8Array(signed, descriptor + 4), descriptor) + new DataView(bytes.buffer).setUint32(bytes.length - 22 + 16, c.offset - 4, true) + expect( + (await importFromJavaThreadDumpZip(bytes.buffer, 'x.zip'))!.profiles[0].getTotalWeight(), + ).toBe(1) + const unsigned = central(bytes.buffer) + unsigned.view.setUint32(unsigned.offset + 16, 0x08074b50, true) + unsigned.view.setUint32(descriptor, 0x08074b50, true) + // Descriptor parsing succeeds; only the deliberately false checksum fails. + await expect(importFromJavaThreadDumpZip(bytes.buffer, 'x.zip')).rejects.toThrow('CRC32 mismatch') +}) + +test('empty DEFLATE directory streams are validated without becoming profile input', async () => { + const zip = new JSZip() + zip.folder('captures') + const original = await zip.generateAsync({type: 'arraybuffer'}) + const c = central(original), + data = 30 + 'captures/'.length + const packed = pako.deflateRaw(new Uint8Array(0)) + const bytes = new Uint8Array(original.byteLength + packed.length) + bytes.set(new Uint8Array(original, 0, data)) + bytes.set(packed, data) + bytes.set(new Uint8Array(original, data), data + packed.length) + const view = new DataView(bytes.buffer), + offset = c.offset + packed.length + view.setUint16(8, 8, true) + view.setUint32(18, packed.length, true) + view.setUint16(offset + 10, 8, true) + view.setUint32(offset + 20, packed.length, true) + view.setUint32(bytes.length - 22 + 16, offset, true) + await expect(importFromJavaThreadDumpZip(bytes.buffer, 'x.zip')).rejects.toThrow( + 'contains no UTF-8 Java', + ) +}) + +test('directory entries are skipped and UTF-8 source symbols survive output chunk boundaries', async () => { + const zip = new JSZip() + zip.folder('captures') + zip.file('captures/a.jstack', ' '.repeat(16300) + '\n' + dump('rún_雪')) + const group = await importFromJavaThreadDumpZip( + await zip.generateAsync({type: 'arraybuffer', compression: 'DEFLATE'}), + 'x.zip', + ) + expect([...group!.profiles[0].getFrames()][0].name).toBe('app.Main.rún_雪') +}) + +test('compressed, entry-count, declared expansion, filename and depth caps reject before inflating', async () => { + const input = await archive({'a.jstack': dump('a'), 'b.jstack': dump('b')}) + for (const limits of [ + {maxCompressedBytes: 20}, + {maxEntries: 1}, + {maxEntryBytes: 10}, + {maxTotalBytes: 150}, + {maxNameBytes: 2}, + ]) + await expect(importFromJavaThreadDumpZip(input, 'x.zip', {limits})).rejects.toThrow( + /limit|name/, + ) + await expect( + importFromJavaThreadDumpZip(await archive({'a/b/c.jstack': dump('a')}), 'x.zip', { + limits: {maxDepth: 2}, + }), + ).rejects.toThrow('path') +}) + +test('unsafe and ambiguous paths are rejected without disk extraction', async () => { + for (const name of ['../a.jstack', '/a.jstack', 'a//b.jstack', 'C:/a.jstack', 'a\\b.jstack']) + await expect( + importFromJavaThreadDumpZip(await archive({[name]: dump('a')}), 'x.zip'), + ).rejects.toThrow('path') + await expect( + importFromJavaThreadDumpZip( + await archive({'A.jstack': dump('a'), 'a.jstack': dump('b')}), + 'x.zip', + ), + ).rejects.toThrow('ambiguous') + await expect( + importFromJavaThreadDumpZip( + await archive({'a.jstack': dump('a'), 'a.jstack/b.jstack': dump('b')}), + 'x.zip', + ), + ).rejects.toThrow('parent directory') +}) + +test('encrypted, inconsistent local headers and symlink entries are rejected', async () => { + const input = await archive({'a.jstack': dump('a')}) + const encrypted = input.slice(0), + c = central(encrypted) + c.view.setUint16(6, 1, true) + c.view.setUint16(c.offset + 8, 1, true) + await expect(importFromJavaThreadDumpZip(encrypted, 'x.zip')).rejects.toThrow('encrypted') + const mismatch = input.slice(0) + new Uint8Array(mismatch)[30] ^= 1 + await expect(importFromJavaThreadDumpZip(mismatch, 'x.zip')).rejects.toThrow('paths disagree') + const symlink = input.slice(0), + s = central(symlink) + s.view.setUint32(s.offset + 38, 0xa1ff0000, true) + await expect(importFromJavaThreadDumpZip(symlink, 'x.zip')).rejects.toThrow('special files') +}) + +test('actual expansion stops in onData even when central/local sizes lie about a compressed bomb', async () => { + const input = await archive({'bomb.bin': new Uint8Array(1024 * 1024)}) + setSize(input, 64) + const push = jest.spyOn(pako.Inflate.prototype, 'push') + try { + await expect( + importFromJavaThreadDumpZip(input, 'x.zip', { + limits: {maxEntryBytes: 128, maxTotalBytes: 256}, + }), + ).rejects.toThrow('actual decompressed bytes') + expect(push).toHaveBeenCalledTimes(1) + } finally { + push.mockRestore() + } +}) + +test('CRC corruption, truncated archives and damaged DEFLATE streams fail without a partial group', async () => { + const input = await archive({'a.jstack': dump('a')}) + const corrupt = input.slice(0), + c = central(corrupt) + c.view.setUint32(14, 123, true) + c.view.setUint32(c.offset + 16, 123, true) + await expect(importFromJavaThreadDumpZip(corrupt, 'x.zip')).rejects.toThrow('CRC32') + await expect( + importFromJavaThreadDumpZip(input.slice(0, input.byteLength - 1), 'x.zip'), + ).rejects.toThrow('end record') + const damaged = input.slice(0) + new Uint8Array(damaged)[30 + 'a.jstack'.length] = 255 + await expect(importFromJavaThreadDumpZip(damaged, 'x.zip')).rejects.toThrow( + /DEFLATE|CRC32|decompressed/, + ) +}) + +test('entry boundary validation and bounded UTF-8 lines reject malformed or unrelated input', async () => { + await expect( + importFromJavaThreadDumpZip(await archive({'a.jstack': 'not a dump'}), 'x.zip'), + ).rejects.toThrow('Full thread dump') + await expect( + importFromJavaThreadDumpZip(await archive({'README.txt': 'ordinary notes'}), 'x.zip'), + ).rejects.toThrow('no UTF-8 Java') + await expect( + importFromJavaThreadDumpZip(await archive({'a.txt': 'x'.repeat(500)}), 'x.zip', { + limits: {maxLineChars: 128}, + }), + ).rejects.toThrow('text line') + const malformed = dump('a') + ' at missing-location\n' + await expect( + importFromJavaThreadDumpZip(await archive({'a.jstack': malformed}), 'x.zip'), + ).rejects.toThrow('Invalid Java stack frame') +}) + +test('cancellation is observed before reading and after yielding during decompression', async () => { + const input = await archive( + {'a.bin': new Uint8Array(1024 * 1024), 'b.jstack': dump('a')}, + 'STORE', + ) + const early = new AbortController() + early.abort() + await expect( + importFromJavaThreadDumpZip(input, 'x.zip', {signal: early.signal}), + ).rejects.toHaveProperty('name', 'AbortError') + let now = 0 + const clock = jest.spyOn(performance, 'now') + clock.mockImplementation(() => (now += 20)) + const controller = new AbortController() + const timer = setTimeout(() => controller.abort(), 0) + try { + await expect( + importFromJavaThreadDumpZip(input, 'x.zip', {signal: controller.signal}), + ).rejects.toHaveProperty('name', 'AbortError') + } finally { + clearTimeout(timer) + clock.mockRestore() + } +}) + +test('oversized File preflight reads only four signature bytes and never materializes the body', async () => { + const slice = jest.fn(() => ({arrayBuffer: async () => new Uint8Array([80, 75, 3, 4]).buffer})) + const file = { + size: JAVA_DUMP_ZIP_LIMITS.maxCompressedBytes + 1, + slice, + arrayBuffer: jest.fn(), + } as unknown as File + await expect(checkJavaThreadDumpZipFileSize(file)).rejects.toThrow('compressed input') + expect(slice).toHaveBeenCalledWith(0, 4) + expect(file.arrayBuffer).not.toHaveBeenCalled() +}) + +test('checked-in synthetic ZIP imports directly', async () => { + const bytes = fs.readFileSync('sample/profiles/java-thread-dump/synthetic.zip') + const input = bytes.buffer.slice( + bytes.byteOffset, + bytes.byteOffset + bytes.byteLength, + ) as ArrayBuffer + expect( + (await importFromJavaThreadDumpZip(input, 'synthetic.zip'))!.profiles[0].getTotalWeight(), + ).toBe(2) +}) diff --git a/src/import/java-thread-dump-zip.ts b/src/import/java-thread-dump-zip.ts new file mode 100644 index 00000000..52fe1ee9 --- /dev/null +++ b/src/import/java-thread-dump-zip.ts @@ -0,0 +1,325 @@ +import * as pako from 'pako' +import {createJavaThreadDumpParser, isJavaThreadDump} from './java-thread-dump' +import {ImportProgress} from './trace-worker-protocol' + +// Classic ZIP32 structures: PKWARE APPNOTE 6.3.10, sections 4.3.7–4.3.16. +// https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT +// pako 1.0.6 onData is overridden: it must not accumulate inflate.chunks/result. +// https://github.com/nodeca/pako/blob/1.0.6/lib/inflate.js +export const JAVA_DUMP_ZIP_LIMITS = Object.freeze({ + maxCompressedBytes: 32 * 1024 * 1024, + maxEntries: 512, + maxEntryBytes: 16 * 1024 * 1024, + maxTotalBytes: 64 * 1024 * 1024, + maxNameBytes: 512, + maxDepth: 8, + maxLineChars: 65536, +}) +export type JavaThreadDumpZipLimits = { + -readonly [K in keyof typeof JAVA_DUMP_ZIP_LIMITS]: number +} +export interface JavaThreadDumpZipOptions { + signal?: AbortSignal + onProgress?: (progress: ImportProgress) => void + // Tests/embedding may lower limits, never raise the production ceilings. + limits?: Partial +} +interface Entry { + name: string + directory: boolean + method: number + compressed: number + size: number + crc: number + data: number + local: number + end: number +} +const fail = (message: string): never => { + throw new Error(`Java thread-dump ZIP: ${message}`) +} +const abort = (signal?: AbortSignal) => { + if (signal?.aborted) throw new DOMException('Import cancelled', 'AbortError') +} +const crcTable = new Uint32Array(256) +for (let n = 0; n < 256; n++) { + let c = n + for (let k = 0; k < 8; k++) c = c & 1 ? 0xedb88320 ^ (c >>> 1) : c >>> 1 + crcTable[n] = c >>> 0 +} + +export function isJavaThreadDumpZip(bytes: Uint8Array): boolean { + return ( + bytes.length >= 4 && + bytes[0] === 0x50 && + bytes[1] === 0x4b && + ((bytes[2] === 3 && bytes[3] === 4) || (bytes[2] === 5 && bytes[3] === 6)) + ) +} + +export async function checkJavaThreadDumpZipFileSize(file: File, signal?: AbortSignal) { + abort(signal) + if (file.size <= JAVA_DUMP_ZIP_LIMITS.maxCompressedBytes) return + const signature = new Uint8Array(await file.slice(0, 4).arrayBuffer()) + abort(signal) + if (isJavaThreadDumpZip(signature)) fail('compressed input exceeds its limit') +} + +function catalog(bytes: Uint8Array, limits: JavaThreadDumpZipLimits): Entry[] { + if (bytes.byteLength > limits.maxCompressedBytes) fail('compressed input exceeds its limit') + const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength) + const u16 = (at: number) => view.getUint16(at, true) + const u32 = (at: number) => view.getUint32(at, true) + let end = -1 + for (let i = bytes.length - 22; i >= Math.max(0, bytes.length - 65557); i--) { + if (u32(i) === 0x06054b50 && i + 22 + u16(i + 20) === bytes.length) { + end = i + break + } + } + if (end < 0) fail('missing or malformed end record') + if (u16(end + 4) || u16(end + 6) || u16(end + 8) !== u16(end + 10)) + fail('split archives are unsupported') + const count = u16(end + 10), + centralSize = u32(end + 12), + central = u32(end + 16) + if (count === 65535 || central === 0xffffffff || centralSize === 0xffffffff) + fail('ZIP64 is unsupported') + if (count > limits.maxEntries) fail('entry count exceeds its limit') + if (central + centralSize !== end) fail('inconsistent central directory') + const names = new Map(), + entries: Entry[] = [] + let offset = central, + declaredTotal = 0 + const extra = (start: number, length: number) => { + const stop = start + length + for (let p = start; p < stop; ) { + if (p + 4 > stop) fail('malformed extra field') + const tag = u16(p), + size = u16(p + 2) + if ([0x0001, 0x7075, 0x9901].includes(tag)) + fail('ZIP64, alternate Unicode paths and encryption are unsupported') + p += 4 + size + if (p > stop) fail('malformed extra field') + } + } + for (let n = 0; n < count; n++) { + if (offset + 46 > end || u32(offset) !== 0x02014b50) fail('malformed central entry') + const flags = u16(offset + 8), + method = u16(offset + 10) + if (flags & ~0x080e) fail('encrypted or unsupported ZIP flags') + if (method !== 0 && method !== 8) fail('only STORE and DEFLATE compression are supported') + const compressed = u32(offset + 20), + size = u32(offset + 24), + local = u32(offset + 42) + const nameLength = u16(offset + 28), + extraLength = u16(offset + 30), + commentLength = u16(offset + 32) + if (u16(offset + 34) || [compressed, size, local].includes(0xffffffff)) + fail('split archives and ZIP64 are unsupported') + if (size > limits.maxEntryBytes || compressed > limits.maxCompressedBytes) + fail('entry size exceeds its limit') + declaredTotal += size + if (declaredTotal > limits.maxTotalBytes) fail('total decompressed size exceeds its limit') + if ( + !nameLength || + nameLength > limits.maxNameBytes || + offset + 46 + nameLength + extraLength + commentLength > end + ) + fail('invalid entry name or header length') + const nameBytes = bytes.subarray(offset + 46, offset + 46 + nameLength) + if (!(flags & 0x0800) && nameBytes.some(b => b > 127)) + fail('non-ASCII names must declare UTF-8') + const name = new TextDecoder('utf-8', {fatal: true}).decode(nameBytes) + const directory = name.endsWith('/') + const path = directory ? name.slice(0, -1) : name + const parts = path.split('/') + if ( + !path || + /[\\:\u0000-\u001f\u007f]/.test(name) || + parts.length > limits.maxDepth || + parts.some(p => !p || p === '.' || p === '..') || + name.normalize('NFC') !== name + ) + fail('unsafe or ambiguous entry path') + const identity = path.toLowerCase() + if (names.has(identity)) fail('duplicate or ambiguous entry path') + names.set(identity, directory) + const unixType = (u32(offset + 38) >>> 16) & 0xf000 + if (unixType && unixType !== 0x8000 && unixType !== 0x4000) + fail('links and special files are unsupported') + if ((unixType === 0x4000 || u32(offset + 38) & 0x10) && !directory) + fail('directory attributes disagree with path') + if (directory && size) fail('nonempty directory entry') + extra(offset + 46 + nameLength, extraLength) + if ( + local + 30 > central || + u32(local) !== 0x04034b50 || + u16(local + 6) !== flags || + u16(local + 8) !== method || + u16(local + 26) !== nameLength + ) + fail('local and central headers disagree') + const localExtra = u16(local + 28), + data = local + 30 + nameLength + localExtra + if (data + compressed > central) fail('entry exceeds archive data bounds') + for (let i = 0; i < nameLength; i++) + if (bytes[local + 30 + i] !== nameBytes[i]) fail('local and central paths disagree') + extra(local + 30 + nameLength, localExtra) + const crc = u32(offset + 16) + if (directory && crc !== 0) fail('directory entry has a nonempty checksum') + for (const [position, expected] of [ + [14, crc], + [18, compressed], + [22, size], + ]) { + const actual = u32(local + position) + if (actual !== expected && (!(flags & 8) || actual !== 0)) + fail('local and central sizes/checksum disagree') + } + let stop = data + compressed + if (flags & 8) { + if (stop + 12 > central) fail('missing data descriptor') + const matches = (at: number) => + at + 12 <= central && u32(at) === crc && u32(at + 4) === compressed && u32(at + 8) === size + // The CRC of an unsigned descriptor may itself equal the signature. + if (matches(stop)) stop += 12 + else if (u32(stop) === 0x08074b50 && matches(stop + 4)) stop += 16 + else fail('invalid data descriptor') + } + entries.push({name, directory, method, compressed, size, crc, local, data, end: stop}) + offset += 46 + nameLength + extraLength + commentLength + } + if (offset !== end) fail('central directory count/size mismatch') + for (const path of names.keys()) { + const pieces = path.split('/') + for (let i = 1; i < pieces.length; i++) { + if (names.get(pieces.slice(0, i).join('/')) === false) + fail('a regular file is also used as a parent directory') + } + } + let next = 0 + for (const entry of entries.slice().sort((a, b) => a.local - b.local)) { + if (entry.local !== next) + fail('overlapping entries, padding or prefixed executables are unsupported') + next = entry.end + } + if (next !== central) fail('unaccounted archive data') + return entries.sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)) +} + +export async function importFromJavaThreadDumpZip( + buffer: ArrayBuffer, + name: string, + options: JavaThreadDumpZipOptions = {}, +) { + abort(options.signal) + const limits: JavaThreadDumpZipLimits = {...JAVA_DUMP_ZIP_LIMITS} + for (const key of Object.keys(limits) as (keyof JavaThreadDumpZipLimits)[]) { + const requested = options.limits?.[key] + if (requested !== undefined) { + if (!Number.isSafeInteger(requested) || requested < 1 || requested > limits[key]) + fail('invalid resource limit') + limits[key] = requested + } + } + const bytes = new Uint8Array(buffer), + entries = catalog(bytes, limits) + const parser = createJavaThreadDumpParser(name) + let total = 0, + recognizedEntries = 0, + lastYield = performance.now() + const progress = () => { + options.onProgress?.({phase: 'Parsing profile', canCancel: true}) + abort(options.signal) + } + for (const entry of entries) { + abort(options.signal) + if (entry.directory && !entry.compressed) continue + const candidate = /\.(?:jstack|txt|log|dump|tdump)$/i.test(entry.name) + const decoder = candidate ? new TextDecoder('utf-8', {fatal: true}) : null + let count = 0, + crc = 0xffffffff, + carry = '', + recognized = false + const line = (value: string) => { + if (!recognized && count <= 65536 && isJavaThreadDump(value)) { + recognized = true + recognizedEntries++ + } + if (recognized) parser.pushLine(value) + } + const text = (value: string) => { + let from = 0, + at: number + while ((at = value.indexOf('\n', from)) !== -1) { + if (carry.length + at - from > limits.maxLineChars) fail('text line exceeds its limit') + line(carry + value.slice(from, at)) + carry = '' + from = at + 1 + } + if (carry.length + value.length - from > limits.maxLineChars) + fail('text line exceeds its limit') + carry += value.slice(from) + } + const consume = (chunk: Uint8Array) => { + abort(options.signal) + count += chunk.length + total += chunk.length + // Check actual expansion before CRC, decoding, line retention or parser work. + if (count > limits.maxEntryBytes || total > limits.maxTotalBytes) + fail('actual decompressed bytes exceed their limit') + if (count > entry.size) fail('actual decompressed size exceeds declared entry size') + for (const value of chunk) crc = crcTable[(crc ^ value) & 255] ^ (crc >>> 8) + if (decoder) text(decoder.decode(chunk, {stream: true})) + } + progress() + if (entry.method === 0) { + for (let p = 0; p < entry.compressed; p += 16384) { + consume(bytes.subarray(entry.data + p, entry.data + Math.min(p + 16384, entry.compressed))) + if (performance.now() - lastYield >= 8) { + await new Promise(resolve => setTimeout(resolve, 0)) + lastYield = performance.now() + progress() + } + } + } else { + const inflate = new pako.Inflate({raw: true, chunkSize: 16384} as pako.InflateOptions & { + chunkSize: number + }) as pako.Inflate & { + ended: boolean + strm: {total_in: number} + } + inflate.onData = data => consume(data as Uint8Array) + for (let p = 0; p < entry.compressed; p += 1024) { + abort(options.signal) + if ( + !inflate.push( + bytes.subarray(entry.data + p, entry.data + Math.min(p + 1024, entry.compressed)), + false, + ) + ) + fail('invalid DEFLATE stream') + if (performance.now() - lastYield >= 8) { + await new Promise(resolve => setTimeout(resolve, 0)) + lastYield = performance.now() + progress() + } + } + if (!inflate.ended || inflate.err || inflate.strm.total_in !== entry.compressed) + fail('truncated or trailing DEFLATE data') + } + if (count !== entry.size || (crc ^ 0xffffffff) >>> 0 !== entry.crc) + fail('decompressed size or CRC32 mismatch') + if (decoder) { + text(decoder.decode()) + line(carry) + } + if (/\.jstack$/i.test(entry.name) && !recognized) + fail('jstack entry has no Full thread dump header near its start') + parser.endEntry() + } + progress() + if (!recognizedEntries) fail('archive contains no UTF-8 Java thread-dump text entries') + return parser.build() +} diff --git a/src/import/java-thread-dump.ts b/src/import/java-thread-dump.ts index e4cd14b3..89d94f52 100644 --- a/src/import/java-thread-dump.ts +++ b/src/import/java-thread-dump.ts @@ -57,6 +57,14 @@ export function importFromJavaThreadDump( contents: TextFileContent, name: string, ): ProfileGroup | null { + const parser = createJavaThreadDumpParser(name) + for (const line of contents.splitLines()) parser.pushLine(line) + return parser.build() +} + +// ZIP imports feed decoded lines directly, preserving one shared output model +// without concatenating every entry into a second full text recording. +export function createJavaThreadDumpParser(name: string) { const builders = new Map() let current: ThreadObservation | null = null let inDump = false @@ -77,21 +85,21 @@ export function importFromJavaThreadDump( } current = null } - for (const raw of contents.splitLines()) { + const pushLine = (raw: string) => { const line = raw.replace(/\r$/, '').replace(/^\uFEFF/, '') if (/^Full thread dump [^\r\n]+/.test(line)) { flush() inDump = true seen.clear() - continue + return } - if (!inDump) continue + if (!inDump) return // jstack repeats stacks in deadlock summaries. They are diagnostic text, // not another observation; only a new Full thread dump opens a snapshot. if (/^Found (?:one|\d+) Java-level deadlocks?:/.test(line)) { flush() inDump = false - continue + return } const header = parseThreadHeader(line) if (header) { @@ -102,7 +110,7 @@ export function importFromJavaThreadDump( ) seen.add(header.identity) current = header - continue + return } if (line.startsWith('"')) throw new Error('Invalid thread header in Java thread dump') if (!line.trim()) { @@ -110,7 +118,7 @@ export function importFromJavaThreadDump( // it closes the observation; later orphan frames must not be silently // attached to the previous thread or discarded from a partial import. if (current?.frames.length) flush() - continue + return } if (/^\s+at(?:\s|$)/.test(line)) { if (!current) throw new Error('Java stack frame has no thread header') @@ -119,8 +127,18 @@ export function importFromJavaThreadDump( current.frames.push(frame) } } - flush() - return sawFrame - ? {name, indexToView: 0, profiles: [...builders.values()].map(b => b.build())} - : null + return { + pushLine, + endEntry() { + flush() + inDump = false + seen.clear() + }, + build(): ProfileGroup | null { + flush() + return sawFrame + ? {name, indexToView: 0, profiles: [...builders.values()].map(b => b.build())} + : null + }, + } } diff --git a/src/import/pprof-metadata.test.ts b/src/import/pprof-metadata.test.ts new file mode 100644 index 00000000..262a8169 --- /dev/null +++ b/src/import/pprof-metadata.test.ts @@ -0,0 +1,73 @@ +import {perftools} from './profile.proto.js' +import Long from 'long' +import {importAsPprofProfile} from './pprof' + +function decode(profile: perftools.profiles.IProfile) { + const bytes = perftools.profiles.Profile.encode(profile).finish() + return importAsPprofProfile(Uint8Array.from(bytes).buffer as ArrayBuffer)! +} + +test('preferred sample type compares protobuf integer values and defaults to the last when absent', () => { + const input = { + stringTable: ['', 'allocations', 'count', 'space', 'bytes', 'run'], + sampleType: [ + {type: Long.fromInt(1), unit: Long.fromInt(2)}, + {type: Long.fromInt(3), unit: Long.fromInt(4)}, + ], + defaultSampleType: Long.fromInt(1), + function: [{id: 1, name: 5}], + location: [{id: 1, line: [{functionId: 1}]}], + sample: [{locationId: [1], value: [7, 900]}], + } + expect(decode(input).getTotalWeight()).toBe(7) + expect(decode(input).getWeightUnit()).toBe('none') + expect(decode({...input, defaultSampleType: Long.ZERO}).getTotalWeight()).toBe(900) + expect(decode({...input, defaultSampleType: Long.fromInt(999)}).getTotalWeight()).toBe(900) +}) + +test('locations of one function retain independent source lines and definition-line fallback', () => { + const profile = decode({ + stringTable: ['', 'samples', 'count', 'run', 'app.go'], + sampleType: [{type: 1, unit: 2}], + function: [{id: 1, name: 3, filename: 4, startLine: 1}], + location: [ + {id: 1, line: [{functionId: 1, line: 10}]}, + {id: 2, line: [{functionId: 1, line: 20}]}, + {id: 3, line: [{functionId: 1}]}, + ], + sample: [ + {locationId: [1], value: [2]}, + {locationId: [2], value: [3]}, + {locationId: [3], value: [4]}, + ], + }) + expect([...profile.getFrames()].map(f => [f.name, f.file, f.line, f.getTotalWeight()])).toEqual([ + ['run', 'app.go', 10, 2], + ['run', 'app.go', 20, 3], + ['run', 'app.go', 1, 4], + ]) +}) + +test('uint64 function and location IDs with identical low words stay distinct', () => { + const high = Long.fromString('18446744069414584321', true) + const profile = decode({ + stringTable: ['', 'samples', 'count', 'first', 'second'], + sampleType: [{type: 1, unit: 2}], + function: [ + {id: 1, name: 3}, + {id: high, name: 4}, + ], + location: [ + {id: 1, line: [{functionId: 1}]}, + {id: high, line: [{functionId: high}]}, + ], + sample: [ + {locationId: [1], value: [2]}, + {locationId: [high], value: [3]}, + ], + }) + expect([...profile.getFrames()].map(f => [f.name, f.getTotalWeight()])).toEqual([ + ['first', 2], + ['second', 3], + ]) +}) diff --git a/src/import/pprof.ts b/src/import/pprof.ts index 6771a243..7b4fd9c3 100644 --- a/src/import/pprof.ts +++ b/src/import/pprof.ts @@ -21,7 +21,7 @@ function getSampleTypeIndex(profile: perftools.profiles.Profile): number { return fallback } - const idx = sampleTypes.findIndex(e => e.type === dflt) + const idx = sampleTypes.findIndex(e => e.type != null && e.type.toString() === dflt.toString()) if (idx === -1) { return fallback } @@ -38,24 +38,33 @@ export function importAsPprofProfile(rawProfile: ArrayBuffer): Profile | null { return null } - function i32(n: number | Long): number { - return typeof n === 'number' ? n : (n as Long).low + function idKey(n: number | Long): string { + return n.toString() } function stringVal(key: number | Long): string | null { - return protoProfile.stringTable[i32(key)] || null + const index = Number(key.toString()) + return Number.isSafeInteger(index) && index >= 0 + ? protoProfile.stringTable[index] || null + : null } - const frameInfoByFunctionID = new Map() + function sourceLine(value: number | Long | null | undefined): number | undefined { + if (value == null) return undefined + const line = Number(value.toString()) + return Number.isSafeInteger(line) && line > 0 ? line : undefined + } + + const frameInfoByFunctionID = new Map() function frameInfoForFunction(f: perftools.profiles.IFunction): FrameInfo | null { const {name, filename, startLine} = f const nameString = (name != null && stringVal(name)) || '(unknown)' const fileNameString = filename != null ? stringVal(filename) : null - const line = startLine != null ? +startLine : null + const line = sourceLine(startLine) - const key = `${nameString}:${fileNameString}:${line}` + const key = JSON.stringify([nameString, fileNameString, line]) const frameInfo: FrameInfo = { key, @@ -77,7 +86,7 @@ export function importAsPprofProfile(rawProfile: ArrayBuffer): Profile | null { if (f.id) { const frameInfo = frameInfoForFunction(f) if (frameInfo != null) { - frameInfoByFunctionID.set(i32(f.id), frameInfo) + frameInfoByFunctionID.set(idKey(f.id), frameInfo) } } } @@ -101,24 +110,28 @@ export function importAsPprofProfile(rawProfile: ArrayBuffer): Profile | null { if (lastLine == null) return null if (lastLine.functionId) { - let funcFrame = frameInfoByFunctionID.get(i32(lastLine.functionId)) - const sourceLine = lastLine.line instanceof Long ? lastLine.line.toNumber() : lastLine.line - if (sourceLine && sourceLine > 0 && funcFrame != null) { - funcFrame.line = sourceLine + const funcFrame = frameInfoByFunctionID.get(idKey(lastLine.functionId)) + if (!funcFrame) return null + const locationLine = sourceLine(lastLine.line) ?? funcFrame.line + // Each location owns its source identity; shared function definitions + // must not be overwritten by another location's line number. + return { + ...funcFrame, + key: JSON.stringify([funcFrame.name, funcFrame.file, locationLine]), + line: locationLine, } - return funcFrame || null } else { return null } } - const frameByLocationID = new Map() + const frameByLocationID = new Map() for (let l of protoProfile.location) { if (l.id != null) { const frameInfo = frameInfoForLocation(l) if (frameInfo) { - frameByLocationID.set(i32(l.id), frameInfo) + frameByLocationID.set(idKey(l.id), frameInfo) } } } @@ -153,7 +166,7 @@ export function importAsPprofProfile(rawProfile: ArrayBuffer): Profile | null { } for (let s of protoProfile.sample) { - const stack = s.locationId ? s.locationId.map(l => frameByLocationID.get(i32(l))) : [] + const stack = s.locationId ? s.locationId.map(l => frameByLocationID.get(idKey(l))) : [] stack.reverse() if (s.value == null || s.value.length <= sampleTypeIndex) { diff --git a/src/import/profile-loader.test.ts b/src/import/profile-loader.test.ts index 9ab0fe10..88c9909f 100644 --- a/src/import/profile-loader.test.ts +++ b/src/import/profile-loader.test.ts @@ -10,6 +10,13 @@ import {snapshotTransferables} from '../lib/profile-snapshot' import {TraceWorkerRequest, TraceWorkerResponse} from './trace-worker-protocol' import {exportProfileGroup} from '../lib/file-format' import {importProfileGroupFromText} from './index' +import * as importerModule from './index' +import * as fs from 'fs' +import * as pako from 'pako' +import {Profile} from '../lib/profile' +import {perftools} from './profile.proto.js' +import {JAVA_DUMP_ZIP_LIMITS} from './java-thread-dump-zip' +import JSZip from 'jszip' const json = JSON.stringify([ {ph: 'X', name: 'outer', pid: 'process', tid: 'thread', ts: 10, dur: 10}, @@ -104,6 +111,33 @@ test('worker creation blocked by the host falls back without detaching input', a expect(input.byteLength).toBeGreaterThan(0) }) +test('small ZIP main-thread imports retain cancellable Parsing progress and honor abort', async () => { + const archive = new JSZip() + archive.file( + 'snapshot.jstack', + fs.readFileSync('sample/profiles/java-thread-dump/synthetic.jstack'), + ) + const input = await archive.generateAsync({type: 'arraybuffer', compression: 'DEFLATE'}) + expect(input.byteLength).toBeLessThan(1024 * 1024) + const controller = new AbortController() + const workerFactory = jest.fn(() => { + throw new Error('Small ZIP must not need a worker') + }) + const onProgress = jest.fn((update: {phase: string; canCancel: boolean}) => { + if (update.phase === 'Parsing profile' && update.canCancel) controller.abort() + }) + await expect( + importProfileBuffer('small.zip', input, { + signal: controller.signal, + onProgress, + workerFactory, + }), + ).rejects.toMatchObject({name: 'AbortError'}) + expect(workerFactory).not.toHaveBeenCalled() + expect(onProgress).toHaveBeenCalledWith({phase: 'Parsing profile', canCancel: true}) + expect(input.byteLength).toBeGreaterThan(0) +}) + test('unrecognized worker input returns its original buffer for normal detection', async () => { const response = await processTraceInWorker( {name: 'unknown.trace', input: new TextEncoder().encode('not a profile').buffer as ArrayBuffer}, @@ -137,7 +171,176 @@ test('worker errors settle the import, and native .cpuprofile/.heapprofile files expect(shouldUseTraceWorker('large.cpuprofile', 2 ** 20)).toBe(true) expect(shouldUseTraceWorker('large.heapprofile', 2 ** 20)).toBe(true) expect(shouldUseTraceWorker('tiny.json', 10)).toBe(false) - expect(shouldUseTraceWorker('large.txt', 2 ** 25)).toBe(false) + expect(shouldUseTraceWorker('large.txt', 2 ** 25)).toBe(true) +}) + +test('all large inputs use worker detection regardless of extension', () => { + for (const name of [ + 'capture.linux-perf.txt', + 'stacks.collapsed', + 'dump.jstack', + 'recording.jfr', + 'heap.pprof', + 'callgrind.out', + 'renamed', + 'unknown.bin.gz', + ]) { + expect(shouldUseTraceWorker(name, 1024 * 1024)).toBe(true) + expect(shouldUseTraceWorker(name, 1024 * 1024 - 1)).toBe(false) + } +}) + +test('large renamed folded input transfers ownership and retains cancellable worker progress', async () => { + const text = 'main;work 1\n'.repeat(100_000) + const input = new TextEncoder().encode(text).buffer as ArrayBuffer + const worker = new FakeWorker() + const progress = jest.fn() + const group = await importProfileBuffer('renamed-recording', input, { + workerFactory: factory(worker), + onProgress: progress, + }) + expect(input.byteLength).toBe(0) + expect(group!.profiles[0].getTotalWeight()).toBe(100_000) + expect([...group!.profiles[0].getFrames()].map(frame => frame.name)).toEqual(['main', 'work']) + expect(wereSymbolsLoaded(group!)).toBe(true) + expect(progress.mock.calls.some(([value]) => value.phase === 'Parsing profile')).toBe(true) + expect(progress.mock.calls.every(([value]) => value.canCancel)).toBe(true) + expect(worker.terminate).toHaveBeenCalled() +}) + +test('generic worker processing preserves perf, Java text, JFR and pprof fixture exports', async () => { + const fixtures = [ + ['sample/profiles/linux-perf/leaf-only.linux-perf.txt', 'renamed-perf'], + ['sample/profiles/java-thread-dump/synthetic.jstack', 'renamed-dump'], + ['sample/profiles/java-flight-recorder/heavy.jfr', 'renamed-jfr'], + ['sample/profiles/pprof/simple.prof', 'renamed-pprof'], + ] + for (const [path, name] of fixtures) { + const input = new Uint8Array(fs.readFileSync(path)).buffer as ArrayBuffer + const expected = await importerModule.importProfilesFromArrayBuffer(name, input) + const response = await processTraceInWorker({name, input}, () => {}) + expect(response.type).toBe('result') + if (response.type !== 'result') throw new Error('Expected worker fixture result') + const transferred = structuredClone(response.group, { + transfer: snapshotTransferables(response.group), + }) + const profiles: Profile[] = [] + for (const snapshot of transferred.profiles) profiles.push(await Profile.fromSnapshot(snapshot)) + expect(exportProfileGroup({...transferred, profiles})).toEqual(exportProfileGroup(expected!)) + } +}) + +test('large binary pprof input uses the worker instead of extension-based main-thread parsing', async () => { + const input = perftools.profiles.Profile.encode({ + stringTable: ['', 'samples', 'count', 'work', 'unused'.repeat(220_000)], + sampleType: [{type: 1, unit: 2}], + function: [{id: 1, name: 3}], + location: [{id: 1, line: [{functionId: 1}]}], + sample: [{locationId: [1], value: [7]}], + }).finish().buffer as ArrayBuffer + expect(input.byteLength).toBeGreaterThanOrEqual(1024 * 1024) + const worker = new FakeWorker() + const group = await importProfileBuffer('profile.bin', input, {workerFactory: factory(worker)}) + expect(input.byteLength).toBe(0) + expect(group!.profiles[0].getTotalWeight()).toBe(7) + expect([...group!.profiles[0].getFrames()].map(frame => frame.name)).toEqual(['work']) +}) + +test('new generic File formats are read once without opening a streaming probe', async () => { + const file = new File(['main;work 1\n'.repeat(100_000)], 'large.collapsed') + const read = jest.spyOn(file, 'arrayBuffer') + const stream = jest.spyOn(file, 'stream') + const response = await processTraceInWorker({name: file.name, input: file}, () => {}) + expect(response.type).toBe('result') + expect(read).toHaveBeenCalledTimes(1) + expect(stream).not.toHaveBeenCalled() +}) + +test('oversized renamed ZIP Files fail before worker streaming or full-buffer reads', async () => { + const prefix = new Blob([new Uint8Array([0x50, 0x4b, 3, 4])]) + const input = { + name: 'renamed-recording', + size: JAVA_DUMP_ZIP_LIMITS.maxCompressedBytes + 1, + slice: jest.fn(() => prefix), + arrayBuffer: jest.fn(() => Promise.reject(new Error('Do not materialize oversized ZIP'))), + stream: jest.fn(() => { + throw new Error('Do not open oversized ZIP stream') + }), + } + await expect( + processTraceInWorker({name: input.name, input: input as unknown as File}, () => {}), + ).rejects.toThrow('compressed input exceeds its limit') + expect(input.slice).toHaveBeenCalledWith(0, 4) + expect(input.arrayBuffer).not.toHaveBeenCalled() + expect(input.stream).not.toHaveBeenCalled() +}) + +test('gzip generic worker formats decompress once and preserve observations', async () => { + const nativeDecompression = DecompressionStream + const decompress = jest.spyOn(globalThis, 'DecompressionStream') + decompress.mockImplementation(format => new nativeDecompression(format)) + try { + const input = new Uint8Array( + pako.gzip(fs.readFileSync('sample/profiles/java-thread-dump/synthetic.jstack')), + ).buffer as ArrayBuffer + const response = await processTraceInWorker({name: 'capture.jstack.gz', input}, () => {}) + expect(response.type).toBe('result') + if (response.type === 'result') + expect(response.group.profiles.map(profile => profile.totalWeight)).toEqual([2, 2, 1]) + expect(decompress).toHaveBeenCalledTimes(1) + expect(decompress).toHaveBeenCalledWith('gzip') + } finally { + decompress.mockRestore() + } +}) + +test('DOM-dependent generic fallback returns transferred bytes before main-thread retry', async () => { + const text = 'main;work 1\n'.repeat(100_000) + const input = new TextEncoder().encode(text).buffer as ArrayBuffer + const parse = jest.spyOn(importerModule, 'importProfilesFromArrayBuffer') + parse.mockRejectedValueOnce(new ReferenceError('DOMParser is not defined')) + const worker = new FakeWorker() + try { + const group = await importProfileBuffer('renamed-input', input, { + workerFactory: factory(worker), + }) + expect(input.byteLength).toBe(0) + expect(group!.profiles[0].getTotalWeight()).toBe(100_000) + expect(parse).toHaveBeenCalledTimes(2) + expect(wereSymbolsLoaded(group!)).toBe(false) + expect(worker.terminate).toHaveBeenCalled() + } finally { + parse.mockRestore() + } +}) + +test('a worker cannot silently fall back to a detached input buffer', async () => { + const worker = new FakeWorker() + worker.postMessage = (request, transfers) => { + structuredClone(request, {transfer: transfers}) + worker.onmessage?.({data: {type: 'fallback'}} as MessageEvent) + } + await expect( + importProfileBuffer('capture.bin', buffer(), {workerFactory: factory(worker)}), + ).rejects.toThrow('return ownership') + expect(worker.terminate).toHaveBeenCalled() +}) + +test('cancellation while a non-JSON worker begins parsing rejects and terminates it', async () => { + const controller = new AbortController() + const worker = new FakeWorker() + const input = new TextEncoder().encode('main;work 1\n'.repeat(100_000)).buffer as ArrayBuffer + await expect( + importProfileBuffer('large.collapsed', input, { + workerFactory: factory(worker), + signal: controller.signal, + onProgress: progress => { + if (progress.phase === 'Parsing profile') controller.abort() + }, + }), + ).rejects.toMatchObject({name: 'AbortError'}) + expect(worker.terminate).toHaveBeenCalled() + expect(input.byteLength).toBe(0) }) test('File imports can return original bytes for the normal detector', async () => { diff --git a/src/import/profile-loader.ts b/src/import/profile-loader.ts index 3e6cd6bb..f0b2abe3 100644 --- a/src/import/profile-loader.ts +++ b/src/import/profile-loader.ts @@ -18,8 +18,11 @@ function checkAbort(signal?: AbortSignal) { if (signal?.aborted) throw new DOMException('Import cancelled', 'AbortError') } -export function shouldUseTraceWorker(name: string, size: number): boolean { - return size >= 1024 * 1024 && /\.(?:json|trace|cpuprofile|heapprofile)(?:\.gz)?$/i.test(name) +export function shouldUseTraceWorker(_name: string, size: number): boolean { + // Detection belongs in the worker's existing generic importer. Extension + // allowlists leave large perf/folded/JFR/pprof files (and renamed captures) + // blocking the UI, even though those importers are already worker-safe. + return size >= 1024 * 1024 } async function loadWithWorker( @@ -72,8 +75,11 @@ async function loadWithWorker( const message = event.data if (message.type === 'progress') options.onProgress?.({phase: message.phase, canCancel: true}) - else if (message.type === 'fallback') resolve(message.buffer || null) - else if (message.type === 'error') reject(new Error(message.message)) + else if (message.type === 'fallback') { + if (!message.buffer && input instanceof ArrayBuffer && input.byteLength === 0) + throw new Error('The profile importer did not return ownership of the input buffer.') + resolve(message.buffer || null) + } else if (message.type === 'error') reject(new Error(message.message)) else { // Release the worker's parsed input and trees before creating UI objects. worker.terminate() @@ -111,7 +117,10 @@ export async function importProfileFile( options.onProgress?.({phase: 'Reading profile', canCancel: false}) const module = await importer() checkAbort(options.signal) - return module.importProfilesFromFile(file) + return module.importProfilesFromFile(file, { + signal: options.signal, + onProgress: options.onProgress, + }) } export async function importProfileBuffer( @@ -129,7 +138,10 @@ export async function importProfileBuffer( options.onProgress?.({phase: 'Reading profile', canCancel: false}) const module = await importer() checkAbort(options.signal) - return module.importProfilesFromArrayBuffer(name, buffer) + return module.importProfilesFromArrayBuffer(name, buffer, { + signal: options.signal, + onProgress: options.onProgress, + }) } export async function importProfileBase64( diff --git a/src/import/trace-worker.ts b/src/import/trace-worker.ts index 7588463c..0f2acb52 100644 --- a/src/import/trace-worker.ts +++ b/src/import/trace-worker.ts @@ -2,13 +2,14 @@ import {importProfilesFromArrayBuffer} from './index' import {TraceWorkerRequest, TraceWorkerResponse, ImportPhase} from './trace-worker-protocol' import {snapshotTransferables} from '../lib/profile-snapshot' import {tryImportDurationTrace} from './streaming-trace' +import {checkJavaThreadDumpZipFileSize} from './java-thread-dump-zip' export async function processTraceInWorker( request: TraceWorkerRequest, progress: (phase: ImportPhase) => void, ): Promise { progress('Reading profile') - progress('Streaming profile') + if (!(request.input instanceof ArrayBuffer)) await checkJavaThreadDumpZipFileSize(request.input) let group = await tryImportDurationTrace(request.input, request.name, { onProgress: () => progress('Streaming profile'), }) @@ -18,7 +19,9 @@ export async function processTraceInWorker( buffer = request.input instanceof ArrayBuffer ? request.input : await request.input.arrayBuffer() progress('Parsing profile') - group = await importProfilesFromArrayBuffer(request.name, buffer) + group = await importProfilesFromArrayBuffer(request.name, buffer, { + onProgress: update => progress(update.phase), + }) } } catch (error) { // Some host-specific formats need DOM APIs unavailable in a worker. Return diff --git a/src/views/help.tsx b/src/views/help.tsx index 01323c97..63d3590d 100644 --- a/src/views/help.tsx +++ b/src/views/help.tsx @@ -37,9 +37,10 @@ function HelpDialog(props: {onClose(): void}) {

Supported formats include speedscope JSON, Chrome and Firefox profiles, duration traces - (plain or gzip), perf, pprof, collapsed stacks, Java Flight Recorder and traditional - Java thread dumps. Extract ZIP archives before importing. Java dumps count observed - stacks, rather than elapsed time. + (plain or gzip), perf, pprof, collapsed stacks, Java Flight Recorder, classic Android + method traces and traditional Java thread dumps. Java dumps can be plain text or a + bounded ZIP of snapshots from one JVM. Extract unsupported archive variants first. Java + dumps count observed stacks, rather than elapsed time.