Skip to content

feat(symbols): skip uploading symbols LaunchDarkly already has - #765

Merged
abelonogov-ld merged 3 commits into
mainfrom
andrey/symbols-upload-dedup
Jul 31, 2026
Merged

feat(symbols): skip uploading symbols LaunchDarkly already has#765
abelonogov-ld merged 3 commits into
mainfrom
andrey/symbols-upload-dedup

Conversation

@abelonogov-ld

@abelonogov-ld abelonogov-ld commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

A rebuild that changes one module still re-sent every symbol file, so ldcli symbols upload took time proportional to the whole build rather than to what actually changed. That is most of the wall clock on a large R8 mapping or a fat dSYM.

upload now asks the backend for skip_existing. Any requested key the backend answers with an empty string is reported as already uploaded and skipped, and the summary becomes Successfully uploaded all symbols (N already present). This covers all three upload paths: the generic React Native/Android loop, Apple dSYMs, and Flutter.

Dedup is on by default. It only applies where the key's id is derived from the bytes being uploaded, which the backend decides (launchdarkly/observability#1609): a Version Lane copy still overwrites, and so does a source bundle, since those are keyed by the id of the mapping or image they explain rather than by their own contents. --no-skip-existing forces a full re-upload, for repairing a truncated or corrupt stored object.

Because skipping is a change in what upload does by default, and the closing "Successfully uploaded all symbols" reads the same whether bytes were sent or not, a run that skipped anything now says so on stderr and names the flag. Someone re-uploading to repair a stored object would otherwise take a no-op run for a successful one. The notice is transitional and can come out after a release or two; the counts stay on stdout so scripts parsing them are unaffected.

Backward compatibility: a backend that predates the argument rejects the whole query at validation time, so the request is retried once without it. That keeps a freshly updated CLI working against an older or self-hosted deployment, at the cost of uploading bytes it might not have needed. Only the unknown-argument case is retried, so a credential or project error still reaches the user instead of being silently downgraded.

Depends on launchdarkly/observability#1609, which adds the argument. This should land after that deploys; until then the fallback path keeps upload working unchanged.

How did you test this change?

New tests in cmd/symbols/upload_dedup_test.go, driving getSymbolUploadUrls against an httptest backend that records each request:

  • the dedup document and skip_existing: true are sent when enabled, and neither is sent with --no-skip-existing
  • an Unknown argument "skip_existing" response triggers exactly one retry, and the retry drops the argument
  • any other GraphQL error is returned without a retry
  • alreadyUploaded treats only the empty string as a skip
  • the summary prints the skipped count on stdout and the notice on stderr, and stays silent on stderr when nothing was skipped

go build ./..., go vet ./cmd/symbols/..., and go test ./cmd/symbols/... pass.


Note

Medium Risk
Changes default upload behavior and presigned-URL handling across all symbol types; mis-dedup could leave stale symbols, mitigated by opt-out flag and backend digest checks for non-content-keyed bundles.

Overview
ldcli symbols upload skips re-sending symbols the backend already stores, so repeat uploads after unchanged builds avoid large PUTs. Dedup is on by default across the generic React Native/Android path, Apple dSYM, and Flutter flows. The CLI asks get_symbol_upload_urls_ld with skip_existing; an empty presigned URL means skip that key. --no-skip-existing forces a full re-upload for repair scenarios.

Source bundles (Apple/Android --include-sources) are keyed off the mapping/image id, not bundle bytes, so skips use parallel MD5 digests (S3 ETag form) so the backend can match stored content. Flutter Version-lane copies still always get URLs so --app-version overwrites keep working; only Id-lane keys can skip.

Older backends that reject the new GraphQL args get one retry without dedup (only on unknown-argument errors). Completion messaging reports (N already present) on stdout and a stderr note naming --no-skip-existing when anything was skipped.

Reviewed by Cursor Bugbot for commit a7418b3. Bugbot is set up for automated code reviews on this repo. Configure here.

A rebuild that changes one module still re-sent every symbol file, so upload
time tracked the size of the whole build rather than what actually changed.

`upload` now asks for skip_existing, and any key answered with an empty string
is reported as already uploaded and skipped. Only content-addressed symbols-id
keys are ever answered that way, so a Version Lane copy still overwrites.

A backend that predates the argument rejects the query outright, so the request
is retried once without it. That keeps a freshly updated CLI working against an
older or self-hosted deployment, at the cost of sending bytes it might not have
needed.

--no-skip-existing forces a re-upload, for repairing a corrupt stored object.

Co-authored-by: Cursor <cursoragent@cursor.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 401a4b5. Configure here.

Comment thread cmd/symbols/upload.go
Comment thread cmd/symbols/upload.go
abelonogov-ld and others added 2 commits July 30, 2026 16:15
Skipping already-stored symbols is a change in what `upload` does by default, and
the closing "Successfully uploaded all symbols" reads the same whether bytes were
sent or not. Anyone re-uploading to repair a stored object would take a no-op run
for a successful one, so a run that skipped anything now says so on stderr and
names --no-skip-existing. The notice is transitional; the counts stay on stdout so
scripts parsing them are unaffected.

Also corrects the source-bundle comments: bundles are keyed by the id of the
mapping or image they explain, so they are never skipped (enforced backend-side).

Co-authored-by: Cursor <cursoragent@cursor.com>
…s id

Source bundles are keyed by the id of the mapping or image they explain, so the
backend can't settle them by existence and they were re-sent on every run with
--include-sources. They now travel with the MD5 of what is about to be uploaded,
and are skipped when that matches what is stored.

The digest costs nothing to compute: both platforms build the bundle in memory
before uploading it. Nothing else sends one, so a large mapping is still settled
by existence alone rather than being read back off disk to hash.

The old-backend retry now triggers on either argument name, since validation
names whichever one it reached first.

Co-authored-by: Cursor <cursoragent@cursor.com>
@abelonogov-ld
abelonogov-ld requested a review from Vadman97 July 30, 2026 23:48

@Vadman97 Vadman97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice optimization!

@abelonogov-ld
abelonogov-ld merged commit 5dfc7a7 into main Jul 31, 2026
8 checks passed
@abelonogov-ld
abelonogov-ld deleted the andrey/symbols-upload-dedup branch July 31, 2026 00:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants