feat(symbols): skip uploading symbols LaunchDarkly already has - #765
Merged
Conversation
A rebuild that changes one module still re-sent every symbol file, so upload time tracked the size of the whole build rather than what actually changed. `upload` now asks for skip_existing, and any key answered with an empty string is reported as already uploaded and skipped. Only content-addressed symbols-id keys are ever answered that way, so a Version Lane copy still overwrites. A backend that predates the argument rejects the query outright, so the request is retried once without it. That keeps a freshly updated CLI working against an older or self-hosted deployment, at the cost of sending bytes it might not have needed. --no-skip-existing forces a re-upload, for repairing a corrupt stored object. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 401a4b5. Configure here.
Skipping already-stored symbols is a change in what `upload` does by default, and the closing "Successfully uploaded all symbols" reads the same whether bytes were sent or not. Anyone re-uploading to repair a stored object would take a no-op run for a successful one, so a run that skipped anything now says so on stderr and names --no-skip-existing. The notice is transitional; the counts stay on stdout so scripts parsing them are unaffected. Also corrects the source-bundle comments: bundles are keyed by the id of the mapping or image they explain, so they are never skipped (enforced backend-side). Co-authored-by: Cursor <cursoragent@cursor.com>
…s id Source bundles are keyed by the id of the mapping or image they explain, so the backend can't settle them by existence and they were re-sent on every run with --include-sources. They now travel with the MD5 of what is about to be uploaded, and are skipped when that matches what is stored. The digest costs nothing to compute: both platforms build the bundle in memory before uploading it. Nothing else sends one, so a large mapping is still settled by existence alone rather than being read back off disk to hash. The old-backend retry now triggers on either argument name, since validation names whichever one it reached first. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
A rebuild that changes one module still re-sent every symbol file, so
ldcli symbols uploadtook time proportional to the whole build rather than to what actually changed. That is most of the wall clock on a large R8 mapping or a fat dSYM.uploadnow asks the backend forskip_existing. Any requested key the backend answers with an empty string is reported as already uploaded and skipped, and the summary becomesSuccessfully uploaded all symbols (N already present). This covers all three upload paths: the generic React Native/Android loop, Apple dSYMs, and Flutter.Dedup is on by default. It only applies where the key's id is derived from the bytes being uploaded, which the backend decides (launchdarkly/observability#1609): a Version Lane copy still overwrites, and so does a source bundle, since those are keyed by the id of the mapping or image they explain rather than by their own contents.
--no-skip-existingforces a full re-upload, for repairing a truncated or corrupt stored object.Because skipping is a change in what
uploaddoes by default, and the closing "Successfully uploaded all symbols" reads the same whether bytes were sent or not, a run that skipped anything now says so on stderr and names the flag. Someone re-uploading to repair a stored object would otherwise take a no-op run for a successful one. The notice is transitional and can come out after a release or two; the counts stay on stdout so scripts parsing them are unaffected.Backward compatibility: a backend that predates the argument rejects the whole query at validation time, so the request is retried once without it. That keeps a freshly updated CLI working against an older or self-hosted deployment, at the cost of uploading bytes it might not have needed. Only the unknown-argument case is retried, so a credential or project error still reaches the user instead of being silently downgraded.
Depends on launchdarkly/observability#1609, which adds the argument. This should land after that deploys; until then the fallback path keeps
uploadworking unchanged.How did you test this change?
New tests in
cmd/symbols/upload_dedup_test.go, drivinggetSymbolUploadUrlsagainst anhttptestbackend that records each request:skip_existing: trueare sent when enabled, and neither is sent with--no-skip-existingUnknown argument "skip_existing"response triggers exactly one retry, and the retry drops the argumentalreadyUploadedtreats only the empty string as a skipgo build ./...,go vet ./cmd/symbols/..., andgo test ./cmd/symbols/...pass.Note
Medium Risk
Changes default upload behavior and presigned-URL handling across all symbol types; mis-dedup could leave stale symbols, mitigated by opt-out flag and backend digest checks for non-content-keyed bundles.
Overview
ldcli symbols uploadskips re-sending symbols the backend already stores, so repeat uploads after unchanged builds avoid large PUTs. Dedup is on by default across the generic React Native/Android path, Apple dSYM, and Flutter flows. The CLI asksget_symbol_upload_urls_ldwithskip_existing; an empty presigned URL means skip that key.--no-skip-existingforces a full re-upload for repair scenarios.Source bundles (Apple/Android
--include-sources) are keyed off the mapping/image id, not bundle bytes, so skips use parallel MD5 digests (S3 ETag form) so the backend can match stored content. Flutter Version-lane copies still always get URLs so--app-versionoverwrites keep working; only Id-lane keys can skip.Older backends that reject the new GraphQL args get one retry without dedup (only on unknown-argument errors). Completion messaging reports
(N already present)on stdout and a stderr note naming--no-skip-existingwhen anything was skipped.Reviewed by Cursor Bugbot for commit a7418b3. Bugbot is set up for automated code reviews on this repo. Configure here.