Auto upload: ideas for predictable behaviour, duplicate-free re-scans and recovery #17801

How to use GitHub

• Please use the 👍 reaction to show that you are interested into the same feature.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Is your feature request related to a problem? Please describe.

Auto upload with "keep original" works most of the time. When it fails, it's hard to notice and hard to recover from:

• A phone silently stops uploading, and nobody notices for weeks (Auto upload has stopped working #14945, Auto upload stopped working #16334, Auto upload photos is unreliable #17212).
• The only way to catch up is "Also upload existing files". That re-uploads every photo on the phone that the app has no record of, including photos already uploaded and later moved or deleted on the server (After update app tries to reupload old photos #16348, Add cancel upload in case of name collision in autoupload  #6911).
• Identical files become conflicts, or IMG (2).jpg copies with "Rename" (Skip uploading of duplicate files #4255, Automatically qualify upload conflicts on file content (e.g. hash comparison) #8948, Auto Upload keeps creating more and more copies of all my files #12861).
• Re-adding a folder, changing its target, or reinstalling loses the app's record of what it uploaded (Track auto upload files when original file is "kept in original folder" #12967, photos are not uploaded, after changing remote directory #16802, Ability to backup / restore app settings #11090).

Reliability has improved a lot over the last year: the new auto upload worker, the rescan worker in 35.0.0, the stuck-worker fix coming in 35.0.2, and the work coordinated in #16550. Thank you for that. Hopefully silent stops will become rare. Uploads can still fail for reasons outside the app's control (OEM battery management, permissions, a full server, expired credentials), so it remains important that failures are visible and that recovery is predictable and doesn't re-upload everything. That's the focus of this proposal; it doesn't replace the ongoing bug fixing.

Many issues and PRs touch parts of this. This issue tries to connect them under one model of how I believe auto upload should behave, so each part can be discussed and built separately. It follows the increment format of #285.

My ideas depend on a few new concepts, such as a "window" deciding which files to consider for upload and an improved upload-log. These are explained in more detail below.

Describe the solution you'd like

Scope: upload only. No two-way sync and no syncing of deletions (that's #17570). Local files are only changed by the existing "original file" action.

Principles:

1. A file is uploaded once per account, not once per folder rule. Re-adding or editing a rule never re-uploads a file that was already uploaded.
2. Files that are already on the server are always skipped. Skip / Rename / Overwrite / Ask only apply when the existing file is different (same name, different content).
3. Every outcome is visible. Each file ends in a named state. Stalls and permanent failures produce a notification, never silence.
4. Recovery is a scan, not a reset. Users never need to delete a rule or clear app data to get uploads going again.

Per-file decision:

candidate (in rule folder, captured inside the "upload from" window - see below)
 ├─ in this account's upload log? ──────────── yes → skip
 └─ target path free on server? ────────────── yes → upload, then log it
     └─ exists: identical? ─────────────────── yes → skip, log it
                 different → collision policy (default: Ask)

I suggest changing a few settings slightly, not adding a lot of new ones:

• "Also upload existing files" becomes "Upload files from: All / ‹date›", compared against the capture date. It's never reset implicitly.
• The collision setting is labelled "If a different file with the same name exists".

Recovery: one new menu entry, "Scan now…", with a From date and a mode:

• This idea is based on Add menu item to force scan files for auto-upload #17462 but instead of a simple button, I think we should add a date-picker and mode-selector (implemented in 2 separate increments below to not block the current work):
  • Re-scan (default): respects the upload log. Safe to run any time.
  • Missing: ignores the log and skips files identical on the server. Warns that moved or deleted files will come back.
  • All: ignores the log and uploads everything within the window (conflicts resolved according to settings).

The time window:

Today, the only options for selecting files to consider for upload is "All" ("Upload existing files" enabled) or "From the config was saved" (The timestamp the app uses when "Upload existing files" are disabled.
My idea is to introduce a simple way to set a time-window to select which files to consider for uploads.

If uploads have failed for a period of time, one might want to resume uploads from a known date when the you know the upload last worked without worrying that hundreds of old files are re-uploaded. Or you can manually stop uploads for a period, and then re-enable them later, and decide if you want to upload all or only parts of the backlog.

The improved upload log:

My idea is an upload log that is computed on the device and keyed by content hash, so it works with any server. Server-side checksums (nextcloud/server#11138, nextcloud/server#56057) would improve the identity check but are not a prerequisite. The server already accepts OC-Checksum (nextcloud/server#51729).

Increments

I have split this up in 6 increments. Increments 1, 2 and 3 are independent of each other and work with what the app has today. Increments 4–6 build on each other.

• 1. Visibility: "last successful upload" per rule, a stall notification, and failed files that don't block the queue. No server changes needed. Relates to 'Processing' indicator for AutoUpload #8154 ("Last auto upload ran at xx:yy"), Failed auto-uploads are not retried #14233, ignored files stuck in upload list (delete button does nothing) #16639, Unable to cancel stalled upload #16207, Improve error infos on upload failure #260. The battery-saver banner (Indicate skipped auto-uploads when device is in battery saver mode #15717) is the pattern to reuse.
  Depends on: nothing.
• 2. Always skip identical files, under every collision policy, and send OC-Checksum on upload. Relates to Automatically qualify upload conflicts on file content (e.g. hash comparison) #8948, Skip uploading of duplicate files #4255, Conflicts for identical files #10194, Auto Upload keeps creating more and more copies of all my files #12861, and the earlier attempt feat: enhance file upload collision handling and add streaming folder… #16231.
  Depends on: nothing. Uses checksums where the server has them, and today's metadata comparison otherwise.
• 3. "Scan now…" with a From date, Re-scan mode only. Relates to Auto upload - Button to manually scan files in the file system and re-upload missing files. #15782, Add option to force rescan auto upload folders and upload any missing files #15051, add a "force automatic uploads now" button #15887 and draft Add menu item to force scan files for auto-upload #17462.
  Depends on: nothing. It respects the app's existing record of uploaded files, and the date limits any re-upload to the chosen window if that record has been lost.
• 4. A per-file upload log keyed by content hash, kept across rule changes, and covering files that were skipped. Relates to Track auto upload files when original file is "kept in original folder" #12967 (and Fix missing local path for autouploaded files #17522).
  Depends on: nothing, but it's the foundation for 5 and 6.
• 5. "Scan now…" modes Missing and All. They're added to the dialog from increment 3.
  Depends on: 3 and 4. Both modes are defined as "ignore the log", so they need a reliable log to ignore.
• 6. A server copy of the log, restored on a new device or reinstall. Android backup is disabled on purpose (see the maintainer comments in Ability to backup / restore app settings #11090), so a per-device log file in the user's own Nextcloud is used instead. Relates to Ability to backup / restore app settings #11090 and Add cancel upload in case of name collision in autoupload  #6911.
  Depends on: 4.

Describe alternatives you've considered

• A plain "force upload" button (add a "force automatic uploads now" button #15887, Add menu item to force scan files for auto-upload #17462): without a date window and a reliable log, it either re-uploads everything or misses the gap.
• Relying on server presence only: it can't know about files that were moved or deleted after upload.
• Android backup or migration of the app database: it's disabled, many users avoid Google backup, and migration tools change DATE_ADDED, mtimes and paths.

Additional context

I'm a self-hosting user. Our family's phones auto-upload to private folders, and we then move the keepers into a shared folder. That's how we ran into these problems. I'm not an Android developer, so I can't vouch for code changes myself. I can help with testing builds, collecting logs, and refining the expected behaviour above. The code reading behind this was done with AI assistance (Claude); I checked the conclusions against the tracker.