feat(web): accept HEIC photos, transcoding them to JPEG on the server - #22
feat(web): accept HEIC photos, transcoding them to JPEG on the server#22Antisophy wants to merge 1 commit into
Conversation
An iPhone camera photo is HEIC. When one is dragged or pasted into the composer, Gecko has no MIME entry for .heic, so the File arrives with an empty type; the composer then dropped it without a word, since nothing outside image/* was ever attached. A photo that vanishes with no trace is indistinguishable from a broken composer, which is how it presented. A browser that does type the file image/heic attached it as-is, producing a send the agent APIs reject. Client: sniff the leading bytes when the browser could not type a file (JPEG, PNG, GIF, WebP and the HEIC/HEIF ftyp brands), accept HEIC, show a labelled chip in place of the preview the browser cannot decode, and name any file that is refused. Files the browser typed as a supported format stay on the synchronous path. Server: a new cydo.web.images normalizes image blocks before the create_task and message handlers see them, transcoding HEIC to JPEG with ImageMagick, resolved through PATH at call time so a missing install degrades to a clear client-visible error. The conversion is asynchronous, since the event loop is shared by every client; when nothing needs converting the handler runs synchronously so ordering is unchanged, and a conversion failure is reported to the client rather than forwarded to the agent. Each conversion is issued from a helper call whose parameter is a fresh variable per call, so several photos in one message each land in their own JPEG block. The unit test drives real HEIC bytes through the pipeline, and imagemagick joins the unittests check environment so it runs in the sandbox rather than skipping itself.
|
I think this is definitely worth fixing, but I am wondering if the fix is in the wrong place. In principle, the "set of used image formats" should generally be owned by the system that manages/shares them, rather than the consumer; us endeavoring to support every image format out there, including future formats, may be a promise or endeavor we would be hard pressed to stick to. From a bit of quick research, it looks like there are paths out there that have started to form to solving this problem at scale, e.g. For the first one, there is one much simpler path - we can transcode the HEIC image client-side via Safari's native support for it. I'm also wondering if client side transcoding is preferable even if we need to bundle a JS/WASM JPEG/HEIC library, since it solves the cross platform and distribution aspects of the ImageMagick dependency. In any case, what use case do we care about - just Firefox on PC with HEIC images, or perhaps primarily iOS with CyDo in Safari? |
| case "create_task": handleCreateTaskMsg(ws, json); break; | ||
| case "create_task": withNormalizedImages(ws, json, (WsMessage m) { handleCreateTaskMsg(ws, m); }); break; |
There was a problem hiding this comment.
Note on the current implementation: This is a very specific shim added to a very generic place. Though I appreciate that this may be the minimal diff, it does degrade overall cohesion. Could you look into a better structured way to apply this, refactoring if needed - maybe a path shared by both handleCreateTaskMsg and handleUserMessage?
2 participants
An iPhone camera photo is HEIC. When one is dragged or pasted into the composer, Gecko has no MIME entry for .heic, so the File arrives with an empty type; the composer then dropped it without a word, since nothing outside image/* was ever attached. A photo that vanishes with no trace is indistinguishable from a broken composer, which is how it presented. A browser that does type the file image/heic attached it as-is, producing a send the agent APIs reject.
Client: sniff the leading bytes when the browser could not type a file (JPEG, PNG, GIF, WebP and the HEIC/HEIF ftyp brands), accept HEIC, show a labelled chip in place of the preview the browser cannot decode, and name any file that is refused. Files the browser typed as a supported format stay on the synchronous path.
Server: a new cydo.web.images normalizes image blocks before the create_task and message handlers see them, transcoding HEIC to JPEG with ImageMagick, resolved through PATH at call time so a missing install degrades to a clear client-visible error. The conversion is asynchronous, since the event loop is shared by every client; when nothing needs converting the handler runs synchronously so ordering is unchanged, and a conversion failure is reported to the client rather than forwarded to the agent. Each conversion is issued from a helper call whose parameter is a fresh variable per call, so several photos in one message each land in their own JPEG block. The unit test drives real HEIC bytes through the pipeline, and imagemagick joins the unittests check environment so it runs in the sandbox rather than skipping itself.