Add picture-in-picture

CHANGELOG.md

@@ -13,4 +13,6 @@ Initial release.
 - Livestreams over WHIP (`publishLivestream`) and WHEP (`receiveLivestream`).
 - Background calls: Android foreground service (`ForegroundService`) and iOS CallKit session (`CallKit`)
   keep microphone capture and playback running while the app is in the background.
+- Picture-in-picture (`PictureInPictureView`, `PictureInPictureOptions`, `PictureInPictureController`),
+  matching the React Native SDK's `RTCPIPView`.
 - Behavioral differences from the Fishjam web SDK are listed in SPEC.md §22.

README.md

@@ -181,6 +181,75 @@ For screen sharing, also add `FOREGROUND_SERVICE_MEDIA_PROJECTION`, add `mediaPr
 </array>
 ```
 
+## Picture-in-picture
+
+`PictureInPictureView` renders a stream and offers its first video track for the system picture-in-picture
+window (iOS 15+, Android 8 / API 26+). Every view with `pipOptions` enabled is offered on its own. On iOS each
+offered view has its own controller. On Android the window belongs to the activity, so the view updated
+most recently, or the one `start()` was called on, is the one shown. It matches the React Native SDK's
+`RTCPIPView`: `pipOptions` takes the same options (`enabled`,
+`preferredSize` (default 1920×1080), `startAutomatically`, `stopAutomatically`, `allowsCameraInBackground`), plus `fallbackLabel`,
+`fallbackBackgroundColor` and `fallbackTextColor`, drawn in the app and in the window when the video track is gone, and
+`PictureInPictureController.start()` / `stop()` replace `startPIP(ref)` / `stopPIP(ref)`.
+
+```dart
+final pip = PictureInPictureController();
+
+PictureInPictureView(
+  mediaStream: remoteTrack.stream,
+  controller: pip,
+  pipOptions: const PictureInPictureOptions(
+    startAutomatically: true,
+    fallbackLabel: 'Left the call',
+    fallbackBackgroundColor: Color(0xFF112233),
+    fallbackTextColor: Color(0xFFFFFFFF),
+  ),
+  onPictureInPictureChange: (active) => setState(() => _inPip = active),
+);
+
+await pip.start(); // e.g. from a button; stop() does nothing on Android
+```
+
+When the video track is gone, the app and the window both draw `fallbackLabel` (default `No video`) in
+`fallbackTextColor` on `fallbackBackgroundColor`. `stopAutomatically` applies on iOS only.
+
+A video CallKit call already keeps the camera running in the background (see above).
+`allowsCameraInBackground: true` (iOS 16+, default `false`, as in the React Native SDK) does the same for an
+app that does not start one. On iOS 18 or newer that needs `voip` in `UIBackgroundModes`. Before iOS 18 it
+also needs the `com.apple.developer.avfoundation.multitasking-camera-access` entitlement.
+
+Keep the call alive in the background with `ForegroundService` and `CallKit` (see above), and add:
+
+**iOS** `Info.plist` (the `audio` entry is already required for background calls):
+
+```xml
+<key>UIBackgroundModes</key>
+<array>
+    <string>audio</string>
+</array>
+```
+
+**Android** `AndroidManifest.xml`, on the activity (keep the `configChanges` the Flutter template sets, so
+the activity is not recreated when picture-in-picture starts):
+
+```xml
+<activity
+    android:name=".MainActivity"
+    android:supportsPictureInPicture="true"
+    android:configChanges="orientation|keyboardHidden|keyboard|screenSize|smallestScreenSize|locale|layoutDirection|fontScale|screenLayout|density|uiMode"
+    ...>
+```
+
+**Android** `MainActivity.kt` must extend `FlutterFragmentActivity`:
+
+```kotlin
+import io.flutter.embedding.android.FlutterFragmentActivity
+
+class MainActivity : FlutterFragmentActivity()
+```
+
+Starting automatically when the app goes to the background works on iOS 15+ and Android 12 (API 31)+.
+
 For prototyping without a backend, the Fishjam dashboard offers a Sandbox API that returns peer tokens. It is
 unauthenticated: never use it in production. The [example app](example/) shows both flows.
 

SPEC.md

@@ -743,6 +743,80 @@ when it joins and stops them when it leaves.
 - The app adds `audio` and `voip` to `UIBackgroundModes`. Out of scope: VoIP push, incoming calls, answering,
   ring timeouts.
 
+
+## 20d. Picture-in-picture
+
+`PictureInPictureView` renders a `MediaStream` and offers its first video track for picture-in-picture (PiP),
+the floating window the OS shows while the app is in the background. It mirrors the React Native SDK's
+`RTCPIPView`, `RTCPIPOptions`, `onPictureInPictureChange` and `startPIP` / `stopPIP`. It talks to the SDK's
+native plugin over the method channel `io.fishjam.client/pip` and the event channel
+`io.fishjam.client/pip_events`.
+
+- Options (`PictureInPictureOptions`): `enabled` (default `true`), `preferredSize` (logical pixels;
+  default 1920×1080, the React Native native default. Android uses it as the aspect ratio, and uses the
+  view size for a side that is 0. iOS sets `preferredContentSize` to this size), `startAutomatically` (default `true`; iOS:
+  `canStartPictureInPictureAutomaticallyFromInline`. iOS starts the window itself when the app
+  backgrounds; calling `startPictureInPicture()` during that transition cancels the hide and leaves
+  the app on screen. Android: `setAutoEnterEnabled`, API 31+),
+  `stopAutomatically` (default `true`, iOS only: PiP stops 0.5 s after the app returns to the foreground),
+  `allowsCameraInBackground` (default `false`, iOS 16+ only: multitasking camera access on the capture
+  session, for an app that does not start a video CallKit call; a video call already does this, §20c). The
+  flag is applied again just before the app resigns active, because the capture session may not exist when
+  the option is first set. `fallbackLabel` (default `No video`), `fallbackBackgroundColor` (default `0xFF1C2945`) and
+  `fallbackTextColor` (default white) are drawn in the app and in the window when this view's video track is gone.
+  `pipOptions: null` or `enabled: false` means that view is not offered. Any number of
+  views may be offered at once. Dart does not choose among them: each offered view stays registered until
+  that view itself stops being offered.
+- Messages, all keyed by an integer `viewId` unique per mounted view:
+  - `update` `{viewId, trackId, rect, mirror, objectFit, preferredSize, startAutomatically,
+    stopAutomatically, allowsCameraInBackground, fallbackLabel, fallbackBackgroundColor, fallbackTextColor}` is sent for each offered view when it becomes offered or
+    any of these values changes. Offering or updating another view does not send `update` or `unregister`
+    for this one. `trackId` is the `flutter_webrtc` id of the first video track, or `null`. `rect`
+    `{x, y, width, height}` is the view's global rect in logical pixels, or `null` before the first layout.
+    `objectFit` is `cover` or `contain`. `preferredSize` `{width, height}` is always sent (default
+    1920×1080). Native code uses that default when the key is missing.
+  - `unregister` `{viewId}` is sent when that view stops being offered (unmounted, `pipOptions: null`, or
+    `enabled: false`), including when other views stay offered. Native code releases that view only.
+  - `start` / `stop` `{viewId}` come from `PictureInPictureController`. They are sent only for a view that
+    is offered; otherwise they do nothing. `stop` does nothing on Android (the user closes or expands the
+    window). `PictureInPictureController.setMediaStream(stream)` sends that view an `update` with the
+    stream's first video track id (or `null`), without waiting for a Flutter rebuild. Apps use it when a
+    remote track changes while the app is in the background, such as after that peer reconnects.
+  - Events `{viewId, isActive}` update `PictureInPictureController.isActive` and call
+    `onPictureInPictureChange` of that view. A view that stops being offered while active becomes inactive
+    without a callback.
+- Native code resolves `trackId` with `FlutterWebRTCPlugin.sharedSingleton` (`trackForId:peerConnectionId:` on
+  iOS, `getTrackForId` on Android, both with no peer connection id).
+- iOS (15+): one `AVPictureInPictureController` per offered view, each with an
+  `AVPictureInPictureVideoCallViewController`. The SDK does not pick a winner. Frames are drawn into an
+  `AVSampleBufferDisplayLayer` while that view's PiP is active, and also while `startAutomatically` is on,
+  because iOS will not start an empty layer when the app backgrounds. The source view is a clear view
+  placed on the video tile's rect (`rect` in the `update` message), which is where iOS animates
+  picture-in-picture from and to. `stop` while the app is in the background removes that source view,
+  because iOS ignores `stopPictureInPicture()` until the app is visible; once the window has closed, the
+  view is put back and that view's controller is created again, so a later `start` or automatic start
+  works. If a video track is set on it again while the app is still in the background and
+  `startAutomatically` is on, the window opens again. `unregister` stops that view's
+  PiP and removes its controller.
+- Android (API 26+): each offered view keeps its own settings, but the window belongs to the activity
+  (`enterPictureInPictureMode`), so one view backs it. That view is the one whose `update` arrived most
+  recently, or the one `start` was called on. A later `update` from another view replaces it; while the
+  window is already active, the new view is reported active and the previous view is not reported inactive.
+  `unregister` for a view that is not backing the window only drops that view's settings. `unregister` for
+  the view that is backing it turns auto-enter off (API 31+). If the window is already open it stays open
+  and draws the fallback label, because Android ignores a request to close it; otherwise listening stops.
+  It does not hand the window to another offered view. The activity must be a `FlutterFragmentActivity`
+  so it can report picture-in-picture mode changes; otherwise a warning is logged and those changes are
+  missed. On
+  enter the children of `android.R.id.content` are hidden and a `SurfaceViewRenderer` fed by the track fills
+  the window; on exit it is removed and the children are restored.
+- Below iOS 15 and below Android 8, `update` is sent and native code ignores it. On other platforms nothing
+  is sent. `isActive` stays `false`.
+- The plugin links WebRTC for this feature: the pod and the Swift package depend on `flutter_webrtc` (its
+  `WebRTC` product), and the Android library compiles against `:flutter_webrtc`.
+- When there is no video track, the app and the window both draw `fallbackLabel` on
+  `fallbackBackgroundColor` in `fallbackTextColor` (D24).
+
 ## 21. Logging
 
 One SDK logger, created from `FishjamClient(debug: bool)`. Messages are prefixed with `[FISHJAM]` and emitted
@@ -779,6 +853,7 @@ through `dart:developer`'s `log` only when `debug` is true. The SDK never calls
 | D25 | ICE `disconnected` only renegotiates, however long it lasts | `disconnected` for 3 s is a retryable loss (§8.4) | do not wait for a late `failed` |
 | D26 | opening the socket has no timeout | 10 s (§2.1) | an attempt on a dead network does not hang |
 | D27 | `addTracksOnReconnect` defaults to false: after a reconnection the user publishes nothing until the app adds its tracks again | defaults to true (§6) | a network switch on mobile should not silently turn off the camera and microphone |
+| D28 | (React Native SDK) `pip.fallbackView` is one view drawn in the app and in the PiP window | the app and the window both draw `fallbackLabel`, `fallbackBackgroundColor` and `fallbackTextColor` (§20d) | a Flutter widget cannot be drawn into the native PiP window |
 
 ## 23. Golden fixtures
 

android/build.gradle.kts

@@ -50,3 +50,14 @@ kotlin {
         jvmTarget = org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_17
     }
 }
+
+// Provided at runtime by flutter_webrtc (WebRTC) and the Flutter embedding (AndroidX).
+// compileOnly: the app keeps the single copy that flutter_webrtc packages.
+// flutter_webrtc 1.6.2 uses io.github.webrtc-sdk:android:150.7871.01. Gradle
+// requires a version on this line, so update it when bumping flutter_webrtc
+// in pubspec.yaml.
+dependencies {
+    compileOnly(project(":flutter_webrtc"))
+    compileOnly("io.github.webrtc-sdk:android:150.7871.01")
+    compileOnly("androidx.fragment:fragment:1.7.1")
+}

android/src/main/kotlin/io/fishjam/client/FishjamClientPlugin.kt

@@ -4,6 +4,7 @@ import android.content.Context
 import android.util.Log
 import io.fishjam.client.foregroundservice.ForegroundServiceController
 import io.fishjam.client.foregroundservice.NotificationPermission
+import io.fishjam.client.pip.PictureInPictureManager
 import io.flutter.embedding.engine.plugins.FlutterPlugin
 import io.flutter.embedding.engine.plugins.activity.ActivityAware
 import io.flutter.embedding.engine.plugins.activity.ActivityPluginBinding
@@ -21,6 +22,12 @@ class FishjamClientPlugin : FlutterPlugin, ActivityAware {
     private lateinit var backgroundChannel: MethodChannel
     private val notificationPermission = NotificationPermission()
     private var activityBinding: ActivityPluginBinding? = null
+    private lateinit var pipChannel: MethodChannel
+    private lateinit var pipEvents: EventChannel
+    private var pipEventSink: EventChannel.EventSink? = null
+    private val pictureInPicture = PictureInPictureManager { viewId, active ->
+        pipEventSink?.success(mapOf("viewId" to viewId, "isActive" to active))
+    }
 
     override fun onAttachedToEngine(binding: FlutterPlugin.FlutterPluginBinding) {
         context = binding.applicationContext
@@ -43,6 +50,18 @@ class FishjamClientPlugin : FlutterPlugin, ActivityAware {
         })
         backgroundChannel = MethodChannel(binding.binaryMessenger, BACKGROUND_CHANNEL)
         backgroundChannel.setMethodCallHandler(::onBackgroundMethodCall)
+        pipChannel = MethodChannel(binding.binaryMessenger, PIP_CHANNEL)
+        pipChannel.setMethodCallHandler(pictureInPicture::handle)
+        pipEvents = EventChannel(binding.binaryMessenger, PIP_EVENTS_CHANNEL)
+        pipEvents.setStreamHandler(object : EventChannel.StreamHandler {
+            override fun onListen(arguments: Any?, events: EventChannel.EventSink) {
+                pipEventSink = events
+            }
+
+            override fun onCancel(arguments: Any?) {
+                pipEventSink = null
+            }
+        })
     }
 
     private fun onAudioMethodCall(call: MethodCall, result: Result) {
@@ -85,12 +104,16 @@ class FishjamClientPlugin : FlutterPlugin, ActivityAware {
         audioEventSink = null
         audioOutput.stopObserving()
         backgroundChannel.setMethodCallHandler(null)
+        pipChannel.setMethodCallHandler(null)
+        pipEvents.setStreamHandler(null)
+        pipEventSink = null
     }
 
     override fun onAttachedToActivity(binding: ActivityPluginBinding) {
         activityBinding = binding
         notificationPermission.activity = binding.activity
         binding.addRequestPermissionsResultListener(notificationPermission)
+        pictureInPicture.attachActivity(binding.activity)
     }
 
     override fun onDetachedFromActivityForConfigChanges() = onDetachedFromActivity()
@@ -103,6 +126,7 @@ class FishjamClientPlugin : FlutterPlugin, ActivityAware {
         activityBinding = null
         notificationPermission.activity = null
         notificationPermission.flush()
+        pictureInPicture.detachActivity()
     }
 
     /**
@@ -134,5 +158,7 @@ class FishjamClientPlugin : FlutterPlugin, ActivityAware {
         private const val AUDIO_CHANNEL = "io.fishjam.client/audio_output"
         private const val AUDIO_EVENTS_CHANNEL = "io.fishjam.client/audio_output_events"
         private const val BACKGROUND_CHANNEL = "io.fishjam.client/background"
+        private const val PIP_CHANNEL = "io.fishjam.client/pip"
+        private const val PIP_EVENTS_CHANNEL = "io.fishjam.client/pip_events"
     }
 }