diff --git a/CHANGELOG.md b/CHANGELOG.md
index d264b72..662f077 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -13,4 +13,6 @@ Initial release.
- Livestreams over WHIP (`publishLivestream`) and WHEP (`receiveLivestream`).
- Background calls: Android foreground service (`ForegroundService`) and iOS CallKit session (`CallKit`)
keep microphone capture and playback running while the app is in the background.
+- Picture-in-picture (`PictureInPictureView`, `PictureInPictureOptions`, `PictureInPictureController`),
+ matching the React Native SDK's `RTCPIPView`.
- Behavioral differences from the Fishjam web SDK are listed in SPEC.md §22.
diff --git a/README.md b/README.md
index e2598b9..498280b 100644
--- a/README.md
+++ b/README.md
@@ -181,6 +181,75 @@ For screen sharing, also add `FOREGROUND_SERVICE_MEDIA_PROJECTION`, add `mediaPr
```
+## Picture-in-picture
+
+`PictureInPictureView` renders a stream and offers its first video track for the system picture-in-picture
+window (iOS 15+, Android 8 / API 26+). Every view with `pipOptions` enabled is offered on its own. On iOS each
+offered view has its own controller. On Android the window belongs to the activity, so the view updated
+most recently, or the one `start()` was called on, is the one shown. It matches the React Native SDK's
+`RTCPIPView`: `pipOptions` takes the same options (`enabled`,
+`preferredSize` (default 1920×1080), `startAutomatically`, `stopAutomatically`, `allowsCameraInBackground`), plus `fallbackLabel`,
+`fallbackBackgroundColor` and `fallbackTextColor`, drawn in the app and in the window when the video track is gone, and
+`PictureInPictureController.start()` / `stop()` replace `startPIP(ref)` / `stopPIP(ref)`.
+
+```dart
+final pip = PictureInPictureController();
+
+PictureInPictureView(
+ mediaStream: remoteTrack.stream,
+ controller: pip,
+ pipOptions: const PictureInPictureOptions(
+ startAutomatically: true,
+ fallbackLabel: 'Left the call',
+ fallbackBackgroundColor: Color(0xFF112233),
+ fallbackTextColor: Color(0xFFFFFFFF),
+ ),
+ onPictureInPictureChange: (active) => setState(() => _inPip = active),
+);
+
+await pip.start(); // e.g. from a button; stop() does nothing on Android
+```
+
+When the video track is gone, the app and the window both draw `fallbackLabel` (default `No video`) in
+`fallbackTextColor` on `fallbackBackgroundColor`. `stopAutomatically` applies on iOS only.
+
+A video CallKit call already keeps the camera running in the background (see above).
+`allowsCameraInBackground: true` (iOS 16+, default `false`, as in the React Native SDK) does the same for an
+app that does not start one. On iOS 18 or newer that needs `voip` in `UIBackgroundModes`. Before iOS 18 it
+also needs the `com.apple.developer.avfoundation.multitasking-camera-access` entitlement.
+
+Keep the call alive in the background with `ForegroundService` and `CallKit` (see above), and add:
+
+**iOS** `Info.plist` (the `audio` entry is already required for background calls):
+
+```xml
+UIBackgroundModes
+
+ audio
+
+```
+
+**Android** `AndroidManifest.xml`, on the activity (keep the `configChanges` the Flutter template sets, so
+the activity is not recreated when picture-in-picture starts):
+
+```xml
+
+```
+
+**Android** `MainActivity.kt` must extend `FlutterFragmentActivity`:
+
+```kotlin
+import io.flutter.embedding.android.FlutterFragmentActivity
+
+class MainActivity : FlutterFragmentActivity()
+```
+
+Starting automatically when the app goes to the background works on iOS 15+ and Android 12 (API 31)+.
+
For prototyping without a backend, the Fishjam dashboard offers a Sandbox API that returns peer tokens. It is
unauthenticated: never use it in production. The [example app](example/) shows both flows.
diff --git a/SPEC.md b/SPEC.md
index 7c0c0cf..9027552 100644
--- a/SPEC.md
+++ b/SPEC.md
@@ -743,6 +743,80 @@ when it joins and stops them when it leaves.
- The app adds `audio` and `voip` to `UIBackgroundModes`. Out of scope: VoIP push, incoming calls, answering,
ring timeouts.
+
+## 20d. Picture-in-picture
+
+`PictureInPictureView` renders a `MediaStream` and offers its first video track for picture-in-picture (PiP),
+the floating window the OS shows while the app is in the background. It mirrors the React Native SDK's
+`RTCPIPView`, `RTCPIPOptions`, `onPictureInPictureChange` and `startPIP` / `stopPIP`. It talks to the SDK's
+native plugin over the method channel `io.fishjam.client/pip` and the event channel
+`io.fishjam.client/pip_events`.
+
+- Options (`PictureInPictureOptions`): `enabled` (default `true`), `preferredSize` (logical pixels;
+ default 1920×1080, the React Native native default. Android uses it as the aspect ratio, and uses the
+ view size for a side that is 0. iOS sets `preferredContentSize` to this size), `startAutomatically` (default `true`; iOS:
+ `canStartPictureInPictureAutomaticallyFromInline`. iOS starts the window itself when the app
+ backgrounds; calling `startPictureInPicture()` during that transition cancels the hide and leaves
+ the app on screen. Android: `setAutoEnterEnabled`, API 31+),
+ `stopAutomatically` (default `true`, iOS only: PiP stops 0.5 s after the app returns to the foreground),
+ `allowsCameraInBackground` (default `false`, iOS 16+ only: multitasking camera access on the capture
+ session, for an app that does not start a video CallKit call; a video call already does this, §20c). The
+ flag is applied again just before the app resigns active, because the capture session may not exist when
+ the option is first set. `fallbackLabel` (default `No video`), `fallbackBackgroundColor` (default `0xFF1C2945`) and
+ `fallbackTextColor` (default white) are drawn in the app and in the window when this view's video track is gone.
+ `pipOptions: null` or `enabled: false` means that view is not offered. Any number of
+ views may be offered at once. Dart does not choose among them: each offered view stays registered until
+ that view itself stops being offered.
+- Messages, all keyed by an integer `viewId` unique per mounted view:
+ - `update` `{viewId, trackId, rect, mirror, objectFit, preferredSize, startAutomatically,
+ stopAutomatically, allowsCameraInBackground, fallbackLabel, fallbackBackgroundColor, fallbackTextColor}` is sent for each offered view when it becomes offered or
+ any of these values changes. Offering or updating another view does not send `update` or `unregister`
+ for this one. `trackId` is the `flutter_webrtc` id of the first video track, or `null`. `rect`
+ `{x, y, width, height}` is the view's global rect in logical pixels, or `null` before the first layout.
+ `objectFit` is `cover` or `contain`. `preferredSize` `{width, height}` is always sent (default
+ 1920×1080). Native code uses that default when the key is missing.
+ - `unregister` `{viewId}` is sent when that view stops being offered (unmounted, `pipOptions: null`, or
+ `enabled: false`), including when other views stay offered. Native code releases that view only.
+ - `start` / `stop` `{viewId}` come from `PictureInPictureController`. They are sent only for a view that
+ is offered; otherwise they do nothing. `stop` does nothing on Android (the user closes or expands the
+ window). `PictureInPictureController.setMediaStream(stream)` sends that view an `update` with the
+ stream's first video track id (or `null`), without waiting for a Flutter rebuild. Apps use it when a
+ remote track changes while the app is in the background, such as after that peer reconnects.
+ - Events `{viewId, isActive}` update `PictureInPictureController.isActive` and call
+ `onPictureInPictureChange` of that view. A view that stops being offered while active becomes inactive
+ without a callback.
+- Native code resolves `trackId` with `FlutterWebRTCPlugin.sharedSingleton` (`trackForId:peerConnectionId:` on
+ iOS, `getTrackForId` on Android, both with no peer connection id).
+- iOS (15+): one `AVPictureInPictureController` per offered view, each with an
+ `AVPictureInPictureVideoCallViewController`. The SDK does not pick a winner. Frames are drawn into an
+ `AVSampleBufferDisplayLayer` while that view's PiP is active, and also while `startAutomatically` is on,
+ because iOS will not start an empty layer when the app backgrounds. The source view is a clear view
+ placed on the video tile's rect (`rect` in the `update` message), which is where iOS animates
+ picture-in-picture from and to. `stop` while the app is in the background removes that source view,
+ because iOS ignores `stopPictureInPicture()` until the app is visible; once the window has closed, the
+ view is put back and that view's controller is created again, so a later `start` or automatic start
+ works. If a video track is set on it again while the app is still in the background and
+ `startAutomatically` is on, the window opens again. `unregister` stops that view's
+ PiP and removes its controller.
+- Android (API 26+): each offered view keeps its own settings, but the window belongs to the activity
+ (`enterPictureInPictureMode`), so one view backs it. That view is the one whose `update` arrived most
+ recently, or the one `start` was called on. A later `update` from another view replaces it; while the
+ window is already active, the new view is reported active and the previous view is not reported inactive.
+ `unregister` for a view that is not backing the window only drops that view's settings. `unregister` for
+ the view that is backing it turns auto-enter off (API 31+). If the window is already open it stays open
+ and draws the fallback label, because Android ignores a request to close it; otherwise listening stops.
+ It does not hand the window to another offered view. The activity must be a `FlutterFragmentActivity`
+ so it can report picture-in-picture mode changes; otherwise a warning is logged and those changes are
+ missed. On
+ enter the children of `android.R.id.content` are hidden and a `SurfaceViewRenderer` fed by the track fills
+ the window; on exit it is removed and the children are restored.
+- Below iOS 15 and below Android 8, `update` is sent and native code ignores it. On other platforms nothing
+ is sent. `isActive` stays `false`.
+- The plugin links WebRTC for this feature: the pod and the Swift package depend on `flutter_webrtc` (its
+ `WebRTC` product), and the Android library compiles against `:flutter_webrtc`.
+- When there is no video track, the app and the window both draw `fallbackLabel` on
+ `fallbackBackgroundColor` in `fallbackTextColor` (D24).
+
## 21. Logging
One SDK logger, created from `FishjamClient(debug: bool)`. Messages are prefixed with `[FISHJAM]` and emitted
@@ -779,6 +853,7 @@ through `dart:developer`'s `log` only when `debug` is true. The SDK never calls
| D25 | ICE `disconnected` only renegotiates, however long it lasts | `disconnected` for 3 s is a retryable loss (§8.4) | do not wait for a late `failed` |
| D26 | opening the socket has no timeout | 10 s (§2.1) | an attempt on a dead network does not hang |
| D27 | `addTracksOnReconnect` defaults to false: after a reconnection the user publishes nothing until the app adds its tracks again | defaults to true (§6) | a network switch on mobile should not silently turn off the camera and microphone |
+| D28 | (React Native SDK) `pip.fallbackView` is one view drawn in the app and in the PiP window | the app and the window both draw `fallbackLabel`, `fallbackBackgroundColor` and `fallbackTextColor` (§20d) | a Flutter widget cannot be drawn into the native PiP window |
## 23. Golden fixtures
diff --git a/android/build.gradle.kts b/android/build.gradle.kts
index 6d4d486..9cd5a16 100644
--- a/android/build.gradle.kts
+++ b/android/build.gradle.kts
@@ -50,3 +50,14 @@ kotlin {
jvmTarget = org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_17
}
}
+
+// Provided at runtime by flutter_webrtc (WebRTC) and the Flutter embedding (AndroidX).
+// compileOnly: the app keeps the single copy that flutter_webrtc packages.
+// flutter_webrtc 1.6.2 uses io.github.webrtc-sdk:android:150.7871.01. Gradle
+// requires a version on this line, so update it when bumping flutter_webrtc
+// in pubspec.yaml.
+dependencies {
+ compileOnly(project(":flutter_webrtc"))
+ compileOnly("io.github.webrtc-sdk:android:150.7871.01")
+ compileOnly("androidx.fragment:fragment:1.7.1")
+}
diff --git a/android/src/main/kotlin/io/fishjam/client/FishjamClientPlugin.kt b/android/src/main/kotlin/io/fishjam/client/FishjamClientPlugin.kt
index 95ddfeb..03029f9 100644
--- a/android/src/main/kotlin/io/fishjam/client/FishjamClientPlugin.kt
+++ b/android/src/main/kotlin/io/fishjam/client/FishjamClientPlugin.kt
@@ -4,6 +4,7 @@ import android.content.Context
import android.util.Log
import io.fishjam.client.foregroundservice.ForegroundServiceController
import io.fishjam.client.foregroundservice.NotificationPermission
+import io.fishjam.client.pip.PictureInPictureManager
import io.flutter.embedding.engine.plugins.FlutterPlugin
import io.flutter.embedding.engine.plugins.activity.ActivityAware
import io.flutter.embedding.engine.plugins.activity.ActivityPluginBinding
@@ -21,6 +22,12 @@ class FishjamClientPlugin : FlutterPlugin, ActivityAware {
private lateinit var backgroundChannel: MethodChannel
private val notificationPermission = NotificationPermission()
private var activityBinding: ActivityPluginBinding? = null
+ private lateinit var pipChannel: MethodChannel
+ private lateinit var pipEvents: EventChannel
+ private var pipEventSink: EventChannel.EventSink? = null
+ private val pictureInPicture = PictureInPictureManager { viewId, active ->
+ pipEventSink?.success(mapOf("viewId" to viewId, "isActive" to active))
+ }
override fun onAttachedToEngine(binding: FlutterPlugin.FlutterPluginBinding) {
context = binding.applicationContext
@@ -43,6 +50,18 @@ class FishjamClientPlugin : FlutterPlugin, ActivityAware {
})
backgroundChannel = MethodChannel(binding.binaryMessenger, BACKGROUND_CHANNEL)
backgroundChannel.setMethodCallHandler(::onBackgroundMethodCall)
+ pipChannel = MethodChannel(binding.binaryMessenger, PIP_CHANNEL)
+ pipChannel.setMethodCallHandler(pictureInPicture::handle)
+ pipEvents = EventChannel(binding.binaryMessenger, PIP_EVENTS_CHANNEL)
+ pipEvents.setStreamHandler(object : EventChannel.StreamHandler {
+ override fun onListen(arguments: Any?, events: EventChannel.EventSink) {
+ pipEventSink = events
+ }
+
+ override fun onCancel(arguments: Any?) {
+ pipEventSink = null
+ }
+ })
}
private fun onAudioMethodCall(call: MethodCall, result: Result) {
@@ -85,12 +104,16 @@ class FishjamClientPlugin : FlutterPlugin, ActivityAware {
audioEventSink = null
audioOutput.stopObserving()
backgroundChannel.setMethodCallHandler(null)
+ pipChannel.setMethodCallHandler(null)
+ pipEvents.setStreamHandler(null)
+ pipEventSink = null
}
override fun onAttachedToActivity(binding: ActivityPluginBinding) {
activityBinding = binding
notificationPermission.activity = binding.activity
binding.addRequestPermissionsResultListener(notificationPermission)
+ pictureInPicture.attachActivity(binding.activity)
}
override fun onDetachedFromActivityForConfigChanges() = onDetachedFromActivity()
@@ -103,6 +126,7 @@ class FishjamClientPlugin : FlutterPlugin, ActivityAware {
activityBinding = null
notificationPermission.activity = null
notificationPermission.flush()
+ pictureInPicture.detachActivity()
}
/**
@@ -134,5 +158,7 @@ class FishjamClientPlugin : FlutterPlugin, ActivityAware {
private const val AUDIO_CHANNEL = "io.fishjam.client/audio_output"
private const val AUDIO_EVENTS_CHANNEL = "io.fishjam.client/audio_output_events"
private const val BACKGROUND_CHANNEL = "io.fishjam.client/background"
+ private const val PIP_CHANNEL = "io.fishjam.client/pip"
+ private const val PIP_EVENTS_CHANNEL = "io.fishjam.client/pip_events"
}
}
diff --git a/android/src/main/kotlin/io/fishjam/client/pip/PictureInPictureManager.kt b/android/src/main/kotlin/io/fishjam/client/pip/PictureInPictureManager.kt
new file mode 100644
index 0000000..951cb22
--- /dev/null
+++ b/android/src/main/kotlin/io/fishjam/client/pip/PictureInPictureManager.kt
@@ -0,0 +1,371 @@
+package io.fishjam.client.pip
+
+import android.app.Activity
+import android.app.PictureInPictureParams
+import android.graphics.Color
+import android.os.Build
+import android.util.Log
+import android.util.Rational
+import android.view.Gravity
+import android.view.View
+import android.view.ViewGroup
+import android.widget.FrameLayout
+import android.widget.TextView
+import androidx.annotation.RequiresApi
+import androidx.core.app.OnPictureInPictureModeChangedProvider
+import androidx.core.app.PictureInPictureModeChangedInfo
+import androidx.core.util.Consumer
+import com.cloudwebrtc.webrtc.FlutterWebRTCPlugin
+import com.cloudwebrtc.webrtc.utils.EglUtils
+import io.flutter.plugin.common.MethodCall
+import io.flutter.plugin.common.MethodChannel.Result
+import org.webrtc.RendererCommon
+import org.webrtc.SurfaceViewRenderer
+import org.webrtc.VideoTrack
+
+/**
+ * Handles `io.fishjam.client/pip`. Port of the React Native SDK's
+ * `PIPManager` (`react-native-webrtc/android/.../PIPManager.java`): each
+ * `PictureInPictureView` on the Dart side registers its own settings here,
+ * keyed by view id, just like each `WebRTCView` there owns one `PIPManager`.
+ *
+ * Android's picture-in-picture window is a property of the `Activity`
+ * (`enterPictureInPictureMode`), not of a view, so only one view's settings
+ * can back the window at a time -- the one most recently updated, or the one
+ * [start] was called on. The React Native SDK has the same constraint for
+ * the same reason, despite also keeping one manager per view.
+ *
+ * While in picture-in-picture the Flutter view is hidden and a
+ * [SurfaceViewRenderer] fed by the same track fills the window, so the window
+ * shows only the video whatever the Flutter UI looks like.
+ */
+class PictureInPictureManager(private val onActiveChanged: (viewId: Int, active: Boolean) -> Unit) {
+ private var activity: Activity? = null
+ private var listening = false
+
+ private class ViewState {
+ var trackId: String? = null
+ var mirror = false
+ var objectFitContain = false
+ var startAutomatically = true
+ var preferredWidth = 0.0
+ var preferredHeight = 0.0
+ var viewWidth = 0.0
+ var viewHeight = 0.0
+ var label = "No video"
+ var backgroundColor = DEFAULT_PLACEHOLDER_BACKGROUND
+ var textColor = Color.WHITE
+ }
+
+ private val views = mutableMapOf()
+
+ /** Whichever view's settings currently back the window. */
+ private var currentViewId: Int? = null
+ private var active = false
+
+ private val hiddenChildren = mutableListOf>()
+ private var container: FrameLayout? = null
+ private var renderer: SurfaceViewRenderer? = null
+ private var rendererTrack: VideoTrack? = null
+
+ private val modeListener = Consumer { info ->
+ if (info.isInPictureInPictureMode) {
+ onEnter()
+ } else {
+ onExit()
+ if (currentViewId == null) stopListening()
+ }
+ }
+
+ fun attachActivity(activity: Activity) {
+ this.activity = activity
+ if (currentViewId != null) {
+ startListening()
+ if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) applyParams()
+ }
+ }
+
+ fun detachActivity() {
+ if (active) onExit()
+ stopListening()
+ activity = null
+ }
+
+ fun handle(call: MethodCall, result: Result) {
+ val args = call.arguments as? Map<*, *> ?: emptyMap()
+ val id = (args["viewId"] as? Number)?.toInt()
+ when (call.method) {
+ "update" -> if (id != null) update(id, args)
+ "start" -> if (id != null) start(id)
+ "stop" -> Unit
+ "unregister" -> if (id != null) unregister(id)
+ else -> return result.notImplemented()
+ }
+ result.success(null)
+ }
+
+ private fun update(id: Int, args: Map<*, *>) {
+ if (Build.VERSION.SDK_INT < Build.VERSION_CODES.O) return
+ val state = views.getOrPut(id) { ViewState() }
+ state.trackId = args["trackId"] as? String
+ state.mirror = args["mirror"] as? Boolean ?: false
+ state.objectFitContain = args["objectFit"] == "contain"
+ state.startAutomatically = args["startAutomatically"] as? Boolean ?: true
+ val preferred = args["preferredSize"] as? Map<*, *>
+ // React Native defaults to 1920×1080. A side of 0 falls back to the
+ // view size in [buildParams].
+ state.preferredWidth = preferredSide(preferred?.get("width"), DEFAULT_PREFERRED_WIDTH)
+ state.preferredHeight = preferredSide(preferred?.get("height"), DEFAULT_PREFERRED_HEIGHT)
+ val rect = args["rect"] as? Map<*, *>
+ state.viewWidth = (rect?.get("width") as? Number)?.toDouble() ?: 0.0
+ state.viewHeight = (rect?.get("height") as? Number)?.toDouble() ?: 0.0
+ state.label = args["fallbackLabel"] as? String ?: "No video"
+ state.backgroundColor = (args["fallbackBackgroundColor"] as? Number)?.toInt()
+ ?: DEFAULT_PLACEHOLDER_BACKGROUND
+ state.textColor = (args["fallbackTextColor"] as? Number)?.toInt() ?: Color.WHITE
+
+ val takesOver = currentViewId != id
+ currentViewId = id
+ if (takesOver && active) onActiveChanged(id, true)
+
+ if (active) {
+ if (state.trackId == null) {
+ showNoVideo(state)
+ } else {
+ renderer?.let {
+ it.setMirror(state.mirror)
+ it.setScalingType(scalingType(state))
+ }
+ attachRendererTrack(state)
+ }
+ }
+
+ startListening()
+ applyParams()
+ }
+
+ @RequiresApi(Build.VERSION_CODES.O)
+ private fun start(id: Int) {
+ val activity = activity ?: return
+ val state = views[id] ?: return
+ currentViewId = id
+ try {
+ activity.enterPictureInPictureMode(buildParams(state))
+ } catch (e: IllegalStateException) {
+ Log.e(TAG, "Failed to enter picture-in-picture", e)
+ }
+ }
+
+ private fun unregister(id: Int) {
+ val state = views.remove(id)
+ if (currentViewId != id) return
+ currentViewId = null
+ if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S && state != null) {
+ state.startAutomatically = false
+ val activity = activity
+ if (activity != null) {
+ try {
+ activity.setPictureInPictureParams(buildParams(state))
+ } catch (e: IllegalStateException) {
+ Log.e(TAG, "Failed to update picture-in-picture params", e)
+ }
+ }
+ }
+ if (active) {
+ // The window stays open: restoring the Flutter view now would show
+ // the app's UI in it, and Android ignores a close request. Show
+ // "No video" until another view takes over or the user closes it.
+ showNoVideo(state)
+ return
+ }
+ stopListening()
+ }
+
+ private fun startListening() {
+ if (listening) return
+ val provider = activity as? OnPictureInPictureModeChangedProvider
+ if (provider == null) {
+ if (activity != null) {
+ Log.w(
+ TAG,
+ "Picture-in-picture needs MainActivity to extend FlutterFragmentActivity",
+ )
+ }
+ return
+ }
+ provider.addOnPictureInPictureModeChangedListener(modeListener)
+ listening = true
+ }
+
+ private fun stopListening() {
+ if (!listening) return
+ (activity as? OnPictureInPictureModeChangedProvider)
+ ?.removeOnPictureInPictureModeChangedListener(modeListener)
+ listening = false
+ }
+
+ @RequiresApi(Build.VERSION_CODES.O)
+ private fun buildParams(state: ViewState): PictureInPictureParams {
+ val builder = PictureInPictureParams.Builder()
+ val width = if (state.preferredWidth > 0) state.preferredWidth else state.viewWidth
+ val height = if (state.preferredHeight > 0) state.preferredHeight else state.viewHeight
+ if (width > 0 && height > 0) {
+ // Android rejects ratios outside 1:2.39 .. 2.39:1.
+ val ratio = (width / height).coerceIn(1 / MAX_RATIO, MAX_RATIO)
+ builder.setAspectRatio(Rational((ratio * 10_000).toInt(), 10_000))
+ }
+ if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S) {
+ builder.setAutoEnterEnabled(state.startAutomatically && listening)
+ }
+ return builder.build()
+ }
+
+ private fun applyParams() {
+ if (Build.VERSION.SDK_INT < Build.VERSION_CODES.O) return
+ val activity = activity ?: return
+ val state = currentViewId?.let { views[it] } ?: return
+ try {
+ activity.setPictureInPictureParams(buildParams(state))
+ } catch (e: IllegalStateException) {
+ Log.e(TAG, "Failed to update picture-in-picture params", e)
+ }
+ }
+
+ private fun onEnter() {
+ setActive(true)
+ val activity = activity ?: return
+ val state = currentViewId?.let { views[it] } ?: return
+ val root = activity.findViewById(android.R.id.content) ?: return
+
+ hiddenChildren.clear()
+ for (i in 0 until root.childCount) {
+ val child = root.getChildAt(i)
+ hiddenChildren.add(child to child.visibility)
+ child.visibility = View.GONE
+ }
+
+ val surface = SurfaceViewRenderer(activity).apply {
+ setBackgroundColor(Color.BLACK)
+ setMirror(state.mirror)
+ setScalingType(scalingType(state))
+ }
+ val frame = FrameLayout(activity).apply {
+ setBackgroundColor(Color.BLACK)
+ addView(surface, FrameLayout.LayoutParams(MATCH_PARENT, MATCH_PARENT))
+ }
+ root.addView(frame, FrameLayout.LayoutParams(MATCH_PARENT, MATCH_PARENT))
+ container = frame
+
+ val eglContext = EglUtils.getRootEglBaseContext()
+ if (eglContext == null) {
+ Log.e(TAG, "Cannot create the picture-in-picture renderer: no EGL context")
+ return
+ }
+ try {
+ surface.init(eglContext, object : RendererCommon.RendererEvents {
+ override fun onFirstFrameRendered() {
+ surface.post { surface.setBackgroundColor(Color.TRANSPARENT) }
+ }
+
+ override fun onFrameResolutionChanged(width: Int, height: Int, rotation: Int) {}
+ })
+ renderer = surface
+ attachRendererTrack(state)
+ } catch (e: RuntimeException) {
+ Log.e(TAG, "Failed to initialize the picture-in-picture renderer", e)
+ }
+ }
+
+ private fun attachRendererTrack(state: ViewState) {
+ val surface = renderer ?: return
+ rendererTrack?.let { track -> runCatching { track.removeSink(surface) } }
+ rendererTrack = null
+ val track = state.trackId?.let {
+ FlutterWebRTCPlugin.sharedSingleton?.getTrackForId(it, null) as? VideoTrack
+ }
+ track ?: return
+ try {
+ track.addSink(surface)
+ rendererTrack = track
+ surface.visibility = View.VISIBLE
+ container?.findViewWithTag(FALLBACK_TAG)?.let { container?.removeView(it) }
+ } catch (e: IllegalStateException) {
+ Log.e(TAG, "Failed to add the picture-in-picture renderer to the track", e)
+ }
+ }
+
+ private fun showNoVideo(state: ViewState?) {
+ detachRendererTrack()
+ val frame = container ?: return
+ renderer?.visibility = View.GONE
+ val label = state?.label ?: "No video"
+ val background = state?.backgroundColor ?: DEFAULT_PLACEHOLDER_BACKGROUND
+ val textColor = state?.textColor ?: Color.WHITE
+ frame.setBackgroundColor(background)
+ val existing = frame.findViewWithTag(FALLBACK_TAG)
+ if (existing != null) {
+ existing.text = label
+ existing.setTextColor(textColor)
+ return
+ }
+ frame.addView(
+ TextView(frame.context).apply {
+ tag = FALLBACK_TAG
+ text = label
+ setTextColor(textColor)
+ gravity = Gravity.CENTER
+ textSize = 18f
+ },
+ FrameLayout.LayoutParams(MATCH_PARENT, MATCH_PARENT),
+ )
+ }
+
+ private fun detachRendererTrack() {
+ val surface = renderer ?: return
+ rendererTrack?.let { track -> runCatching { track.removeSink(surface) } }
+ rendererTrack = null
+ surface.clearImage()
+ }
+
+ private fun onExit() {
+ setActive(false)
+ renderer?.let { surface ->
+ rendererTrack?.let { track -> runCatching { track.removeSink(surface) } }
+ surface.release()
+ }
+ renderer = null
+ rendererTrack = null
+
+ val root = activity?.findViewById(android.R.id.content)
+ container?.let { root?.removeView(it) }
+ container = null
+ for ((child, visibility) in hiddenChildren) child.visibility = visibility
+ hiddenChildren.clear()
+ }
+
+ private fun setActive(value: Boolean) {
+ if (active == value) return
+ active = value
+ currentViewId?.let { onActiveChanged(it, value) }
+ }
+
+ private fun preferredSide(value: Any?, fallback: Double): Double =
+ (value as? Number)?.toDouble() ?: fallback
+
+ private fun scalingType(state: ViewState) =
+ if (state.objectFitContain) {
+ RendererCommon.ScalingType.SCALE_ASPECT_FIT
+ } else {
+ RendererCommon.ScalingType.SCALE_ASPECT_FILL
+ }
+
+ private companion object {
+ const val TAG = "FishjamPictureInPicture"
+ const val FALLBACK_TAG = "pip-fallback"
+ const val DEFAULT_PLACEHOLDER_BACKGROUND = 0xFF1C2945.toInt()
+ const val DEFAULT_PREFERRED_WIDTH = 1920.0
+ const val DEFAULT_PREFERRED_HEIGHT = 1080.0
+ const val MAX_RATIO = 2.39
+ const val MATCH_PARENT = ViewGroup.LayoutParams.MATCH_PARENT
+ }
+}
diff --git a/docs/parity.md b/docs/parity.md
index 2c21e4a..6495eca 100644
--- a/docs/parity.md
+++ b/docs/parity.md
@@ -77,6 +77,20 @@ against the real `FishjamClient` in `test/client/client_fixtures_test.dart`.
| `ReceiveLivestreamResult.stream` / `stop()` / `getStatistics()` | same names | |
| `LivestreamError` (string enum, thrown) | `LivestreamException(LivestreamError)` | |
+## Picture-in-picture (React Native SDK)
+
+The React Native SDK's picture-in-picture (`@fishjam-cloud/react-native-webrtc`, SPEC §20d):
+
+| React Native SDK | Dart | Notes |
+|---|---|---|
+| `RTCPIPView` | `PictureInPictureView` | owns its renderer; `mediaStream` is a `MediaStream?` |
+| `pip: RTCPIPOptions` | `pipOptions: PictureInPictureOptions?` | same fields and defaults; `null` means not offered |
+| `enabled`, `preferredSize`, `startAutomatically`, `stopAutomatically`, `allowsCameraInBackground` | same names | `preferredSize` is a `Size`, default 1920×1080 |
+| `pip.fallbackView` | `fallbackLabel`, `fallbackBackgroundColor`, `fallbackTextColor` | drawn in the app and in the window (D24) |
+| `onPictureInPictureChange({nativeEvent: {isActive}})` | `onPictureInPictureChange(bool isActive)` | |
+| `startPIP(ref)` / `stopPIP(ref)` | `PictureInPictureController.start()` / `stop()` | the controller is passed to the view |
+| | `PictureInPictureController.isActive` | |
+
## Errors, guards and constants
| Web SDK | Dart | Notes |
diff --git a/example/android/app/src/main/AndroidManifest.xml b/example/android/app/src/main/AndroidManifest.xml
index 50bb465..22fb203 100644
--- a/example/android/app/src/main/AndroidManifest.xml
+++ b/example/android/app/src/main/AndroidManifest.xml
@@ -26,6 +26,7 @@
android:theme="@style/LaunchTheme"
android:configChanges="orientation|keyboardHidden|keyboard|screenSize|smallestScreenSize|locale|layoutDirection|fontScale|screenLayout|density|uiMode"
android:hardwareAccelerated="true"
+ android:supportsPictureInPicture="true"
android:windowSoftInputMode="adjustResize">