diff --git a/CHANGELOG.md b/CHANGELOG.md index d264b72..662f077 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,4 +13,6 @@ Initial release. - Livestreams over WHIP (`publishLivestream`) and WHEP (`receiveLivestream`). - Background calls: Android foreground service (`ForegroundService`) and iOS CallKit session (`CallKit`) keep microphone capture and playback running while the app is in the background. +- Picture-in-picture (`PictureInPictureView`, `PictureInPictureOptions`, `PictureInPictureController`), + matching the React Native SDK's `RTCPIPView`. - Behavioral differences from the Fishjam web SDK are listed in SPEC.md §22. diff --git a/README.md b/README.md index e2598b9..498280b 100644 --- a/README.md +++ b/README.md @@ -181,6 +181,75 @@ For screen sharing, also add `FOREGROUND_SERVICE_MEDIA_PROJECTION`, add `mediaPr ``` +## Picture-in-picture + +`PictureInPictureView` renders a stream and offers its first video track for the system picture-in-picture +window (iOS 15+, Android 8 / API 26+). Every view with `pipOptions` enabled is offered on its own. On iOS each +offered view has its own controller. On Android the window belongs to the activity, so the view updated +most recently, or the one `start()` was called on, is the one shown. It matches the React Native SDK's +`RTCPIPView`: `pipOptions` takes the same options (`enabled`, +`preferredSize` (default 1920×1080), `startAutomatically`, `stopAutomatically`, `allowsCameraInBackground`), plus `fallbackLabel`, +`fallbackBackgroundColor` and `fallbackTextColor`, drawn in the app and in the window when the video track is gone, and +`PictureInPictureController.start()` / `stop()` replace `startPIP(ref)` / `stopPIP(ref)`. + +```dart +final pip = PictureInPictureController(); + +PictureInPictureView( + mediaStream: remoteTrack.stream, + controller: pip, + pipOptions: const PictureInPictureOptions( + startAutomatically: true, + fallbackLabel: 'Left the call', + fallbackBackgroundColor: Color(0xFF112233), + fallbackTextColor: Color(0xFFFFFFFF), + ), + onPictureInPictureChange: (active) => setState(() => _inPip = active), +); + +await pip.start(); // e.g. from a button; stop() does nothing on Android +``` + +When the video track is gone, the app and the window both draw `fallbackLabel` (default `No video`) in +`fallbackTextColor` on `fallbackBackgroundColor`. `stopAutomatically` applies on iOS only. + +A video CallKit call already keeps the camera running in the background (see above). +`allowsCameraInBackground: true` (iOS 16+, default `false`, as in the React Native SDK) does the same for an +app that does not start one. On iOS 18 or newer that needs `voip` in `UIBackgroundModes`. Before iOS 18 it +also needs the `com.apple.developer.avfoundation.multitasking-camera-access` entitlement. + +Keep the call alive in the background with `ForegroundService` and `CallKit` (see above), and add: + +**iOS** `Info.plist` (the `audio` entry is already required for background calls): + +```xml +UIBackgroundModes + + audio + +``` + +**Android** `AndroidManifest.xml`, on the activity (keep the `configChanges` the Flutter template sets, so +the activity is not recreated when picture-in-picture starts): + +```xml + +``` + +**Android** `MainActivity.kt` must extend `FlutterFragmentActivity`: + +```kotlin +import io.flutter.embedding.android.FlutterFragmentActivity + +class MainActivity : FlutterFragmentActivity() +``` + +Starting automatically when the app goes to the background works on iOS 15+ and Android 12 (API 31)+. + For prototyping without a backend, the Fishjam dashboard offers a Sandbox API that returns peer tokens. It is unauthenticated: never use it in production. The [example app](example/) shows both flows. diff --git a/SPEC.md b/SPEC.md index 7c0c0cf..9027552 100644 --- a/SPEC.md +++ b/SPEC.md @@ -743,6 +743,80 @@ when it joins and stops them when it leaves. - The app adds `audio` and `voip` to `UIBackgroundModes`. Out of scope: VoIP push, incoming calls, answering, ring timeouts. + +## 20d. Picture-in-picture + +`PictureInPictureView` renders a `MediaStream` and offers its first video track for picture-in-picture (PiP), +the floating window the OS shows while the app is in the background. It mirrors the React Native SDK's +`RTCPIPView`, `RTCPIPOptions`, `onPictureInPictureChange` and `startPIP` / `stopPIP`. It talks to the SDK's +native plugin over the method channel `io.fishjam.client/pip` and the event channel +`io.fishjam.client/pip_events`. + +- Options (`PictureInPictureOptions`): `enabled` (default `true`), `preferredSize` (logical pixels; + default 1920×1080, the React Native native default. Android uses it as the aspect ratio, and uses the + view size for a side that is 0. iOS sets `preferredContentSize` to this size), `startAutomatically` (default `true`; iOS: + `canStartPictureInPictureAutomaticallyFromInline`. iOS starts the window itself when the app + backgrounds; calling `startPictureInPicture()` during that transition cancels the hide and leaves + the app on screen. Android: `setAutoEnterEnabled`, API 31+), + `stopAutomatically` (default `true`, iOS only: PiP stops 0.5 s after the app returns to the foreground), + `allowsCameraInBackground` (default `false`, iOS 16+ only: multitasking camera access on the capture + session, for an app that does not start a video CallKit call; a video call already does this, §20c). The + flag is applied again just before the app resigns active, because the capture session may not exist when + the option is first set. `fallbackLabel` (default `No video`), `fallbackBackgroundColor` (default `0xFF1C2945`) and + `fallbackTextColor` (default white) are drawn in the app and in the window when this view's video track is gone. + `pipOptions: null` or `enabled: false` means that view is not offered. Any number of + views may be offered at once. Dart does not choose among them: each offered view stays registered until + that view itself stops being offered. +- Messages, all keyed by an integer `viewId` unique per mounted view: + - `update` `{viewId, trackId, rect, mirror, objectFit, preferredSize, startAutomatically, + stopAutomatically, allowsCameraInBackground, fallbackLabel, fallbackBackgroundColor, fallbackTextColor}` is sent for each offered view when it becomes offered or + any of these values changes. Offering or updating another view does not send `update` or `unregister` + for this one. `trackId` is the `flutter_webrtc` id of the first video track, or `null`. `rect` + `{x, y, width, height}` is the view's global rect in logical pixels, or `null` before the first layout. + `objectFit` is `cover` or `contain`. `preferredSize` `{width, height}` is always sent (default + 1920×1080). Native code uses that default when the key is missing. + - `unregister` `{viewId}` is sent when that view stops being offered (unmounted, `pipOptions: null`, or + `enabled: false`), including when other views stay offered. Native code releases that view only. + - `start` / `stop` `{viewId}` come from `PictureInPictureController`. They are sent only for a view that + is offered; otherwise they do nothing. `stop` does nothing on Android (the user closes or expands the + window). `PictureInPictureController.setMediaStream(stream)` sends that view an `update` with the + stream's first video track id (or `null`), without waiting for a Flutter rebuild. Apps use it when a + remote track changes while the app is in the background, such as after that peer reconnects. + - Events `{viewId, isActive}` update `PictureInPictureController.isActive` and call + `onPictureInPictureChange` of that view. A view that stops being offered while active becomes inactive + without a callback. +- Native code resolves `trackId` with `FlutterWebRTCPlugin.sharedSingleton` (`trackForId:peerConnectionId:` on + iOS, `getTrackForId` on Android, both with no peer connection id). +- iOS (15+): one `AVPictureInPictureController` per offered view, each with an + `AVPictureInPictureVideoCallViewController`. The SDK does not pick a winner. Frames are drawn into an + `AVSampleBufferDisplayLayer` while that view's PiP is active, and also while `startAutomatically` is on, + because iOS will not start an empty layer when the app backgrounds. The source view is a clear view + placed on the video tile's rect (`rect` in the `update` message), which is where iOS animates + picture-in-picture from and to. `stop` while the app is in the background removes that source view, + because iOS ignores `stopPictureInPicture()` until the app is visible; once the window has closed, the + view is put back and that view's controller is created again, so a later `start` or automatic start + works. If a video track is set on it again while the app is still in the background and + `startAutomatically` is on, the window opens again. `unregister` stops that view's + PiP and removes its controller. +- Android (API 26+): each offered view keeps its own settings, but the window belongs to the activity + (`enterPictureInPictureMode`), so one view backs it. That view is the one whose `update` arrived most + recently, or the one `start` was called on. A later `update` from another view replaces it; while the + window is already active, the new view is reported active and the previous view is not reported inactive. + `unregister` for a view that is not backing the window only drops that view's settings. `unregister` for + the view that is backing it turns auto-enter off (API 31+). If the window is already open it stays open + and draws the fallback label, because Android ignores a request to close it; otherwise listening stops. + It does not hand the window to another offered view. The activity must be a `FlutterFragmentActivity` + so it can report picture-in-picture mode changes; otherwise a warning is logged and those changes are + missed. On + enter the children of `android.R.id.content` are hidden and a `SurfaceViewRenderer` fed by the track fills + the window; on exit it is removed and the children are restored. +- Below iOS 15 and below Android 8, `update` is sent and native code ignores it. On other platforms nothing + is sent. `isActive` stays `false`. +- The plugin links WebRTC for this feature: the pod and the Swift package depend on `flutter_webrtc` (its + `WebRTC` product), and the Android library compiles against `:flutter_webrtc`. +- When there is no video track, the app and the window both draw `fallbackLabel` on + `fallbackBackgroundColor` in `fallbackTextColor` (D24). + ## 21. Logging One SDK logger, created from `FishjamClient(debug: bool)`. Messages are prefixed with `[FISHJAM]` and emitted @@ -779,6 +853,7 @@ through `dart:developer`'s `log` only when `debug` is true. The SDK never calls | D25 | ICE `disconnected` only renegotiates, however long it lasts | `disconnected` for 3 s is a retryable loss (§8.4) | do not wait for a late `failed` | | D26 | opening the socket has no timeout | 10 s (§2.1) | an attempt on a dead network does not hang | | D27 | `addTracksOnReconnect` defaults to false: after a reconnection the user publishes nothing until the app adds its tracks again | defaults to true (§6) | a network switch on mobile should not silently turn off the camera and microphone | +| D28 | (React Native SDK) `pip.fallbackView` is one view drawn in the app and in the PiP window | the app and the window both draw `fallbackLabel`, `fallbackBackgroundColor` and `fallbackTextColor` (§20d) | a Flutter widget cannot be drawn into the native PiP window | ## 23. Golden fixtures diff --git a/android/build.gradle.kts b/android/build.gradle.kts index 6d4d486..9cd5a16 100644 --- a/android/build.gradle.kts +++ b/android/build.gradle.kts @@ -50,3 +50,14 @@ kotlin { jvmTarget = org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_17 } } + +// Provided at runtime by flutter_webrtc (WebRTC) and the Flutter embedding (AndroidX). +// compileOnly: the app keeps the single copy that flutter_webrtc packages. +// flutter_webrtc 1.6.2 uses io.github.webrtc-sdk:android:150.7871.01. Gradle +// requires a version on this line, so update it when bumping flutter_webrtc +// in pubspec.yaml. +dependencies { + compileOnly(project(":flutter_webrtc")) + compileOnly("io.github.webrtc-sdk:android:150.7871.01") + compileOnly("androidx.fragment:fragment:1.7.1") +} diff --git a/android/src/main/kotlin/io/fishjam/client/FishjamClientPlugin.kt b/android/src/main/kotlin/io/fishjam/client/FishjamClientPlugin.kt index 95ddfeb..03029f9 100644 --- a/android/src/main/kotlin/io/fishjam/client/FishjamClientPlugin.kt +++ b/android/src/main/kotlin/io/fishjam/client/FishjamClientPlugin.kt @@ -4,6 +4,7 @@ import android.content.Context import android.util.Log import io.fishjam.client.foregroundservice.ForegroundServiceController import io.fishjam.client.foregroundservice.NotificationPermission +import io.fishjam.client.pip.PictureInPictureManager import io.flutter.embedding.engine.plugins.FlutterPlugin import io.flutter.embedding.engine.plugins.activity.ActivityAware import io.flutter.embedding.engine.plugins.activity.ActivityPluginBinding @@ -21,6 +22,12 @@ class FishjamClientPlugin : FlutterPlugin, ActivityAware { private lateinit var backgroundChannel: MethodChannel private val notificationPermission = NotificationPermission() private var activityBinding: ActivityPluginBinding? = null + private lateinit var pipChannel: MethodChannel + private lateinit var pipEvents: EventChannel + private var pipEventSink: EventChannel.EventSink? = null + private val pictureInPicture = PictureInPictureManager { viewId, active -> + pipEventSink?.success(mapOf("viewId" to viewId, "isActive" to active)) + } override fun onAttachedToEngine(binding: FlutterPlugin.FlutterPluginBinding) { context = binding.applicationContext @@ -43,6 +50,18 @@ class FishjamClientPlugin : FlutterPlugin, ActivityAware { }) backgroundChannel = MethodChannel(binding.binaryMessenger, BACKGROUND_CHANNEL) backgroundChannel.setMethodCallHandler(::onBackgroundMethodCall) + pipChannel = MethodChannel(binding.binaryMessenger, PIP_CHANNEL) + pipChannel.setMethodCallHandler(pictureInPicture::handle) + pipEvents = EventChannel(binding.binaryMessenger, PIP_EVENTS_CHANNEL) + pipEvents.setStreamHandler(object : EventChannel.StreamHandler { + override fun onListen(arguments: Any?, events: EventChannel.EventSink) { + pipEventSink = events + } + + override fun onCancel(arguments: Any?) { + pipEventSink = null + } + }) } private fun onAudioMethodCall(call: MethodCall, result: Result) { @@ -85,12 +104,16 @@ class FishjamClientPlugin : FlutterPlugin, ActivityAware { audioEventSink = null audioOutput.stopObserving() backgroundChannel.setMethodCallHandler(null) + pipChannel.setMethodCallHandler(null) + pipEvents.setStreamHandler(null) + pipEventSink = null } override fun onAttachedToActivity(binding: ActivityPluginBinding) { activityBinding = binding notificationPermission.activity = binding.activity binding.addRequestPermissionsResultListener(notificationPermission) + pictureInPicture.attachActivity(binding.activity) } override fun onDetachedFromActivityForConfigChanges() = onDetachedFromActivity() @@ -103,6 +126,7 @@ class FishjamClientPlugin : FlutterPlugin, ActivityAware { activityBinding = null notificationPermission.activity = null notificationPermission.flush() + pictureInPicture.detachActivity() } /** @@ -134,5 +158,7 @@ class FishjamClientPlugin : FlutterPlugin, ActivityAware { private const val AUDIO_CHANNEL = "io.fishjam.client/audio_output" private const val AUDIO_EVENTS_CHANNEL = "io.fishjam.client/audio_output_events" private const val BACKGROUND_CHANNEL = "io.fishjam.client/background" + private const val PIP_CHANNEL = "io.fishjam.client/pip" + private const val PIP_EVENTS_CHANNEL = "io.fishjam.client/pip_events" } } diff --git a/android/src/main/kotlin/io/fishjam/client/pip/PictureInPictureManager.kt b/android/src/main/kotlin/io/fishjam/client/pip/PictureInPictureManager.kt new file mode 100644 index 0000000..951cb22 --- /dev/null +++ b/android/src/main/kotlin/io/fishjam/client/pip/PictureInPictureManager.kt @@ -0,0 +1,371 @@ +package io.fishjam.client.pip + +import android.app.Activity +import android.app.PictureInPictureParams +import android.graphics.Color +import android.os.Build +import android.util.Log +import android.util.Rational +import android.view.Gravity +import android.view.View +import android.view.ViewGroup +import android.widget.FrameLayout +import android.widget.TextView +import androidx.annotation.RequiresApi +import androidx.core.app.OnPictureInPictureModeChangedProvider +import androidx.core.app.PictureInPictureModeChangedInfo +import androidx.core.util.Consumer +import com.cloudwebrtc.webrtc.FlutterWebRTCPlugin +import com.cloudwebrtc.webrtc.utils.EglUtils +import io.flutter.plugin.common.MethodCall +import io.flutter.plugin.common.MethodChannel.Result +import org.webrtc.RendererCommon +import org.webrtc.SurfaceViewRenderer +import org.webrtc.VideoTrack + +/** + * Handles `io.fishjam.client/pip`. Port of the React Native SDK's + * `PIPManager` (`react-native-webrtc/android/.../PIPManager.java`): each + * `PictureInPictureView` on the Dart side registers its own settings here, + * keyed by view id, just like each `WebRTCView` there owns one `PIPManager`. + * + * Android's picture-in-picture window is a property of the `Activity` + * (`enterPictureInPictureMode`), not of a view, so only one view's settings + * can back the window at a time -- the one most recently updated, or the one + * [start] was called on. The React Native SDK has the same constraint for + * the same reason, despite also keeping one manager per view. + * + * While in picture-in-picture the Flutter view is hidden and a + * [SurfaceViewRenderer] fed by the same track fills the window, so the window + * shows only the video whatever the Flutter UI looks like. + */ +class PictureInPictureManager(private val onActiveChanged: (viewId: Int, active: Boolean) -> Unit) { + private var activity: Activity? = null + private var listening = false + + private class ViewState { + var trackId: String? = null + var mirror = false + var objectFitContain = false + var startAutomatically = true + var preferredWidth = 0.0 + var preferredHeight = 0.0 + var viewWidth = 0.0 + var viewHeight = 0.0 + var label = "No video" + var backgroundColor = DEFAULT_PLACEHOLDER_BACKGROUND + var textColor = Color.WHITE + } + + private val views = mutableMapOf() + + /** Whichever view's settings currently back the window. */ + private var currentViewId: Int? = null + private var active = false + + private val hiddenChildren = mutableListOf>() + private var container: FrameLayout? = null + private var renderer: SurfaceViewRenderer? = null + private var rendererTrack: VideoTrack? = null + + private val modeListener = Consumer { info -> + if (info.isInPictureInPictureMode) { + onEnter() + } else { + onExit() + if (currentViewId == null) stopListening() + } + } + + fun attachActivity(activity: Activity) { + this.activity = activity + if (currentViewId != null) { + startListening() + if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) applyParams() + } + } + + fun detachActivity() { + if (active) onExit() + stopListening() + activity = null + } + + fun handle(call: MethodCall, result: Result) { + val args = call.arguments as? Map<*, *> ?: emptyMap() + val id = (args["viewId"] as? Number)?.toInt() + when (call.method) { + "update" -> if (id != null) update(id, args) + "start" -> if (id != null) start(id) + "stop" -> Unit + "unregister" -> if (id != null) unregister(id) + else -> return result.notImplemented() + } + result.success(null) + } + + private fun update(id: Int, args: Map<*, *>) { + if (Build.VERSION.SDK_INT < Build.VERSION_CODES.O) return + val state = views.getOrPut(id) { ViewState() } + state.trackId = args["trackId"] as? String + state.mirror = args["mirror"] as? Boolean ?: false + state.objectFitContain = args["objectFit"] == "contain" + state.startAutomatically = args["startAutomatically"] as? Boolean ?: true + val preferred = args["preferredSize"] as? Map<*, *> + // React Native defaults to 1920×1080. A side of 0 falls back to the + // view size in [buildParams]. + state.preferredWidth = preferredSide(preferred?.get("width"), DEFAULT_PREFERRED_WIDTH) + state.preferredHeight = preferredSide(preferred?.get("height"), DEFAULT_PREFERRED_HEIGHT) + val rect = args["rect"] as? Map<*, *> + state.viewWidth = (rect?.get("width") as? Number)?.toDouble() ?: 0.0 + state.viewHeight = (rect?.get("height") as? Number)?.toDouble() ?: 0.0 + state.label = args["fallbackLabel"] as? String ?: "No video" + state.backgroundColor = (args["fallbackBackgroundColor"] as? Number)?.toInt() + ?: DEFAULT_PLACEHOLDER_BACKGROUND + state.textColor = (args["fallbackTextColor"] as? Number)?.toInt() ?: Color.WHITE + + val takesOver = currentViewId != id + currentViewId = id + if (takesOver && active) onActiveChanged(id, true) + + if (active) { + if (state.trackId == null) { + showNoVideo(state) + } else { + renderer?.let { + it.setMirror(state.mirror) + it.setScalingType(scalingType(state)) + } + attachRendererTrack(state) + } + } + + startListening() + applyParams() + } + + @RequiresApi(Build.VERSION_CODES.O) + private fun start(id: Int) { + val activity = activity ?: return + val state = views[id] ?: return + currentViewId = id + try { + activity.enterPictureInPictureMode(buildParams(state)) + } catch (e: IllegalStateException) { + Log.e(TAG, "Failed to enter picture-in-picture", e) + } + } + + private fun unregister(id: Int) { + val state = views.remove(id) + if (currentViewId != id) return + currentViewId = null + if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S && state != null) { + state.startAutomatically = false + val activity = activity + if (activity != null) { + try { + activity.setPictureInPictureParams(buildParams(state)) + } catch (e: IllegalStateException) { + Log.e(TAG, "Failed to update picture-in-picture params", e) + } + } + } + if (active) { + // The window stays open: restoring the Flutter view now would show + // the app's UI in it, and Android ignores a close request. Show + // "No video" until another view takes over or the user closes it. + showNoVideo(state) + return + } + stopListening() + } + + private fun startListening() { + if (listening) return + val provider = activity as? OnPictureInPictureModeChangedProvider + if (provider == null) { + if (activity != null) { + Log.w( + TAG, + "Picture-in-picture needs MainActivity to extend FlutterFragmentActivity", + ) + } + return + } + provider.addOnPictureInPictureModeChangedListener(modeListener) + listening = true + } + + private fun stopListening() { + if (!listening) return + (activity as? OnPictureInPictureModeChangedProvider) + ?.removeOnPictureInPictureModeChangedListener(modeListener) + listening = false + } + + @RequiresApi(Build.VERSION_CODES.O) + private fun buildParams(state: ViewState): PictureInPictureParams { + val builder = PictureInPictureParams.Builder() + val width = if (state.preferredWidth > 0) state.preferredWidth else state.viewWidth + val height = if (state.preferredHeight > 0) state.preferredHeight else state.viewHeight + if (width > 0 && height > 0) { + // Android rejects ratios outside 1:2.39 .. 2.39:1. + val ratio = (width / height).coerceIn(1 / MAX_RATIO, MAX_RATIO) + builder.setAspectRatio(Rational((ratio * 10_000).toInt(), 10_000)) + } + if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S) { + builder.setAutoEnterEnabled(state.startAutomatically && listening) + } + return builder.build() + } + + private fun applyParams() { + if (Build.VERSION.SDK_INT < Build.VERSION_CODES.O) return + val activity = activity ?: return + val state = currentViewId?.let { views[it] } ?: return + try { + activity.setPictureInPictureParams(buildParams(state)) + } catch (e: IllegalStateException) { + Log.e(TAG, "Failed to update picture-in-picture params", e) + } + } + + private fun onEnter() { + setActive(true) + val activity = activity ?: return + val state = currentViewId?.let { views[it] } ?: return + val root = activity.findViewById(android.R.id.content) ?: return + + hiddenChildren.clear() + for (i in 0 until root.childCount) { + val child = root.getChildAt(i) + hiddenChildren.add(child to child.visibility) + child.visibility = View.GONE + } + + val surface = SurfaceViewRenderer(activity).apply { + setBackgroundColor(Color.BLACK) + setMirror(state.mirror) + setScalingType(scalingType(state)) + } + val frame = FrameLayout(activity).apply { + setBackgroundColor(Color.BLACK) + addView(surface, FrameLayout.LayoutParams(MATCH_PARENT, MATCH_PARENT)) + } + root.addView(frame, FrameLayout.LayoutParams(MATCH_PARENT, MATCH_PARENT)) + container = frame + + val eglContext = EglUtils.getRootEglBaseContext() + if (eglContext == null) { + Log.e(TAG, "Cannot create the picture-in-picture renderer: no EGL context") + return + } + try { + surface.init(eglContext, object : RendererCommon.RendererEvents { + override fun onFirstFrameRendered() { + surface.post { surface.setBackgroundColor(Color.TRANSPARENT) } + } + + override fun onFrameResolutionChanged(width: Int, height: Int, rotation: Int) {} + }) + renderer = surface + attachRendererTrack(state) + } catch (e: RuntimeException) { + Log.e(TAG, "Failed to initialize the picture-in-picture renderer", e) + } + } + + private fun attachRendererTrack(state: ViewState) { + val surface = renderer ?: return + rendererTrack?.let { track -> runCatching { track.removeSink(surface) } } + rendererTrack = null + val track = state.trackId?.let { + FlutterWebRTCPlugin.sharedSingleton?.getTrackForId(it, null) as? VideoTrack + } + track ?: return + try { + track.addSink(surface) + rendererTrack = track + surface.visibility = View.VISIBLE + container?.findViewWithTag(FALLBACK_TAG)?.let { container?.removeView(it) } + } catch (e: IllegalStateException) { + Log.e(TAG, "Failed to add the picture-in-picture renderer to the track", e) + } + } + + private fun showNoVideo(state: ViewState?) { + detachRendererTrack() + val frame = container ?: return + renderer?.visibility = View.GONE + val label = state?.label ?: "No video" + val background = state?.backgroundColor ?: DEFAULT_PLACEHOLDER_BACKGROUND + val textColor = state?.textColor ?: Color.WHITE + frame.setBackgroundColor(background) + val existing = frame.findViewWithTag(FALLBACK_TAG) + if (existing != null) { + existing.text = label + existing.setTextColor(textColor) + return + } + frame.addView( + TextView(frame.context).apply { + tag = FALLBACK_TAG + text = label + setTextColor(textColor) + gravity = Gravity.CENTER + textSize = 18f + }, + FrameLayout.LayoutParams(MATCH_PARENT, MATCH_PARENT), + ) + } + + private fun detachRendererTrack() { + val surface = renderer ?: return + rendererTrack?.let { track -> runCatching { track.removeSink(surface) } } + rendererTrack = null + surface.clearImage() + } + + private fun onExit() { + setActive(false) + renderer?.let { surface -> + rendererTrack?.let { track -> runCatching { track.removeSink(surface) } } + surface.release() + } + renderer = null + rendererTrack = null + + val root = activity?.findViewById(android.R.id.content) + container?.let { root?.removeView(it) } + container = null + for ((child, visibility) in hiddenChildren) child.visibility = visibility + hiddenChildren.clear() + } + + private fun setActive(value: Boolean) { + if (active == value) return + active = value + currentViewId?.let { onActiveChanged(it, value) } + } + + private fun preferredSide(value: Any?, fallback: Double): Double = + (value as? Number)?.toDouble() ?: fallback + + private fun scalingType(state: ViewState) = + if (state.objectFitContain) { + RendererCommon.ScalingType.SCALE_ASPECT_FIT + } else { + RendererCommon.ScalingType.SCALE_ASPECT_FILL + } + + private companion object { + const val TAG = "FishjamPictureInPicture" + const val FALLBACK_TAG = "pip-fallback" + const val DEFAULT_PLACEHOLDER_BACKGROUND = 0xFF1C2945.toInt() + const val DEFAULT_PREFERRED_WIDTH = 1920.0 + const val DEFAULT_PREFERRED_HEIGHT = 1080.0 + const val MAX_RATIO = 2.39 + const val MATCH_PARENT = ViewGroup.LayoutParams.MATCH_PARENT + } +} diff --git a/docs/parity.md b/docs/parity.md index 2c21e4a..6495eca 100644 --- a/docs/parity.md +++ b/docs/parity.md @@ -77,6 +77,20 @@ against the real `FishjamClient` in `test/client/client_fixtures_test.dart`. | `ReceiveLivestreamResult.stream` / `stop()` / `getStatistics()` | same names | | | `LivestreamError` (string enum, thrown) | `LivestreamException(LivestreamError)` | | +## Picture-in-picture (React Native SDK) + +The React Native SDK's picture-in-picture (`@fishjam-cloud/react-native-webrtc`, SPEC §20d): + +| React Native SDK | Dart | Notes | +|---|---|---| +| `RTCPIPView` | `PictureInPictureView` | owns its renderer; `mediaStream` is a `MediaStream?` | +| `pip: RTCPIPOptions` | `pipOptions: PictureInPictureOptions?` | same fields and defaults; `null` means not offered | +| `enabled`, `preferredSize`, `startAutomatically`, `stopAutomatically`, `allowsCameraInBackground` | same names | `preferredSize` is a `Size`, default 1920×1080 | +| `pip.fallbackView` | `fallbackLabel`, `fallbackBackgroundColor`, `fallbackTextColor` | drawn in the app and in the window (D24) | +| `onPictureInPictureChange({nativeEvent: {isActive}})` | `onPictureInPictureChange(bool isActive)` | | +| `startPIP(ref)` / `stopPIP(ref)` | `PictureInPictureController.start()` / `stop()` | the controller is passed to the view | +| | `PictureInPictureController.isActive` | | + ## Errors, guards and constants | Web SDK | Dart | Notes | diff --git a/example/android/app/src/main/AndroidManifest.xml b/example/android/app/src/main/AndroidManifest.xml index 50bb465..22fb203 100644 --- a/example/android/app/src/main/AndroidManifest.xml +++ b/example/android/app/src/main/AndroidManifest.xml @@ -26,6 +26,7 @@ android:theme="@style/LaunchTheme" android:configChanges="orientation|keyboardHidden|keyboard|screenSize|smallestScreenSize|locale|layoutDirection|fontScale|screenLayout|density|uiMode" android:hardwareAccelerated="true" + android:supportsPictureInPicture="true" android:windowSoftInputMode="adjustResize">