Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ Run analyze, format and tests before every commit.
| `lib/src/data/` | reliable/lossy data channels and `ChannelMessage` framing |
| `lib/src/stats/` | periodic stats report |
| `lib/src/livestream/` | WHIP/WHEP over plain HTTP, independent of the room connection |
| `lib/src/audio_output/`, `lib/src/background/` | Dart side of the SDK's native plugin: audio routing, Android foreground service, iOS CallKit |
| `lib/src/models/`, `wire.dart`, `logger.dart`, `version.dart` | public immutable types and shared helpers |
| `lib/src/fishjam_client.dart`, `lib/src/livestream_api.dart`, `lib/fishjam_client.dart` | the public facade; wiring and type conversion only |
| `test/fakes/`, `test/support/`, `test/fixtures/` | fakes, fixture replayer, golden sequences |
Expand Down
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,6 @@ Initial release.
- Automatic reconnection (`ReconnectConfig`), optionally re-publishing local tracks.
- Periodic statistics reporting to the server; `getStatistics()`.
- Livestreams over WHIP (`publishLivestream`) and WHEP (`receiveLivestream`).
- Background calls: Android foreground service (`ForegroundService`) and iOS CallKit session (`CallKit`)
keep microphone capture and playback running while the app is in the background.
- Behavioral differences from the Fishjam web SDK are listed in SPEC.md §22.
65 changes: 65 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,71 @@ A lost connection is re-established automatically (`ReconnectConfig`):

Livestreams without a room use `publishLivestream` (WHIP) and `receiveLivestream` (WHEP).

## Background calls

By default the OS suspends the app shortly after it goes to the background, which cuts the call's audio.
Start these after joining and stop them after leaving:

```dart
// Android: a foreground service with an ongoing-call notification. No-op on iOS.
await ForegroundService.instance.start(const ForegroundServiceConfig(
enableMicrophone: true,
enableCamera: true,
notificationTitle: 'Your call is ongoing',
notificationContent: 'Tap to return to the call.',
));

// iOS: shows the call in the system call UI (CallKit). No-op on Android.
await CallKit.instance.startSession(const CallKitConfig(displayName: 'Ann', isVideo: true));
CallKit.instance.actions.listen((action) {
switch (action) {
case CallKitEnded(): // the user ended the call from the system UI
case CallKitMuted(:final isMuted): // apply to the microphone track
default:
}
});

// After leaving:
await ForegroundService.instance.stop();
await CallKit.instance.endSession();
```

Start the foreground service after `getUserMedia`: the camera and microphone service types apply only once
those permissions are granted. On Android 13+ the first `start` asks for the notifications permission.

The SDK's plugin declares nothing in your app, so add the entries yourself.

**Android** `AndroidManifest.xml`:

```xml
<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MICROPHONE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_CAMERA" />
<uses-permission android:name="android.permission.POST_NOTIFICATIONS" />

<application ...>
<service
android:name="io.fishjam.client.foregroundservice.FishjamForegroundService"
android:foregroundServiceType="camera|microphone"
android:exported="false"
android:stopWithTask="true" />
</application>
```

For screen sharing, also add `FOREGROUND_SERVICE_MEDIA_PROJECTION`, add `mediaProjection` to
`foregroundServiceType`, and use this order: `Helper.requestCapturePermission()`, then
`ForegroundService.instance.start(... enableScreenSharing: true)`, then `getDisplayMedia`.

**iOS** `Info.plist`:

```xml
<key>UIBackgroundModes</key>
<array>
<string>audio</string>
<string>voip</string>
</array>
```

For prototyping without a backend, the Fishjam dashboard offers a Sandbox API that returns peer tokens. It is
unauthenticated: never use it in production. The [example app](example/) shows both flows.

Expand Down
56 changes: 56 additions & 0 deletions SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -627,6 +627,62 @@ React Native SDK's `AudioOutputManager` and talks to the SDK's native plugin ove
in progress. The iOS route observer starts when Dart subscribes to `onAudioOutputChanged`, so a change is
delivered even when no device getter has run yet.

## 20c. Background calls

Two singletons keep a call running while the app is in the background. They mirror the React Native SDK's
`useForegroundService` and `useCallKitService` / `useCallKit` / `useCallKitEvent`. Both talk to the SDK's
native plugin over the method channel `io.fishjam.client/background`; CallKit actions arrive on the event
channel `io.fishjam.client/callkit_events`. The client never starts either on its own: the app starts them
when it joins and stops them when it leaves.

### Android foreground service

`ForegroundService.instance.start(ForegroundServiceConfig)` and `stop()`. Both are no-ops on other platforms.

- Config: `enableCamera`, `enableMicrophone`, `enableScreenSharing` (all `false` by default), `channelId`
(default `com.fishjam.foregroundservice.channel`), `channelName` (default `Fishjam Notifications`),
`notificationTitle`, `notificationContent`, `importance` (`low` | `high`, default `high`), `onlyAlertOnce`
(default `false`). Unset strings keep the native defaults.
- `start` sends `startForegroundService` with the config as a map; `importance` is sent by name and unset
strings are omitted. Calling `start` again replaces the config of the running service.
- On Android 13+ the first `start` asks for `POST_NOTIFICATIONS`. A refusal is logged and the service starts
anyway: the notification is hidden but the app keeps running.
- The service type is the union of `camera` (when enabled and `CAMERA` is granted), `microphone` (when enabled
and `RECORD_AUDIO` is granted) and `mediaProjection` (when `enableScreenSharing`, API 29+). With no type
left the service stops.
- `stop` sends `stopForegroundService` and stops the service.
- The plugin's manifest declares nothing. The app declares the service
`io.fishjam.client.foregroundservice.FishjamForegroundService` and the `FOREGROUND_SERVICE*` permissions,
so apps that do not use it never need the matching Play Store declarations.
- Screen sharing: flutter_webrtc reports no capture start, so `mediaProjection` is added as soon as
`enableScreenSharing` is set. Android 14+ requires the user's consent before that, so the order is
`Helper.requestCapturePermission()`, then `start(enableScreenSharing: true)`, then `getDisplayMedia`.

### iOS CallKit

`CallKit.instance`. Every method is a no-op on other platforms; the getters return `false` there and
`actions` never emits.

- `startSession(CallKitConfig)`: `{displayName, handle, isVideo, isDialing}`. `handle` defaults to
`displayName`, `isDialing` to `false`. An empty `displayName` throws `ArgumentError` without calling native
code. Sends `startCallKitSession`. A second session while one is active is ignored by native code.
Without `isDialing` the call is reported as connected right away; with it, the app calls
`reportOutgoingCallConnected()` once the other side joins.
- `endSession([CallEndedReason reason = local])` sends `endCallKitSession` with the reason name. `local` and
`rejected` end the call through a CallKit transaction; the others report the call as ended with the
matching CallKit reason (`missed` → unanswered, `remote` → remote ended, `answeredElsewhere`, `failed`).
- `setHeld(onHold:)` / `setMuted(muted:)` ask CallKit to change the state; the result arrives as an action.
- `hasActiveSession()` and `isHeld()` read native state. They return futures because platform channels have
no synchronous calls.
- `actions` (broadcast) emits `CallKitStarted`, `CallKitEnded(reason)`, `CallKitFailed(message)`,
`CallKitMuted(isMuted)`, `CallKitHeld(isHeld)`. Native events are maps with one key (`started`, `ended`,
`failed`, `muted`, `held`); unknown keys are dropped and an unknown end reason becomes `local`.
- CallKit activates the `AVAudioSession`. The plugin forwards `didActivate` / `didDeactivate` to flutter_webrtc's
`RTCAudioSession`, looked up by name at runtime so the plugin has no build dependency on the WebRTC pod.
This keeps microphone capture and playback running. It does not keep the camera running.
- The app adds `audio` and `voip` to `UIBackgroundModes`. Out of scope: VoIP push, incoming calls, answering,
ring timeouts.

## 21. Logging

One SDK logger, created from `FishjamClient(debug: bool)`. Messages are prefixed with `[FISHJAM]` and emitted
Expand Down
51 changes: 50 additions & 1 deletion android/src/main/kotlin/io/fishjam/client/FishjamClientPlugin.kt
Original file line number Diff line number Diff line change
@@ -1,19 +1,29 @@
package io.fishjam.client

import android.content.Context
import android.util.Log
import io.fishjam.client.foregroundservice.ForegroundServiceController
import io.fishjam.client.foregroundservice.NotificationPermission
import io.flutter.embedding.engine.plugins.FlutterPlugin
import io.flutter.embedding.engine.plugins.activity.ActivityAware
import io.flutter.embedding.engine.plugins.activity.ActivityPluginBinding
import io.flutter.plugin.common.EventChannel
import io.flutter.plugin.common.MethodCall
import io.flutter.plugin.common.MethodChannel
import io.flutter.plugin.common.MethodChannel.Result

class FishjamClientPlugin : FlutterPlugin {
class FishjamClientPlugin : FlutterPlugin, ActivityAware {
private lateinit var context: Context
private lateinit var audioChannel: MethodChannel
private lateinit var audioEvents: EventChannel
private lateinit var audioOutput: AudioOutputManager
private var audioEventSink: EventChannel.EventSink? = null
private lateinit var backgroundChannel: MethodChannel
private val notificationPermission = NotificationPermission()
private var activityBinding: ActivityPluginBinding? = null

override fun onAttachedToEngine(binding: FlutterPlugin.FlutterPluginBinding) {
context = binding.applicationContext
disableFlutterWebrtcAudioRouting()
audioOutput = AudioOutputManager(binding.applicationContext) { event ->
audioEventSink?.success(event)
Expand All @@ -31,6 +41,8 @@ class FishjamClientPlugin : FlutterPlugin {
audioEventSink = null
}
})
backgroundChannel = MethodChannel(binding.binaryMessenger, BACKGROUND_CHANNEL)
backgroundChannel.setMethodCallHandler(::onBackgroundMethodCall)
}

private fun onAudioMethodCall(call: MethodCall, result: Result) {
Expand All @@ -50,11 +62,47 @@ class FishjamClientPlugin : FlutterPlugin {
}
}

private fun onBackgroundMethodCall(call: MethodCall, result: Result) {
when (call.method) {
"startForegroundService" -> {
val config = call.arguments as? Map<*, *> ?: emptyMap<String, Any>()
notificationPermission.request(context) {
ForegroundServiceController.start(context, config)
result.success(null)
}
}
"stopForegroundService" -> {
ForegroundServiceController.stop(context)
result.success(null)
}
else -> result.notImplemented()
}
}
Comment thread
Magmusacy marked this conversation as resolved.
Resolved

override fun onDetachedFromEngine(binding: FlutterPlugin.FlutterPluginBinding) {
audioChannel.setMethodCallHandler(null)
audioEvents.setStreamHandler(null)
audioEventSink = null
audioOutput.stopObserving()
backgroundChannel.setMethodCallHandler(null)
}

override fun onAttachedToActivity(binding: ActivityPluginBinding) {
activityBinding = binding
notificationPermission.activity = binding.activity
binding.addRequestPermissionsResultListener(notificationPermission)
}

override fun onDetachedFromActivityForConfigChanges() = onDetachedFromActivity()

override fun onReattachedToActivityForConfigChanges(binding: ActivityPluginBinding) =
onAttachedToActivity(binding)

override fun onDetachedFromActivity() {
activityBinding?.removeRequestPermissionsResultListener(notificationPermission)
activityBinding = null
notificationPermission.activity = null
notificationPermission.flush()
}

/**
Expand Down Expand Up @@ -85,5 +133,6 @@ class FishjamClientPlugin : FlutterPlugin {
private const val TAG = "FishjamClientPlugin"
private const val AUDIO_CHANNEL = "io.fishjam.client/audio_output"
private const val AUDIO_EVENTS_CHANNEL = "io.fishjam.client/audio_output_events"
private const val BACKGROUND_CHANNEL = "io.fishjam.client/background"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
package io.fishjam.client.foregroundservice

import android.app.Notification
import android.app.NotificationChannel
import android.app.NotificationManager
import android.app.PendingIntent
import android.app.Service
import android.content.Intent
import android.content.pm.ServiceInfo
import android.os.Build
import android.os.IBinder
import android.util.Log

/**
* Foreground service that keeps the app process, and so the call, running while the app is in the
* background. The app declares it in its AndroidManifest.xml.
*/
class FishjamForegroundService : Service() {
override fun onBind(intent: Intent?): IBinder? = null

override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
if (intent != null) {
startInForeground(intent)
}
return START_NOT_STICKY
}

private fun startInForeground(intent: Intent) {
val channelId = intent.getStringExtra(EXTRA_CHANNEL_ID) ?: return
val channelName = intent.getStringExtra(EXTRA_CHANNEL_NAME) ?: return
val importance = intent.getStringExtra(EXTRA_IMPORTANCE) ?: "high"
val types = intent.getIntArrayExtra(EXTRA_TYPES)?.fold(0) { acc, type -> acc or type } ?: 0

createChannel(channelId, channelName, importance)
val notification = buildNotification(
channelId,
intent.getStringExtra(EXTRA_TITLE),
intent.getStringExtra(EXTRA_CONTENT),
importance,
intent.getBooleanExtra(EXTRA_ONLY_ALERT_ONCE, false),
)

if (Build.VERSION.SDK_INT < Build.VERSION_CODES.Q) {
startForeground(NOTIFICATION_ID, notification)
return
}
try {
startForeground(NOTIFICATION_ID, notification, types)
} catch (e: SecurityException) {
// Android 14+ refuses mediaProjection before the user consents to screen capture.
// A service that never reaches the foreground crashes the app, so retry without it.
val withoutProjection = types and ServiceInfo.FOREGROUND_SERVICE_TYPE_MEDIA_PROJECTION.inv()
Log.w(TAG, "Could not start with types $types, retrying with $withoutProjection", e)
startForeground(NOTIFICATION_ID, notification, withoutProjection)
Comment thread
Magmusacy marked this conversation as resolved.
Resolved
}
}

private fun buildNotification(
channelId: String,
title: String?,
content: String?,
importance: String,
onlyAlertOnce: Boolean,
): Notification {
val launchIntent = packageManager.getLaunchIntentForPackage(packageName)?.apply {
addFlags(Intent.FLAG_ACTIVITY_SINGLE_TOP or Intent.FLAG_ACTIVITY_CLEAR_TOP)
}
val contentIntent = launchIntent?.let {
PendingIntent.getActivity(
this, 0, it, PendingIntent.FLAG_UPDATE_CURRENT or PendingIntent.FLAG_IMMUTABLE,
)
}
val builder = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
Notification.Builder(this, channelId)
} else {
@Suppress("DEPRECATION")
Notification.Builder(this).setPriority(
if (importance == "low") Notification.PRIORITY_LOW else Notification.PRIORITY_HIGH,
)
}
return builder
.setContentTitle(title)
.setContentText(content)
.setContentIntent(contentIntent)
.setSmallIcon(android.R.drawable.ic_dialog_info)
.setOnlyAlertOnce(onlyAlertOnce)
.setOngoing(true)
.build()
}

private fun createChannel(channelId: String, channelName: String, importance: String) {
if (Build.VERSION.SDK_INT < Build.VERSION_CODES.O) return
val level = if (importance == "low") {
NotificationManager.IMPORTANCE_LOW
} else {
NotificationManager.IMPORTANCE_HIGH
}
getSystemService(NotificationManager::class.java)
?.createNotificationChannel(NotificationChannel(channelId, channelName, level))
}

companion object {
private const val TAG = "FishjamForegroundService"
private const val NOTIFICATION_ID = 0x46_4A_46_53

const val EXTRA_CHANNEL_ID = "channelId"
const val EXTRA_CHANNEL_NAME = "channelName"
const val EXTRA_TITLE = "notificationTitle"
const val EXTRA_CONTENT = "notificationContent"
const val EXTRA_IMPORTANCE = "importance"
const val EXTRA_ONLY_ALERT_ONCE = "onlyAlertOnce"
const val EXTRA_TYPES = "foregroundServiceTypes"
}
}
Loading
Loading