Should web applications be aware of reaction effects added by OS to camera feeds?

[youennf]

See https://bugs.webkit.org/show_bug.cgi?id=262233 for background.
It would seem useful to at least allow web applications to know that such effects are being enabled, similarly to other effects like background blur.

[youennf]

@riju, any thoughts?

[eladalon1983]

useful to at least allow web applications to know

You must have also thought on allowing applications to enable/disable these effects. (Or possibly only enable or only disable.) Anything you could share on that front?

[youennf]

If we expose an API, it would probably be similar to background blur. Some UAs would expose [true, false] capabilities, some might only expose [false] or [true] depending on what the user selected through the OS UI.

[riju]

ChromeOS and macOS currently only allow you to observe background blur changes that the user can make through their operating system UI such as the Control Center in macOS.
Windows allows you to observe and control background blur.

We have the OnCofigurationChange event to allow UAs to detect changes like this. Is this enough or we want something more ?

Also do you know if other platforms have these Reaction Effects like iOS ?

[youennf]

We have the OnCofigurationChange event to allow UAs to detect changes like this. Is this enough or we want something more ?

I think this is enough in terms of generic mechanism.
We might want to have a specific field for OS-level effects, which are not background blur.
I am unsure how much specific to reactions we would like to have it.

Also do you know if other platforms have these Reaction Effects like iOS ?

I only know macOS and iOS have it.

[eladalon1983]

I imagine that some Web applications would want to build their own effects, and not be limited to the subset of reactions the OS provides.

I can also imagine that some Web applications might have users who consider certain OS-provided reactions inappropriate and might wish to disable them for some demographics. (This is a much lesser concern than the previous one. Edit: Crossed out the claim it's a lesser concern after thinking of the case of discussing someone passing away over a video-call and then suddenly getting balloons or fireworks...)

[fippo]

From what I've seen the reactions look very different in terms of style from to the app-chosen emoji reactions (which are useful).

While users of Apple devices may recognize them, overlaying them on streams acquired by getUserMedia and transmitting them to users outside that ecosystem ends up being at least awkward.

[nbutko]

What options are given to iOS native camera apps in this regard? Can they disable injected content into video streams? What about ARKit apps, do those also have OS modified input streams?

[nienkedekker]

My two cents: it'd be really nice to have if we could detect whether reaction effects are enabled or not. In the WebRTC app I work on, we have implemented our own audience reactions and I worry having two different reaction effects can be jarring for users, especially those who are not in the Apple ecosystem and have no idea it's even a thing.

[huibk]

Web Applications losing the ability to get access to a raw camera feed is a significant regression of the web platform and causes various, now documented problems for applications and their customers:

• effects and reactions can conflict technically: a platform effect or reaction distorts processing in the application
• it causes user confusion where you can turn on and off things, why UX is inconsistent
• it increases power consumption when two independent effect pipelines are running
• it enables abuse, bypassing anti abuse features in the application, f.i. sending negative reactions in the class room or sensitive meetings

I'd suggest the spec requires access to raw camera feeds in a similar way that microphone streams can be acquired without processing.

[nbulankin]

It's absolutely strange that we can't control it. In a health app where a doctor and a client call each other, this is absolutely unacceptable; patients are at a loss.

[fippo]

https://www.theverge.com/2023/10/13/23916475/apple-video-reactions-telehealth-therapy-awkward-simplepractice isn't particularly thrilled either

[alexandermaisey]

It's absolutely strange that we can't control it. In a health app where a doctor and a client call each other, this is absolutely unacceptable; patients are at a loss.

I am responsible for a suite of video call applications intended for use on live television broadcast and it's an embarrassment that we cannot control this.

[eladalon1983]

I am responsible for a suite of video call applications intended for use on live television broadcast and it's an embarrassment that we cannot control this.

Thank you, that is a very illustrative example. In my eyes, there is plenty of embarrassment to go around, with ample blushing potential for the person going on live TV, the TV network, the software providers involved, and last but not least, the manufacturers of the OS injecting the reactions.

I believe that, ideally, the OS should deputize the browser turn off reactions on a per-Web-app basis. From the operating system's perspective, that essentially boils down to allowing native applications to allow/suppress individual gesture-driven reactions. In my eyes, this should be palatable for the OS, because gesture-driven reactions are heuristic and non-privacy-sensitive. (Menu-driven reactions would be an exception to the former; blurring an exception to the latter.)

Wdyt, @youennf?

[fippo]

Looks like there has been movement. It would have been great to have a better way of learning than there is a native API soon than a twitter thread and MacRumors:

https://developer.apple.com/documentation/ios-ipados-release-notes/ios-ipados-17_4-release-notes#New-Features

[dontcallmedom-bot]

This issue was discussed in WebRTC March 26 2024 meeting – 26 March 2024 (Should web applications be aware of reaction effects added by OS to camera feeds? #118)

[jnoring-hv]

I work for a company that does video interviewing (both on-demand, and live). This is incredibly embarrassing and unprofessional for job seekers. Absolutely floored Apple did this.