Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Windows] Fix can't open mic alone when built-in AEC is enabled. #29

Merged
merged 1 commit into from
Jun 7, 2022
Merged

[Windows] Fix can't open mic alone when built-in AEC is enabled. #29

merged 1 commit into from
Jun 7, 2022

Conversation

cloudwebrtc
Copy link
Member

Fixed the issue when the built-in AEC is valid under Windows, the mic cannot be captured when StartRecording is called before StartPlayout.

Trigger conditions for this bug:
1, windows and supports built-in AEC enabled
2. StartRecording is called before StartPlayOut, such as SendOnly mode, or AudioSendStream is enabled before AudioRecvStream

The raised issue, unable to capture mic and no audio stream sent

@cloudwebrtc cloudwebrtc mentioned this pull request Jun 4, 2022
Closed
@cloudwebrtc cloudwebrtc requested review from davidliu and davidzhao June 4, 2022 11:09
@cloudwebrtc cloudwebrtc mentioned this pull request Jun 4, 2022
Closed
@cloudwebrtc cloudwebrtc changed the title fix: Fix can't open mic alone when built-in AEC is enabled. [Windows] Fix can't open mic alone when built-in AEC is enabled. Jun 4, 2022
@cloudwebrtc cloudwebrtc mentioned this pull request Jun 6, 2022
Closed
@cloudwebrtc cloudwebrtc merged commit 57b3569 into m97_release Jun 7, 2022
@cloudwebrtc cloudwebrtc deleted the fix/mic-issue-for-built-aec-enabled branch June 7, 2022 01:13
davidliu pushed a commit that referenced this pull request Jul 17, 2022
@cloudwebrtc @davidliu
fix: Fix can't open mic alone when built-in AEC is enabled. (#29)
d37102f
davidliu pushed a commit that referenced this pull request Jul 21, 2022
@cloudwebrtc @davidliu
fix: Fix can't open mic alone when built-in AEC is enabled. (#29)
50a3302
cloudwebrtc added a commit that referenced this pull request Jan 18, 2023
@cloudwebrtc
fix: Fix can't open mic alone when built-in AEC is enabled. (#29)
c88fe70
cloudwebrtc added a commit that referenced this pull request Jun 6, 2023
@cloudwebrtc
fix: Fix can't open mic alone when built-in AEC is enabled. (#29)
8806408
cloudwebrtc added a commit that referenced this pull request Jun 12, 2023
@cloudwebrtc @hiroshihorie
Merge pull request #2 from webrtc-sdk/listen-only-audio-session
e7e9883 
allow listen-only mode in AudioUnit, adjust when category changes

release mic when category changes (#5)

Change defaults to iOS defaults (#7)

Sync audio session config (#8)

* use `AVAudioSession` defaults
* remove isRecordingEnabled

feat: support bypass voice processing for iOS. (#15)

Remove MacBookPro audio pan right code (#22)

fix: Fix can't open mic alone when built-in AEC is enabled. (#29)

feat: add audio device changes detect for windows. (#41)

* feat: add audio device changes detect for windows.

* Update audio_device_core_win.cc

fix iOS/macOS/Android compile.

fix Linux compile (#47)

AudioUnit: Don't rely on category switch for mic indicator to turn off (#52)

* progress

* tweak

* clean

* simplify audio unit restart

call to SetupAudioBuffersForActiveAudioSession() might not be needed since sample rate won't change during restart. This might help reduce the unwanted noise when restarting audio unit.

* clean

Stop recording on mute (turn off mic indicator) (#55)

* initial impl

* more comments

* more comment

* adjust indent

* comments

Cherry pick audio selection from m97 release (#35)

* [Mac] Allow audio device selection (#21)

* first attempt

* remove unused dep

* init playout / recording

* use AudioDeviceID as guid

* switch device method

* equality

* default device

* `isDefault` property

* dont format default device name

* type param

* bypass

* refactor

* fix

* append Audio to thread labels

* ref

* lk headers

* low level apis

* fix thread checks

Some methods of ADM needs to be run on worker thread, otherwise RTC's thread check will fail.

* switch to default device when removed

* close mixerManager if didn't switch to default device

* default audio device switched

* expose devices update handler

* fix ios compile

* fix bug: don't always recreate RTCAudioDeviceModule

* handle guid.

Co-authored-by: Hiroshi Horie <[email protected]>
cloudwebrtc added a commit that referenced this pull request Jun 12, 2023
@cloudwebrtc @hiroshihorie @davidzhao
Audio Device Optimization
53559dd 
allow listen-only mode in AudioUnit, adjust when category changes (#2)

release mic when category changes (#5)

Change defaults to iOS defaults (#7)

Sync audio session config (#8)

feat: support bypass voice processing for iOS. (#15)

Remove MacBookPro audio pan right code (#22)

fix: Fix can't open mic alone when built-in AEC is enabled. (#29)

feat: add audio device changes detect for windows. (#41)

fix Linux compile (#47)

AudioUnit: Don't rely on category switch for mic indicator to turn off (#52)

Stop recording on mute (turn off mic indicator) (#55)

Cherry pick audio selection from m97 release (#35)

[Mac] Allow audio device selection (#21)

Co-authored-by: Hiroshi Horie <[email protected]>
Co-authored-by: David Zhao <[email protected]>
cloudwebrtc added a commit that referenced this pull request Jun 12, 2023
@cloudwebrtc @hiroshihorie @davidzhao
Audio Device Optimization
22a58c8 
allow listen-only mode in AudioUnit, adjust when category changes (#2)

release mic when category changes (#5)

Change defaults to iOS defaults (#7)

Sync audio session config (#8)

feat: support bypass voice processing for iOS. (#15)

Remove MacBookPro audio pan right code (#22)

fix: Fix can't open mic alone when built-in AEC is enabled. (#29)

feat: add audio device changes detect for windows. (#41)

fix Linux compile (#47)

AudioUnit: Don't rely on category switch for mic indicator to turn off (#52)

Stop recording on mute (turn off mic indicator) (#55)

Cherry pick audio selection from m97 release (#35)

[Mac] Allow audio device selection (#21)

Co-authored-by: Hiroshi Horie <[email protected]>
Co-authored-by: David Zhao <[email protected]>
@cloudwebrtc cloudwebrtc mentioned this pull request Jun 12, 2023
Merged
cloudwebrtc added a commit that referenced this pull request Jun 12, 2023
@cloudwebrtc @hiroshihorie @davidzhao
Audio Device Optimization
56b7119 
allow listen-only mode in AudioUnit, adjust when category changes (#2)

release mic when category changes (#5)

Change defaults to iOS defaults (#7)

Sync audio session config (#8)

feat: support bypass voice processing for iOS. (#15)

Remove MacBookPro audio pan right code (#22)

fix: Fix can't open mic alone when built-in AEC is enabled. (#29)

feat: add audio device changes detect for windows. (#41)

fix Linux compile (#47)

AudioUnit: Don't rely on category switch for mic indicator to turn off (#52)

Stop recording on mute (turn off mic indicator) (#55)

Cherry pick audio selection from m97 release (#35)

[Mac] Allow audio device selection (#21)

Co-authored-by: Hiroshi Horie <[email protected]>
Co-authored-by: David Zhao <[email protected]>
cloudwebrtc added a commit that referenced this pull request Jul 12, 2023
@cloudwebrtc @hiroshihorie @davidzhao
Audio Device Optimization
66e5b16 
allow listen-only mode in AudioUnit, adjust when category changes (#2)

release mic when category changes (#5)

Change defaults to iOS defaults (#7)

Sync audio session config (#8)

feat: support bypass voice processing for iOS. (#15)

Remove MacBookPro audio pan right code (#22)

fix: Fix can't open mic alone when built-in AEC is enabled. (#29)

feat: add audio device changes detect for windows. (#41)

fix Linux compile (#47)

AudioUnit: Don't rely on category switch for mic indicator to turn off (#52)

Stop recording on mute (turn off mic indicator) (#55)

Cherry pick audio selection from m97 release (#35)

[Mac] Allow audio device selection (#21)

RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80)

Co-authored-by: Hiroshi Horie <[email protected]>
Co-authored-by: David Zhao <[email protected]>
cloudwebrtc added a commit that referenced this pull request Jul 13, 2023
@cloudwebrtc @hiroshihorie @davidzhao
Audio Device Optimization
272127d 
allow listen-only mode in AudioUnit, adjust when category changes (#2)

release mic when category changes (#5)

Change defaults to iOS defaults (#7)

Sync audio session config (#8)

feat: support bypass voice processing for iOS. (#15)

Remove MacBookPro audio pan right code (#22)

fix: Fix can't open mic alone when built-in AEC is enabled. (#29)

feat: add audio device changes detect for windows. (#41)

fix Linux compile (#47)

AudioUnit: Don't rely on category switch for mic indicator to turn off (#52)

Stop recording on mute (turn off mic indicator) (#55)

Cherry pick audio selection from m97 release (#35)

[Mac] Allow audio device selection (#21)

RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80)

Co-authored-by: Hiroshi Horie <[email protected]>
Co-authored-by: David Zhao <[email protected]>
cloudwebrtc added a commit that referenced this pull request May 20, 2024
@cloudwebrtc @hiroshihorie @davidzhao
Audio Device Optimization
dd4d064 
allow listen-only mode in AudioUnit, adjust when category changes (#2)

release mic when category changes (#5)

Change defaults to iOS defaults (#7)

Sync audio session config (#8)

feat: support bypass voice processing for iOS. (#15)

Remove MacBookPro audio pan right code (#22)

fix: Fix can't open mic alone when built-in AEC is enabled. (#29)

feat: add audio device changes detect for windows. (#41)

fix Linux compile (#47)

AudioUnit: Don't rely on category switch for mic indicator to turn off (#52)

Stop recording on mute (turn off mic indicator) (#55)

Cherry pick audio selection from m97 release (#35)

[Mac] Allow audio device selection (#21)

RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80)

Co-authored-by: Hiroshi Horie <[email protected]>
Co-authored-by: David Zhao <[email protected]>
cloudwebrtc added a commit that referenced this pull request May 20, 2024
@cloudwebrtc @hiroshihorie @davidzhao
Audio Device Optimization
5652b78 
allow listen-only mode in AudioUnit, adjust when category changes (#2)

release mic when category changes (#5)

Change defaults to iOS defaults (#7)

Sync audio session config (#8)

feat: support bypass voice processing for iOS. (#15)

Remove MacBookPro audio pan right code (#22)

fix: Fix can't open mic alone when built-in AEC is enabled. (#29)

feat: add audio device changes detect for windows. (#41)

fix Linux compile (#47)

AudioUnit: Don't rely on category switch for mic indicator to turn off (#52)

Stop recording on mute (turn off mic indicator) (#55)

Cherry pick audio selection from m97 release (#35)

[Mac] Allow audio device selection (#21)

RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80)

Co-authored-by: Hiroshi Horie <[email protected]>
Co-authored-by: David Zhao <[email protected]>
cloudwebrtc added a commit that referenced this pull request May 21, 2024
@cloudwebrtc @hiroshihorie
@davidzhao @davidliu
Audio Device Optimization
414e5ed 
allow listen-only mode in AudioUnit, adjust when category changes (#2)

release mic when category changes (#5)

Change defaults to iOS defaults (#7)

Sync audio session config (#8)

feat: support bypass voice processing for iOS. (#15)

Remove MacBookPro audio pan right code (#22)

fix: Fix can't open mic alone when built-in AEC is enabled. (#29)

feat: add audio device changes detect for windows. (#41)

fix Linux compile (#47)

AudioUnit: Don't rely on category switch for mic indicator to turn off (#52)

Stop recording on mute (turn off mic indicator) (#55)

Cherry pick audio selection from m97 release (#35)

[Mac] Allow audio device selection (#21)

RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80)

Allow custom audio processing by exposing AudioProcessingModule (#85)

Expose audio sample buffers for Android (#89)

feat: add external audio processor for android. (#103)

android: make audio output attributes modifiable (#118)

Co-authored-by: Hiroshi Horie <[email protected]>
Co-authored-by: David Zhao <[email protected]>
Co-authored-by: davidliu <[email protected]>
cloudwebrtc added a commit that referenced this pull request May 21, 2024
@cloudwebrtc @hiroshihorie
@davidzhao @davidliu
Audio Device Optimization
7454824 
allow listen-only mode in AudioUnit, adjust when category changes (#2)

release mic when category changes (#5)

Change defaults to iOS defaults (#7)

Sync audio session config (#8)

feat: support bypass voice processing for iOS. (#15)

Remove MacBookPro audio pan right code (#22)

fix: Fix can't open mic alone when built-in AEC is enabled. (#29)

feat: add audio device changes detect for windows. (#41)

fix Linux compile (#47)

AudioUnit: Don't rely on category switch for mic indicator to turn off (#52)

Stop recording on mute (turn off mic indicator) (#55)

Cherry pick audio selection from m97 release (#35)

[Mac] Allow audio device selection (#21)

RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80)

Allow custom audio processing by exposing AudioProcessingModule (#85)

Expose audio sample buffers for Android (#89)

feat: add external audio processor for android. (#103)

android: make audio output attributes modifiable (#118)

Fix external audio processor sample rate calculation (#108)

Expose remote audio sample buffers on RTCAudioTrack (#84)

Fix memory leak when creating audio CMSampleBuffer #86

Co-authored-by: Hiroshi Horie <[email protected]>
Co-authored-by: David Zhao <[email protected]>
Co-authored-by: davidliu <[email protected]>
@cloudwebrtc cloudwebrtc mentioned this pull request May 21, 2024
Merged
cloudwebrtc added a commit that referenced this pull request Jun 12, 2024
@cloudwebrtc @hiroshihorie
@davidzhao @davidliu @graszka22 @theomonnom
Update to m125. (#119)
46226b5 
Use M125 as the latest version and migrate historical patches to m125

Patches Group:

## 1. Update README.md
b6c65fc
* Add Apache-2.0 license and some note to README.md. (#9)
* Updated readme detailing changes from original (#42)
* Adding membrane framework (#51)
* Updated readme (#83)

## 2. Audio Device Optimization
7454824
* allow listen-only mode in AudioUnit, adjust when category changes
(#2)
* release mic when category changes
(#5)
* Change defaults to iOS defaults
(#7)
* Sync audio session config
(#8)
* feat: support bypass voice processing for iOS.
(#15)
* Remove MacBookPro audio pan right code
(#22)
* fix: Fix can't open mic alone when built-in AEC is enabled.
(#29)
* feat: add audio device changes detect for windows.
(#41)
* fix Linux compile (#47)
* AudioUnit: Don't rely on category switch for mic indicator to turn off
(#52)
* Stop recording on mute (turn off mic indicator)
(#55)
* Cherry pick audio selection from m97 release
(#35)
* [Mac] Allow audio device selection
(#21)
* RTCAudioDeviceModule.outputDevice / inputDevice getter and setter
(#80)
* Allow custom audio processing by exposing AudioProcessingModule
(#85)
* Expose audio sample buffers for Android
(#89)
* feat: add external audio processor for android.
(#103)
* android: make audio output attributes modifiable
(#118)
* Fix external audio processor sample rate calculation
(#108)
* Expose remote audio sample buffers on RTCAudioTrack
(#84)
* Fix memory leak when creating audio CMSampleBuffer
#86

## 3. Simulcast/SVC support for iOS/Android.
b0b9fe9
    
- Simulcast support for iOS SDK (#4)
- Support for simulcast in Android SDK (#3)
- include simulcast headers for mac also (#10)
- Fix simulcast using hardware encoder on Android (#48)
- Add scalabilityMode support for AV1/VP9. (#90)

## 4. Android improvements.
9aaaab5
- Start/Stop receiving stream method for VideoTrack (#25)
- Properly remove observer upon deconstruction (#26)
- feat: Expose setCodecPreferences/getCapabilities for android. (#61)
- fix: add WrappedVideoDecoderFactory.java. (#74)

## 5. Darwin improvements
a13ea17
- [Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28)
- Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40)
- rotationOverride should not be assign (#44)
- [ObjC] Expose properties / methods required for AV1 codec support
(#60)
- Workaround: Render PixelBuffer in RTCMTLVideoView (#58)
- Improve iOS/macOS H264 encoder (#70)
- fix: fix video encoder not resuming correctly upon foregrounding
(#75).
- add PrivacyInfo.xcprivacy to darwin frameworks. (#112)
- Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114)
- Thread-safe `RTCInitFieldTrialDictionary` (#116)
- Set RTCCameraVideoCapturer initial zoom factor (#121)
- Unlock configuration before starting capture session (#122)

## 6. Desktop Capture for macOS.
841d78f
- [Mac] feat: Support screen capture for macOS. (#24) (#36)
- fix: Get thumbnails asynchronously. (#37)
- fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash
caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be
cropped. (#63)
- Fix the crash when setting the fps of the virtual camera. (#62)

## 7. Frame Cryptor Support.
fc08745
- feat: Frame Cryptor (aes gcm/cbc). (#54)
- feat: key ratchet/derive. (#66)
- fix: skip invalid key when decryption failed. (#81)
- Improve e2ee, add setSharedKey to KeyProvider. (#88)
- add failure tolerance for framecryptor. (#91)
- fix h264 freeze. (#93)
- Fix/send frame cryptor events from signaling thread (#95)
- more improvements for E2EE. (#96)
- remove too verbose logs (#107)
- Add key ring size to keyProviderOptions. (#109)

## 8. Other improvements.
eed6c8a
- Added yuv_helper (#57)
- ABGRToI420, ARGBToI420 & ARGBToRGB24 (#65)
- more yuv wrappers (#87)
- Fix naming for yuv helper (#113)
- Fix missing `RTC_OBJC_TYPE` macros (#100)

---------

Co-authored-by: Hiroshi Horie <[email protected]>
Co-authored-by: David Zhao <[email protected]>
Co-authored-by: davidliu <[email protected]>
Co-authored-by: Angelika Serwa <[email protected]>
Co-authored-by: Théo Monnom <[email protected]>
npazkevich pushed a commit to npazkevich/webrtc that referenced this pull request Jun 24, 2024
@cloudwebrtc @hiroshihorie
@davidzhao @npazkevich
Audio Device Optimization
68f3000 
allow listen-only mode in AudioUnit, adjust when category changes (webrtc-sdk#2)

release mic when category changes (webrtc-sdk#5)

Change defaults to iOS defaults (webrtc-sdk#7)

Sync audio session config (webrtc-sdk#8)

feat: support bypass voice processing for iOS. (webrtc-sdk#15)

Remove MacBookPro audio pan right code (webrtc-sdk#22)

fix: Fix can't open mic alone when built-in AEC is enabled. (webrtc-sdk#29)

feat: add audio device changes detect for windows. (webrtc-sdk#41)

fix Linux compile (webrtc-sdk#47)

AudioUnit: Don't rely on category switch for mic indicator to turn off (webrtc-sdk#52)

Stop recording on mute (turn off mic indicator) (webrtc-sdk#55)

Cherry pick audio selection from m97 release (webrtc-sdk#35)

[Mac] Allow audio device selection (webrtc-sdk#21)

RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (webrtc-sdk#80)

Co-authored-by: Hiroshi Horie <[email protected]>
Co-authored-by: David Zhao <[email protected]>
santhoshvai pushed a commit to GetStream/webrtc that referenced this pull request Nov 20, 2024
@cloudwebrtc @santhoshvai
Update to m125. (#119)
61fc0e6 
Use M125 as the latest version and migrate historical patches to m125

Patches Group:

## 1. Update README.md
webrtc-sdk/webrtc@b6c65fc
* Add Apache-2.0 license and some note to README.md. (#9)
* Updated readme detailing changes from original (#42)
* Adding membrane framework (#51)
* Updated readme (#83)

## 2. Audio Device Optimization
webrtc-sdk/webrtc@7454824
* allow listen-only mode in AudioUnit, adjust when category changes
(webrtc-sdk/webrtc#2)
* release mic when category changes
(webrtc-sdk/webrtc#5)
* Change defaults to iOS defaults
(webrtc-sdk/webrtc#7)
* Sync audio session config
(webrtc-sdk/webrtc#8)
* feat: support bypass voice processing for iOS.
(webrtc-sdk/webrtc#15)
* Remove MacBookPro audio pan right code
(webrtc-sdk/webrtc#22)
* fix: Fix can't open mic alone when built-in AEC is enabled.
(webrtc-sdk/webrtc#29)
* feat: add audio device changes detect for windows.
(webrtc-sdk/webrtc#41)
* fix Linux compile (webrtc-sdk/webrtc#47)
* AudioUnit: Don't rely on category switch for mic indicator to turn off
(webrtc-sdk/webrtc#52)
* Stop recording on mute (turn off mic indicator)
(webrtc-sdk/webrtc#55)
* Cherry pick audio selection from m97 release
(webrtc-sdk/webrtc#35)
* [Mac] Allow audio device selection
(webrtc-sdk/webrtc#21)
* RTCAudioDeviceModule.outputDevice / inputDevice getter and setter
(webrtc-sdk/webrtc#80)
* Allow custom audio processing by exposing AudioProcessingModule
(webrtc-sdk/webrtc#85)
* Expose audio sample buffers for Android
(webrtc-sdk/webrtc#89)
* feat: add external audio processor for android.
(webrtc-sdk/webrtc#103)
* android: make audio output attributes modifiable
(webrtc-sdk/webrtc#118)
* Fix external audio processor sample rate calculation
(webrtc-sdk/webrtc#108)
* Expose remote audio sample buffers on RTCAudioTrack
(webrtc-sdk/webrtc#84)
* Fix memory leak when creating audio CMSampleBuffer
webrtc-sdk/webrtc#86

## 3. Simulcast/SVC support for iOS/Android.
webrtc-sdk/webrtc@b0b9fe9

- Simulcast support for iOS SDK (#4)
- Support for simulcast in Android SDK (#3)
- include simulcast headers for mac also (#10)
- Fix simulcast using hardware encoder on Android (#48)
- Add scalabilityMode support for AV1/VP9. (#90)

## 4. Android improvements.
webrtc-sdk/webrtc@9aaaab5
- Start/Stop receiving stream method for VideoTrack (#25)
- Properly remove observer upon deconstruction (#26)
- feat: Expose setCodecPreferences/getCapabilities for android. (#61)
- fix: add WrappedVideoDecoderFactory.java. (#74)

## 5. Darwin improvements
webrtc-sdk/webrtc@a13ea17
- [Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28)
- Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40)
- rotationOverride should not be assign (#44)
- [ObjC] Expose properties / methods required for AV1 codec support
(#60)
- Workaround: Render PixelBuffer in RTCMTLVideoView (#58)
- Improve iOS/macOS H264 encoder (#70)
- fix: fix video encoder not resuming correctly upon foregrounding
(#75).
- add PrivacyInfo.xcprivacy to darwin frameworks. (#112)
- Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114)
- Thread-safe `RTCInitFieldTrialDictionary` (#116)
- Set RTCCameraVideoCapturer initial zoom factor (#121)
- Unlock configuration before starting capture session (#122)

## 6. Desktop Capture for macOS.
webrtc-sdk/webrtc@841d78f
- [Mac] feat: Support screen capture for macOS. (#24) (#36)
- fix: Get thumbnails asynchronously. (#37)
- fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash
caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be
cropped. (#63)
- Fix the crash when setting the fps of the virtual camera. (#62)

## 7. Frame Cryptor Support.
webrtc-sdk/webrtc@fc08745
- feat: Frame Cryptor (aes gcm/cbc). (#54)
- feat: key ratchet/derive. (#66)
- fix: skip invalid key when decryption failed. (#81)
- Improve e2ee, add setSharedKey to KeyProvider. (#88)
- add failure tolerance for framecryptor. (#91)
- fix h264 freeze. (#93)
- Fix/send frame cryptor events from signaling thread (#95)
- more improvements for E2EE. (#96)
- remove too verbose logs (#107)
- Add key ring size to keyProviderOptions. (#109)

## 8. Other improvements.
webrtc-sdk/webrtc@eed6c8a
- Added yuv_helper (#57)
- ABGRToI420, ARGBToI420 & ARGBToRGB24 (#65)
- more yuv wrappers (#87)
- Fix naming for yuv helper (#113)
- Fix missing `RTC_OBJC_TYPE` macros (#100)

---------

Co-authored-by: Hiroshi Horie <[email protected]>
Co-authored-by: David Zhao <[email protected]>
Co-authored-by: davidliu <[email protected]>
Co-authored-by: Angelika Serwa <[email protected]>
Co-authored-by: Théo Monnom <[email protected]>
# Conflicts:
#	README.md
#	media/engine/webrtc_video_engine.cc
#	media/engine/webrtc_video_engine.h
#	modules/audio_device/audio_device_impl.cc
#	sdk/BUILD.gn
#	sdk/android/BUILD.gn
#	sdk/android/api/org/webrtc/RtpParameters.java
#	sdk/android/api/org/webrtc/SimulcastVideoEncoder.java
#	sdk/android/api/org/webrtc/SimulcastVideoEncoderFactory.java
#	sdk/android/api/org/webrtc/VideoCodecInfo.java
#	sdk/android/src/jni/pc/rtp_parameters.cc
#	sdk/android/src/jni/simulcast_video_encoder.cc
#	sdk/android/src/jni/simulcast_video_encoder.h
#	sdk/android/src/jni/video_codec_info.cc
#	sdk/objc/api/peerconnection/RTCAudioDeviceModule+Private.h
#	sdk/objc/api/peerconnection/RTCAudioDeviceModule.h
#	sdk/objc/api/peerconnection/RTCAudioDeviceModule.mm
#	sdk/objc/api/peerconnection/RTCAudioTrack.mm
#	sdk/objc/api/peerconnection/RTCIODevice+Private.h
#	sdk/objc/api/peerconnection/RTCIODevice.mm
#	sdk/objc/api/peerconnection/RTCPeerConnectionFactory.h
#	sdk/objc/api/peerconnection/RTCPeerConnectionFactory.mm
#	sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.h
#	sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.mm
#	sdk/objc/base/RTCAudioRenderer.h
#	sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.h
#	sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.mm
kanat pushed a commit to GetStream/webrtc that referenced this pull request Nov 22, 2024
@santhoshvai @cloudwebrtc
@hiroshihorie @davidliu
Sync with livekit's m125 (#42)
25e8120 
* Update to m125. (#119)

Use M125 as the latest version and migrate historical patches to m125

Patches Group:

## 1. Update README.md
webrtc-sdk/webrtc@b6c65fc
* Add Apache-2.0 license and some note to README.md. (#9)
* Updated readme detailing changes from original (#42)
* Adding membrane framework (#51)
* Updated readme (#83)

## 2. Audio Device Optimization
webrtc-sdk/webrtc@7454824
* allow listen-only mode in AudioUnit, adjust when category changes
(webrtc-sdk/webrtc#2)
* release mic when category changes
(webrtc-sdk/webrtc#5)
* Change defaults to iOS defaults
(webrtc-sdk/webrtc#7)
* Sync audio session config
(webrtc-sdk/webrtc#8)
* feat: support bypass voice processing for iOS.
(webrtc-sdk/webrtc#15)
* Remove MacBookPro audio pan right code
(webrtc-sdk/webrtc#22)
* fix: Fix can't open mic alone when built-in AEC is enabled.
(webrtc-sdk/webrtc#29)
* feat: add audio device changes detect for windows.
(webrtc-sdk/webrtc#41)
* fix Linux compile (webrtc-sdk/webrtc#47)
* AudioUnit: Don't rely on category switch for mic indicator to turn off
(webrtc-sdk/webrtc#52)
* Stop recording on mute (turn off mic indicator)
(webrtc-sdk/webrtc#55)
* Cherry pick audio selection from m97 release
(webrtc-sdk/webrtc#35)
* [Mac] Allow audio device selection
(webrtc-sdk/webrtc#21)
* RTCAudioDeviceModule.outputDevice / inputDevice getter and setter
(webrtc-sdk/webrtc#80)
* Allow custom audio processing by exposing AudioProcessingModule
(webrtc-sdk/webrtc#85)
* Expose audio sample buffers for Android
(webrtc-sdk/webrtc#89)
* feat: add external audio processor for android.
(webrtc-sdk/webrtc#103)
* android: make audio output attributes modifiable
(webrtc-sdk/webrtc#118)
* Fix external audio processor sample rate calculation
(webrtc-sdk/webrtc#108)
* Expose remote audio sample buffers on RTCAudioTrack
(webrtc-sdk/webrtc#84)
* Fix memory leak when creating audio CMSampleBuffer
webrtc-sdk/webrtc#86

## 3. Simulcast/SVC support for iOS/Android.
webrtc-sdk/webrtc@b0b9fe9

- Simulcast support for iOS SDK (#4)
- Support for simulcast in Android SDK (#3)
- include simulcast headers for mac also (#10)
- Fix simulcast using hardware encoder on Android (#48)
- Add scalabilityMode support for AV1/VP9. (#90)

## 4. Android improvements.
webrtc-sdk/webrtc@9aaaab5
- Start/Stop receiving stream method for VideoTrack (#25)
- Properly remove observer upon deconstruction (#26)
- feat: Expose setCodecPreferences/getCapabilities for android. (#61)
- fix: add WrappedVideoDecoderFactory.java. (#74)

## 5. Darwin improvements
webrtc-sdk/webrtc@a13ea17
- [Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28)
- Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40)
- rotationOverride should not be assign (#44)
- [ObjC] Expose properties / methods required for AV1 codec support
(#60)
- Workaround: Render PixelBuffer in RTCMTLVideoView (#58)
- Improve iOS/macOS H264 encoder (#70)
- fix: fix video encoder not resuming correctly upon foregrounding
(#75).
- add PrivacyInfo.xcprivacy to darwin frameworks. (#112)
- Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114)
- Thread-safe `RTCInitFieldTrialDictionary` (#116)
- Set RTCCameraVideoCapturer initial zoom factor (#121)
- Unlock configuration before starting capture session (#122)

## 6. Desktop Capture for macOS.
webrtc-sdk/webrtc@841d78f
- [Mac] feat: Support screen capture for macOS. (#24) (#36)
- fix: Get thumbnails asynchronously. (#37)
- fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash
caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be
cropped. (#63)
- Fix the crash when setting the fps of the virtual camera. (#62)

## 7. Frame Cryptor Support.
webrtc-sdk/webrtc@fc08745
- feat: Frame Cryptor (aes gcm/cbc). (#54)
- feat: key ratchet/derive. (#66)
- fix: skip invalid key when decryption failed. (#81)
- Improve e2ee, add setSharedKey to KeyProvider. (#88)
- add failure tolerance for framecryptor. (#91)
- fix h264 freeze. (#93)
- Fix/send frame cryptor events from signaling thread (#95)
- more improvements for E2EE. (#96)
- remove too verbose logs (#107)
- Add key ring size to keyProviderOptions. (#109)

## 8. Other improvements.
webrtc-sdk/webrtc@eed6c8a
- Added yuv_helper (#57)
- ABGRToI420, ARGBToI420 & ARGBToRGB24 (#65)
- more yuv wrappers (#87)
- Fix naming for yuv helper (#113)
- Fix missing `RTC_OBJC_TYPE` macros (#100)

---------

Co-authored-by: Hiroshi Horie <[email protected]>
Co-authored-by: David Zhao <[email protected]>
Co-authored-by: davidliu <[email protected]>
Co-authored-by: Angelika Serwa <[email protected]>
Co-authored-by: Théo Monnom <[email protected]>
# Conflicts:
#	README.md
#	media/engine/webrtc_video_engine.cc
#	media/engine/webrtc_video_engine.h
#	modules/audio_device/audio_device_impl.cc
#	sdk/BUILD.gn
#	sdk/android/BUILD.gn
#	sdk/android/api/org/webrtc/RtpParameters.java
#	sdk/android/api/org/webrtc/SimulcastVideoEncoder.java
#	sdk/android/api/org/webrtc/SimulcastVideoEncoderFactory.java
#	sdk/android/api/org/webrtc/VideoCodecInfo.java
#	sdk/android/src/jni/pc/rtp_parameters.cc
#	sdk/android/src/jni/simulcast_video_encoder.cc
#	sdk/android/src/jni/simulcast_video_encoder.h
#	sdk/android/src/jni/video_codec_info.cc
#	sdk/objc/api/peerconnection/RTCAudioDeviceModule+Private.h
#	sdk/objc/api/peerconnection/RTCAudioDeviceModule.h
#	sdk/objc/api/peerconnection/RTCAudioDeviceModule.mm
#	sdk/objc/api/peerconnection/RTCAudioTrack.mm
#	sdk/objc/api/peerconnection/RTCIODevice+Private.h
#	sdk/objc/api/peerconnection/RTCIODevice.mm
#	sdk/objc/api/peerconnection/RTCPeerConnectionFactory.h
#	sdk/objc/api/peerconnection/RTCPeerConnectionFactory.mm
#	sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.h
#	sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.mm
#	sdk/objc/base/RTCAudioRenderer.h
#	sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.h
#	sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.mm

* fix: duplicate simulcast entries

* remove duplicate declaration

* remove duplicate audioDeviceModule

* fix: removed livekit's external audio processor

* fix: add back simulcast factories

* Fix missing RTC_OBJC_TYPE macros

* Fix missing headers and Metal linking

# Conflicts:
#	sdk/BUILD.gn

* Fix Mac Catalyst `RTCCameraVideoCapturer` rotation (#126)

* Fix set frame transformer (#125)

* Fix webrtc_voice_engine not notifying mute change (#128)

Looks like this line was missed during the m125 update.

webrtc-sdk/webrtc@272127d#diff-56f5e0c459b287281ef3b0431d3f4129e8e4be4c6955d845bcb22210f08b7ba5R2289

Adding it back in so that mic is properly released when muted.
# Conflicts:
#	media/engine/webrtc_voice_engine.cc

* android: Allow for skipping checking the audio playstate if needed (#129)

Pausing/stopping the audio track can lead to a race condition against
the AudioTrackThread due to this assert. Normally this is fine since
directly pausing/stopping isn't possible, but user is using reflection
to workaround another audio issue (muted participants still have a
sending audio stream which keeps the audio alive, affecting global sound
if in the background).

Not a full fix, as would like to manually control the audio track
directly (needs a bigger fix to handle proper synchronization before
allowing public access), but this will work through reflection (user
takes responsibility for usage).

* Allow to pass in capture session to RTCCameraVideoCapturer (#132)

Expose initializers to pass in capture session to RTCCameraVideoCapturer
so we can use AVCaptureMultiCamSession etc to capture front and back
simultaneously for iOS.

* Fix NetworkMonitor race condition when dispatching native observers (#135)

There is a race condition in NetworkMonitor where native observers may
be removed concurrently with a notification being dispatched, leading to
a dangling pointer dereference (trying to dispatch an observer that was
already removed and destroyed), and from there a crash with access
violation.

By ensuring dispatching to native observers is done within the
synchronization lock that guards additions/removals of native observers
protects against this race condition. Since native observers callbacks
are posted to the networking thread in the C++ side anyway, there should
be no risk of deadlock/starvation due to long-running observers.

Bug: webrtc:15837
Change-Id: Id2b788f102dbd25de76ceed434c4cd68aa9a569e
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/338643
Reviewed-by: Taylor Brandstetter <[email protected]>
Commit-Queue: Harald Alvestrand <[email protected]>
Reviewed-by: Harald Alvestrand <[email protected]>
Cr-Commit-Position: refs/heads/main@{#42256}

Co-authored-by: Guy Hershenbaum <[email protected]>

* Support for Vision Pro (#131)

TODO:
- [x]  fix compile for RTCCameraVideoCapturer
- [ ]  fix RTCMTLRenderer ?

---------

Co-authored-by: Hiroshi Horie <[email protected]>

* Multicam support (#137)

TODO: 
- [x] Return `.systemPreferredCamera` for devices (visionOS only).
- [x] Use `AVCaptureMultiCamSession` only if `isMultiCamSupported` is
true.
- [x] Silence statusBarOrientation warning.

---------

Co-authored-by: [email protected] <[email protected]>

* tvOS support (#139)

17.0+ only atm

---------

Co-authored-by: cloudwebrtc <[email protected]>

* Add isDisposed to MediaStreamTrack (#140)

* chore: handle invalid cipher from key size. (#142)

* Allow software AEC for Simulator (#143)

~Allow to use "googEchoCancellation" constraint for software AEC.
For devices "googEchoCancellation" should be false to use
VoiceProcessingIO.~

* Fix AudioRenderer crash & expose AVAudioPCMBuffer (#144)

* fix: Fix bug for bypass voice processing. (#147)

* chore: remove aes cbc for framecryptor. (#145)

* Change audio renderer output format (#149)

Instead of converting to Float, output original Int data without
conversion.
Output the raw format and convert when required.

* Fixed issue with missing network interfaces on iOS (#151)

Related issue: webrtc-sdk/webrtc#148
Cherry-pick :
https://webrtc.googlesource.com/src/+/fea60ef8e72fb17b4f8a5363aff7e63ab8027b4f

Fixed issue with network interfaces due to a missing return value in the
"nw_path_enumerate_interfaces(...)" block. Exposed in iOS 18,
RTCNetworkMonitor::initWithObserver will only enumerate the first
interface, instead of all device interfaces

Bug: webrtc:359245764
Change-Id: Ifb9f28c33306c0096476a4afb0cdb4d734e87b2c
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/359541
Auto-Submit: Corby <[email protected]>
Commit-Queue: Jonas Oreland <[email protected]>
Reviewed-by: Kári Helgason <[email protected]>
Reviewed-by: Jonas Oreland <[email protected]>
Cr-Commit-Position: refs/heads/main@{#42818}

Co-authored-by: Corby Hoback <[email protected]>

* Custom audio input for Android (#154)

# Conflicts:
#	sdk/android/api/org/webrtc/audio/JavaAudioDeviceModule.java
#	sdk/android/src/java/org/webrtc/audio/WebRtcAudioRecord.java

---------

Co-authored-by: CloudWebRTC <[email protected]>
Co-authored-by: Hiroshi Horie <[email protected]>
Co-authored-by: davidliu <[email protected]>
Co-authored-by: Guy Hershenbaum <[email protected]>
Co-authored-by: Corby Hoback <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@davidliu davidliu davidliu approved these changes

@davidzhao davidzhao Awaiting requested review from davidzhao

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@cloudwebrtc @davidliu