Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Audio visualization helpers #474

Merged
merged 27 commits into from
Oct 27, 2024
Merged

Audio visualization helpers #474

merged 27 commits into from
Oct 27, 2024

Conversation

hiroshihorie
Copy link
Member

No description provided.

vDSP_meanv(ptr, 1, &mean, UInt(count))
}
return mean
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can simplify this function using ArraySlice to give responsibility for safe pointer arithmetic back to swift instead:

    @inline(__always) private func _computeAverage(_ array: ArraySlice<Float>) -> Float {
        var mean: Float = 0
        array.withUnsafeBufferPointer { bufferPtr in
            vDSP_meanv(bufferPtr.baseAddress!, 1, &mean, vDSP_Length(array.count))
        }
        return mean
    }

then you call it above with _computeAverage(magnitudes[magsStartIdx..<magsEndIdx])

let count = stopIdx - startIdx
array.withUnsafeBufferPointer { bufferPtr in
let ptr = bufferPtr.baseAddress! + startIdx
vDSP_meanv(ptr, 1, &mean, UInt(count))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vDSP_Length(count)

public let scaleType: ScaleType

private let bufferHalfSize: Int
private let bufferLog2Size: Int
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should consider declaring these int types as vDSP_Length instead. At the very least, they should be UInt here and cast to vDSP_Length when passed to accelerate later

deinit {
vDSP_destroy_fftsetup(fftSetup)
realPointer.deallocate()
imaginaryPointer.deallocate()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason you're managing raw pointers in this class rather than just using arrays and nested withUnsafeMutableBufferPointer calls as in the apple sample code? looks scary to me 🤷 https://developer.apple.com/documentation/accelerate/vdsp/fast_fourier_transforms/finding_the_component_frequencies_in_a_composite_sine_wave

import Foundation

// Simple ring-buffer used for internal audio processing. Not thread-safe.
class FloatRingBuffer {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't really matter but you could call this RingBuffer<T: Numeric> and replace all of the Float with T below and it would still work as-is.

@hiroshihorie hiroshihorie requested a review from bcherry October 5, 2024 04:13
@hiroshihorie
Copy link
Member Author

@bcherry Optimized the code can you take a look 🙏

@hiroshihorie hiroshihorie merged commit 26d7725 into main Oct 27, 2024
12 checks passed
@hiroshihorie hiroshihorie deleted the hiroshi/audio-visualize branch October 27, 2024 15:43
hiroshihorie added a commit that referenced this pull request Nov 6, 2024
commit 2e05b21
Author: hiroshihorie <[email protected]>
Date:   Mon Oct 28 22:21:56 2024 +0900

    v2.0.17

commit 86a8634
Author: hiroshihorie <[email protected]>
Date:   Tue Oct 29 15:02:17 2024 +0900

    Move parts of visualization code into components

commit 027fea7
Author: hiroshihorie <[email protected]>
Date:   Tue Oct 29 02:25:54 2024 +0900

    rtc 125.6422.11

commit 26d7725
Author: Hiroshi Horie <[email protected]>
Date:   Mon Oct 28 00:43:37 2024 +0900

    Audio visualization helpers (#474)

commit 4a73d39
Author: hiroshihorie <[email protected]>
Date:   Sun Oct 27 22:59:11 2024 +0900

    swiftformat

commit 04bf827
Author: Bogdan Vatamanu <[email protected]>
Date:   Sun Oct 27 15:57:21 2024 +0200

    Remove leaked renderers appeared on camera switch (#507)

    Hello @hiroshihorie!

    I found that inside **VideoView** on the camera switch the previous
    **primaryRenderer** gets leaked because it's not removed from the view's
    hierarchy when the transition ends.

    See:

    ![image](https://github.com/user-attachments/assets/b57769be-57c6-4a66-b7b6-d55aeb6bbedd)

    Co-authored-by: Hiroshi Horie <[email protected]>

commit 89a0fa2
Author: Bogdan Vatamanu <[email protected]>
Date:   Sun Oct 27 13:14:05 2024 +0200

    Fix boundaries setting for "SampleBufferVideoRenderer" re-layouting (#506)

    Hello @hiroshihorie!
    I'm experiencing issues with the camera's front-to-back switch feature,
    which occasionally causes the video frame to be incorrectly oriented in
    landscape mode when it should be in portrait mode.

    To resolve this issue, I've found that it's essential to update the
    frame of the **AVSampleBufferDisplayLayer** after applying the
    transformation matrix. This is because the transformation matrix also
    rotates the frame, which can result in incorrect orientation if the
    frame is not updated accordingly.

commit 5869cea
Author: Hiroshi Horie <[email protected]>
Date:   Thu Oct 24 18:38:51 2024 +0900

    Agent state property (#495)

commit 62a70ea
Author: hiroshihorie <[email protected]>
Date:   Sun Oct 20 00:22:27 2024 +0900

    Temporarily remove strict concurrency warnings

commit 1c9a681
Author: Hiroshi Horie <[email protected]>
Date:   Sun Oct 20 00:06:47 2024 +0900

    WebRTC 125.6422.10 (#502)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants