Whisper iOS transcription app

[bjnortier]

This is based on the whisper.cpp implementation by @ggerganov. It uses the tiny model and all processing is done on-device.

It's in the App Store:
https://apps.apple.com/za/app/hello-transcribe/id6443919768

[ggerganov]

Awesome!
There is an idea to shrink the model using [8-bit weights + scaling factors] instead of 16-bit weights, which would reduce the app size by factor of 2. But the accuracy might suffer too much.

[nyadla-sys]

@bjnortier is it possible to implement the iOS app based on whisper.tflite(40MB model weights are in int8)

Refer this to run inference in python
https://colab.research.google.com/github/usefulsensors/openai-whisper/blob/main/notebooks/tflite_from_huggingface_whisper.ipynb

Refer README to run inference on C++
https://github.com/usefulsensors/openai-whisper

to run tflite files on iOS refer the below link
https://www.tensorflow.org/lite/guide/ios

[nyadla-sys]

Seems like I would need tom implement a custom Tensorflow Task?

Seems like I would need tom implement a custom Tensorflow Task?

Could you please implement an iOS app using whisper.tflite model ?

[bjnortier]

I'm looking into it... I had some issues getting the TFLite Sound Classifier example app to work, but it seems doable using the C++ log Mel spectrogram

[nyadla-sys]

Please use c++ logmel spectrogram and use inference using tflite framework with coreml/gpu delegate
it may perform better as most of the kernels optimized for iOS using coreml/gpu delegate

[impara]

Other than speed, what is the benefit or the differences between the .tflite & the .cpp implementations?

[bjnortier]

.cpp version has a but more functionality, and I could get .tflite to work at all with the bigger multilingual models

[Abdurrafey-Siddiqui]

This is based on the whisper.cpp implementation by @ggerganov. It uses the tiny model and all processing is done on-device.

It's in the App Store: https://apps.apple.com/za/app/hello-transcribe/id6443919768

Hi could you also add the options to tweak the transcribe settings for example temperature, initial_prompt ect...

[bjnortier]

Those options are in the original OpenAI python implementation but are not available in the .cpp implementation I'm using

[chrismarquardt]

This is pretty awesome. Thanks for making this!. I assume you're using the English-only tiny.en model? Just wondering for us multi-linguals. The models perform quite well in some other languages too.

[floriandotorg]

Awesome project! Was looking for something exactly like this. I think multi-language support, as well as at least the medium model would be necessary to make this really useful.

Also, if you'd add in-app audio recording and live-transcribtion, it would easily be the best transcribing app in the App Store.

[bjnortier]

Working on it!

[AIceegg]

congrats to this amazing project, @bjnortier! this is so much better than current mobile transcription apps! are you planning to implement multi-language (f.e. german) and in-app features? would love to support you with that - unfortunately, not my area.

[v-puppala]

@bjnortier this is really cool! Which language did you use? Objective C or Swift? Also was there any preprocessing you did with the raw audio data collected by the app prior to feeding it into the model?

[bjnortier]

Swift & SwiftUI, but it uses the C++ implementation of Whisper.
The only preprocessing currently is to convert to 16-bit 16kHz PCM (WAV) from other formats.

[TerenzeY]

Love this, but mine seems to be crashing whenever I tap on the Transcribe button. I must be doing something wrong..?

Initializing Whisper...
whisper_model_load: invalid model file '/private/var/containers/Bundle/Application/87E677AB-2598-4D1F-88A5-DE72C1E1DDF9/whisper-ios-demo.app/ggml-tiny.bin' (bad magic)
whisper_init: failed to load model from '/private/var/containers/Bundle/Application/87E677AB-2598-4D1F-88A5-DE72C1E1DDF9/whisper-ios-demo.app/ggml-tiny.bin'
opened WAV. size: 992576

[TerenzeY]

133 bytes...

[TerenzeY]

How can I compile the right one?

[TerenzeY]

any thoughts @bjnortier ?

[bjnortier]

A. You are asking questions about the demo app I put on Github, not the app
I have on the App Store mentioned in this thread, so pls ask question there.
B. You need to install Git LFS to get the correct binary file, it’s 74.1MB as you can see here:
https://github.com/bjnortier/whisper-ios-demo/blob/main/resources/ggml-tiny.bin

[TerenzeY]

got it thanks.

[jsharf]

This needs apple watch integration! It would be awesome if I could just talk into my watch from time to time and get a transcription of it on my phone later.

[bjnortier]

I'll have to get a watch then :)

[whitesided]

Seems like jsharf is volunteering to get you one for the holiday season!

[theocat0]

Hey, I love what you're doing here. I'm kind of new to coding I'm not too sure how this works. But I'm curious if this app takes advantage of Apple silicon?

I know this must be a lot to do, but a really cool suggestion: able to attach the audio files with the transcription and to be able to scrub through the audio file using the transcription. Also, a search bar so you can search your personal library of transcriptions.

Example: say I'm looking for the word "Jeremy," and so I search it up, and it will highlight all the moments in the audio file where it said Jeremy, and you can click it, and it will bring you exactly where it was said on the audio file (you can even have a broad search for all files!!!)

These are only suggestions:))
I love what you're doing!!!!!

[bjnortier]

Awesome suggestions, thanks. There's an app that has some of these features already if you want to try it: https://apps.apple.com/za/app/transcribe-speech-to-text/id1241342461

In the long run I would be able to add this but I have to see if it's commercially viable to put the effort in

[whitesided]

Any chance you could update the platforms so this could be run on an Apple silicon Mac? I run the original python via pipx but I know a few people who'd be delighted to fiddle with it.

[bjnortier]

Possible, just not a high priority right now, will add it to the list :)

[sawhney17]

Check out the MacWhsiper app!

[UEUEE]

Hi, I have a question We need to recognize a specific phrase. And make recognized words bold. Can we use whisper?
I have checked your app and found it takes pretty much time before words appeared. Does it wait till the end of the phrase before show it?

[bjnortier]

Do you mean like "Hey Siri"? It's a very specific phrase and I don't think you need such big (i.e. slower) model to detect a specific phrase. My app does dictation pretty much as fast as the processor can handle it. I.e. it does a transcription, while that's happening it accumulates more audio samples, when it's done it runs again on the current samples.

[cyturralde]

This is a great app. Just curious. Is there a way to make it differentiate between speakers?

[bjnortier]

Thanks! The answer is no currently but could be added, just not sure what you mean by differentiate?

[cyturralde]

In this case I was thinking it would distinguish between speakers and transcribe it out more like a chat like this:

Person 1: text that person one spoke
Person 2: text the person two spoke
Person 1: some more of person one speaking
Person 2: more of person two speaking.

and so on and so on. I think this would be really useful for basically all meetings of various kinds.

[bjnortier]

Ah I see. Thanks. Yes very useful for meetings.
I don't think Whisper has that built-in, so you would have to use another model.
Some discussion here:
#264

But yes would be very useful. Will add it to the list of future features

[sawhney17]

Differentiation in this case would refer to speaker diarization. i.e telling which speaker is speaking what and like splitting the transcription

…

On 26 Jan 2023, 12:19 PM +0400, Ben Nortier ***@***.***>, wrote: Thanks! The answer is no currently but could be added, just not sure what you mean by differentiate? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[bjnortier]

Cool, thanks

[azertiv]

Thank you for this very practical application! Several suggestions:

• Add a widget on the lock screen, which would allow you to save an idea directly and quickly;
• Be able to dictate text directly from the keyboard, but I guess it's complex to implement...

[bjnortier]

1. Hi, the next version will include Shortcuts, so you will be able to create a shortcut to create a note etc., which you can use via Siri. A widget would be useful... will add it to the feature requests
2. I've investigate the keyboard idea and discussed with someone who also wanted that feature. iOS has a hard memory limit for keyboards (device dependent, on mine it's 70MB I think), but it's too small to load even the "tiny" model. This is to ensure a keyboard is fast. The alternative is to make a keyboard that uses a server-side transcription engine, which has privacy implications. I am considering adding server-side transcription for people who do want to use it.

[antgiant]

I don't know if it helps but a while ago I noticed that Microsoft's SwiftKey Keyboard launched an app when you pressed the transcribe button. (Looks like the don't anymore though.) Makes me wonder if they were working around some limitation by launching the full app and starting a local server so that they could stream the audio that way.

[dvkch]

A shortcut would be awesome ! I'd like to be able to make a shortcut to run a video through this app, transcribe once in original language, transcribe a second time in english, and add both subtitles to the video, to be able to send videos to my barely-speaking french friends that want to learn a bit 😊 that would allow them to see everyday phone videos with both french and english subtitles !

If you ever implement a shortcut, it would be amazing to select the language and the english translation when using it.

[aehlke]

@bjnortier how about app extensions? or app clips? not sure if either are usable from within keyboard

[bjnortier]

@aehlke I haven't tried but I think it won't be possible due to memory constraints.

[RootMePLS]

Application is crashed after upgrading to PRO(By all Pro features) and turning on CoreML.

iPhone 11
iOS 16.4.1
App version 2.3.0

After reinstalling the app, I tried to "Restore Purchases", but nothing happened, only after uploading and restarting the app, download buttons appears.

Steps to reproduce:

• Upgrade to PRO
• Download Accurate Multi-lang
• Turn on "Use CoreML"
• Choose Accurate Multi-lang
• Crash

My proposal is, to block Turn on "Use CoreML" if iPhone model < 13

[bjnortier]

Thanks, I will look into why Restore doesn't work :/

I don't want to block CoreML because some people have been able to get it working on an iPhone 12, but I could it off by default for < 13, which would help. That is why I have added messaging and warnings about CoreML to 2.3.0, so people can decide for themselves

[RootMePLS]

Got it

[chrismarquardt]

Loving on-device quality transcription, thanks for making this! I now tried to automate a few things, what I was, hoping was to be able to seamlessly integrate whisper into a shortcut, and then use its output text in the next step. but the shortcuts don’t seem to work on iOS 16.5

“Invalid action metadata
Unfortunately, there is nothing
Shortcuts can do about this; it is due to a bug in the host app. We'd recommend contacting the maker of that app and letting them know of this problem.”

[bjnortier]

Can I add you to TestFlight? Then you can check the latest Beta. There have been some issues with Dictation

[chrismarquardt]

Sure thing, I'd love to give it a spin and report back.

[ninopiamonte]

Hi! Can I try the beta too? I've been running into the same timeout problems with longer sessions. Thank you

[bjnortier]

@ninopiamonte Added you 👍

[ninopiamonte]

Thank you! I've been testing since you added me and the "small.en" model works fine on my iPhone 13.

[chrismarquardt]

By the way, kudos for getting CoreML to work. I run this on an iPhone 13 pro and it's orders of magnitude faster than just on CPU. How is this even possible?

[bjnortier]

Can't really take credit for the CoreML work, I can only take credit for integrating it. It's pretty impressive.

[josegomez]

can you add the option to translate to other languages than english?

[bjnortier]

Whisper doesn't have support for other languages, if I were to add it it would be a separate system (i.e. major effort).

[josegomez]

ah , got it I thought the translation was using another service already.

[7k50]

Just installed Hello Transcribe for iPhone X. Seems not to work, can't get any results: I only see a blue bar repeatedly going from left to right. Perhaps this app isn't compatible with iPhone X? If so, it might be possible to disable download for this app for unsupported devices in apples App Store.

[bjnortier]

It's probably just really slow on the iPhone X. I have users on iPhone 8 who use it it's just really slow so I don't want to disable devices.

[7k50]

It's probably just really slow on the iPhone X. I have users on iPhone 8 who use it it's just really slow so I don't want to disable devices.

You're right, it worked once I waited a few more moments. In that case, please keep it enabled for iPhone X and under. The speed is still relatively useful. I was probably tricked by the constantly resetting blue progress bar. I think that this UX, when it's constantly resetting like that, gives the user (me) the intuition that the app is broken, when also coupled with a slower than average transcription speed as in my case.

[bjnortier]

So the progress bar has a transparent indicator and a solid indicator, I tried to make a UI that doesn't need another spinner or something, but I can see it's confusing. I'll change it

[chrismarquardt]

Feature request for the Shortcuts integration: Would it be possible for the multilinguals among us to make a language selection when calling the Transcribe action from Shortcuts? The reason is the higher CoreML speed and efficiency with a language pre-selected.

[bjnortier]

Ah yes good idea, will add it to the backlog

[TFWol]

What is http://hello-transcribe.b-cdn.net/ ? The app beaconed to that url.

[TFWol]

@bjnortier bump

[bjnortier]

Sorry didn't see this. It's the CDN (bunny.net) for the Whisper models.

[TFWol]

Gotcha, thanks!

[Aamirfarooq13]

I'm looking app or shortcut for transcription multilingual OpenAi whisper v3 support for iPhone 14 Pro Max, anyone know about this ?

[bjnortier]

You can use my app, Large V3 is supported in the Beta, can add you to TestFlight if you'd like (send me your email address).

https://apple.co/49eTzRn

[Aamirfarooq13]

I'm using Hello Transcribe app in my iPhone 14 Pro Max with iOS 17.4.1
Thanks for one time purchase option into app very reasonable price, I bought now I'm tension free no more minutes limit and nothing else, my reason to use this app I run small YouTube channel for that I record my video in my native language Urdu (Pakistan) before upload to YouTube I need SRT file for because when I upload video with SRT file YouTube shows close caption to my videos, it is big benefit YouTube algorithm to understand my text translate in other languages automatically I get global audience reach, this is behind me whole idea, Hello Transcribe today I used after long time because when I was using whisper v3 crash app but today this not happen maybe you fixed issue, anyway, my big concerns I require fine tune model for Urdu (Pakistan) language, Hello Transcribe app voice recognition skip some word in Urdu language, if you can fix this issue it will more useful for me, I hope you find this my message

[mak448a]

Will iOS 15 support ever be added?