May 27, 2023

Meta’s New Multilingual speech-to-text, text-to-speech

on Meta AI:

Collecting audio data for thousands of languages was our first challenge because the largest existing speech datasets cover at most 100 languages. To overcome it, we turned to religious texts, such as the Bible, that have been translated in many different languages and whose translations have been widely studied for text-based language translation research.

Something very ironic about using the Bible to train multilingual models. The Tower of Babel is right at the start.

We trained multilingual speech recognition models on over 1,100 languages using a 1B parameter wav2vec 2.0 model. As the number of languages increases, performance does decrease, but only very slightly: Moving from 61 to 1,107 languages increases the character error rate by only about 0.4 percent but increases the language coverage by over 18 times.

The number of languages is mind-blowing to me. A functional universal translator is at hand.

In a like-for-like comparison with OpenAI’s Whisper, we found that models trained on the Massively Multilingual Speech data achieve half the word error rate, but Massively Multilingual Speech covers 11 times more languages. This demonstrates that our model can perform very well compared with the best current speech models.

Was very excited about this, but some commenters pointed out that while most languages error rates is superior, it error rate is similar to Whisper in English.


Previous post
Microsoft Fabric AI Data Platform Arun Ulagaratchagan, on Azure Blog: Today we are unveiling Microsoft Fabric—an end-to-end, unified analytics platform that brings together all the
Next post
Pocket Getting Redesigned iOS App Jay Peters, on The Verge: Reader app Pocket is launching a redesigned version of its iOS app, Pocket owner Mozilla announced on Tuesday. The