Humans have a hard time reading lips, but AIs don’t.

Lip reading is a very difficult discipline to master and yet essential for the deaf and hard of hearing.

Benedictine monk Pedro Ponce de León is recognized by historians as the first person to have the idea, and the ability to read lips. He also founded a school within his monastery in Ona, Castile at the beginning of the 16th century.

Since then, lip reading (lip reading) has become an art mastered by many people around the world. On the front line, the deaf and hard of hearing who thus manage to communicate with the hearing world. However, this solution still has a number of weaknesses. A 2011 study ensures that a hearing person, who has never really worked this method would be able to recognize between 3 and 4 words in a sentence of 12.

A human can read between 10 and 15% of the words

Overall an average person should manage to understand between 10 and 15% of a speech. If the basic idea had to be grasped in this way, not without difficulty, this would not be the case for the small nuances and subtleties. This same study, conducted by researchers at the University of Oklahoma, ensures that deaf or hard of hearing people who work on lip reading on a daily basis manage to understand nearly 30% of a text.

The best manage to reach a score of 45%. This is more than enough to understand an everyday discussion, but not enough for a more fluid and worked speech. It is this inability to read lips fluently that leads translators in sign language on the sides of our televisions during presidential speeches or during certain programs.

AIs much better than humans

But in recent years, new researchers have been interested in lip-reading. They explain that an artificial intelligence is much stronger than us, in particular because it manages to see subtleties in the micro-movements of our lips. An AI can also process hundreds of thousands of situations every second, while a human is far from this count.

Today’s state-of-the-art intelligent systems boast an accuracy rate of around 95%. It’s much better than a human, but the computer still makes a few mistakes. Yannis Assael, a researcher at the University of Oxford developed LipNet by taking into account not only the movements of the lips, but also of the tongue and teeth.

Some sounds use these parts of our mouth, without us even realizing it. For a human it is impossible to see it, but sophisticated systems manage to make the nuance between a sound created by the teeth and another by the lips. This increases the level of precision of these tools a little further.

A new system that adapts to all languages

With the “Bad Lip Reading” system set up a few years ago by a group of researchers, Fabian Campbell-West, technical director of application development, explains that this method makes it possible to read both English and French or Mandarin.

“Each language has syntax and pronunciation rules that will affect how it is interpreted. In general, the methods of understanding are the same. He explains that tonic languages ​​are a real challenge for their system. “They use the same word with different pitch changes (like musical pitch) to convey meaning. »

But finally this nuance is visible. “The change in tone always results in physiological changes that are manifest. The researcher adds that machine-reading systems also take the whole sentence into account to guess the next word. This allows the number of words to be studied for the AI ​​to be limited in advance, making its work simpler and therefore faster.

This work around AI could materialize in many ways. It would thus be possible to understand the words of silent films, but also to translate a speech to a large audience with almost perfect reliability. For people who are deaf and hard of hearing, having such a developed AI could help them improve and better interpret the small nuances on our lips.

We wish to give thanks to the writer of this short article for this remarkable content

Humans have a hard time reading lips, but AIs don’t.

Explore our social media accounts and other pages related to them