Emotions are a core part of the human experience. We all have them. What can be difficult is knowing exactly what another person is feeling. Humanity has been trying to understand emotions and emotional responses for hundreds of years. From Darwin’s theory of emotional universality to modern facial mapping–we’ve been scientifically studying emotions for a while.
Recent studies show that we can identify at least 21 unique facial expressions that are connected to emotional responses in western culture. We’ll explain what that means for AI emotion recognition.
The History of Studies Facial Expressions
Way back in 1898, Charles Darwin was the first to suggest that facial expressions are universal in his publication of The Expression of the Emotions in Man and Animals. It was a core hinge-point in his theory of evolution. He theorized that facial expressions are biologically innate and evolutionarily adaptive.
“It is always advisable to perceive clearly our ignorance”
― Charles Darwin, The Expression of the Emotions in Man and Animals
And we hope Darwin meant it, too! His early theories were found to be inconclusive quickly in favor of culture-specific interpretations of emotion.
By 1972, leading psychologists Paul Ekman, Wallace V. Friesen, and Phoebe Elisworth publish Emotion in the Human Face, where they set out to understand and standardize how we analyze emotions. They really set the stage for the culture-first approach, beyond a scientific approach. They posed their position simply: “If different cultures speak different languages and have varying traditions, why would emotional expression be the same?” They spent over a decade conducting research all over the world to test their theory and fully debunk Darwin’s connection between biology and emotions, so today we know for sure that facial expressions are totally connected with cultural values.
Elkman, Friesen, and Elisworth analyzed people, literature, art, music, and more to understand how emotions are displayed and communicated across all aspects of culture and different humans. We now call their collective research and findings universality studies.
This culture-first approach to universality is widely accepted today, but, what exactly are the facial expressions and emotions that we have been studying for the last 50+ years?
Simple Facial Expressions
So what exactly are the facial expressions we’ve been studying?
Until recently, we were able to identify 7 core facial expressions in western culture:
- Happiness
- Surprise
- Disgust
- Sadness
- Fear
- Anger
- Contempt
These broad categories were determined using surface-level visual analysis. But it probably doesn’t come as a surprise to you that these facial expressions do not map directly–or cleanly–to emotions. If you’ve ever cried happy tears or laughed in anger, you know what we mean.
Guidelines for Researching Facial Expressions
So if emotions are cultural, how do we research facial expressions and analyze them? Of course, with strict adherence to guidelines for best practices. It’s important to say that mixed emotions are completely absent in this model. They are also barely represented as a footnote, so we don’t have many tools to analyze them.
Also, internal emotions are almost impossible to visually analyze. (Even if our moms have some weird sixth sense), but fortunately, recent studies mapped out more expressions using machine learning are able to go much deeper into human emotion than ever before.
21st Century Developments in Facial Expression Analysis
In 2014, scientists at Ohio State University found at least 21 unique and identifiable facial expressions. That triples the core 7 that we have been using since the early 1900s.
“We’ve gone beyond facial expressions for simple emotions like ‘happy’ or ‘sad.’ We found a strong consistency in how people move their facial muscles to express 21 categories of emotions.That is simply stunning. That tells us that these 21 emotions are expressed in the same way by nearly everyone, at least in our culture”.
–Aleix Martinez, Ohio State University
This groundbreaking research proves that emotions are expressed universally in our culture and that the emotions we express are incredibly diverse.
Here is the full list of all 21 facial expressions:
- Happy
- Sad
- Fearful
- Angry
- Surprised
- Disgusted
- Contempt
- Happily surprised
- Sadly fearful
- Sadly angry
- Sadly surprised
- Sadly disgusted
- Fearfully angry
- Fearfully surprised
- Fearfully disgusted
- Angrily surprised
- Angrily disgusted
- Disgustedly surprised
- Appalled
- Hatred
- Awed
These facial expressions mix together the 7 core emotions and prove what psychologists have theorized for decades: most facial expressions are from mixed emotions.
And that makes a lot of sense. When was the last time that you were 100% happy, 100% surprised, or 100% sad? Humans are complex and so are our emotions. This model makes room for that, just like Elevate’s edge-AI facial expression analysis.
These fine points make it easier to actually understand what people are feeling, not just what expression dominates their faces the most.
Cultural Variance
The recent study at Ohio State, and the dozens of others that are ongoing right now, have made awesome strides in how machine learning can be used to scientifically study and understand human emotions. But we are lightyears away from being done.
Just like Darwin and his contemporaries in the 1800s, we’ve proven that people express emotions universally in western culture. But the world is so much bigger than that and modern technology is connecting us more than ever before. Elevate is committed to using diverse datasets of faces from all over the world with varying ethnicities, ages, and genders to better understand and represent people from all over the globe.
How Facial Expressions and Emotions are Different
The way we understand the emotions of others is pretty much connected with what we see on people’s faces. We all know that words are not always enough to transmit what we are feeling and, sometimes, we don´t know how to describe what we are feeling. So facial expressions are different from emotions.
Facial expressions are just one of the many ways that we signal how we are feeling. Analyzing emotions alone is not enough. That’s why multimodal AI analytics is so important when attempting to understand how people feel. At Elevate, we analyze facial expressions, body language, and tonal input. Even with all of these taken into consideration, we are only capturing a small percentage of how a person feels. This analysis gives you the tools you need to bring empathy into the discussion.
As Arlin Cuncic, author of Therapy in Focus: What to Expect from CBT for Social Anxiety Disorder and 7 Weeks to Reduce Anxiety explains:
“The value in understanding facial expressions is to gather information about how the other person is feeling and guide your interaction accordingly. For example, if someone appears disinterested, they might just be tired, and it might be time to end the conversation”
–Arlin Cuncic, master’s in clinical psychology from the York University
