Social Engineering Blogs

An Aggregator for Blogs About Social Engineering and Related Fields

How to Read Facial Expressions of Emotion when People are Wearing Masks

How can we identify indicators of the face, both veracity and deceptive, even when people are wearing masks?

It is true that when people wear masks, the amount of visual cues that we have (the observable surface of the face) is greatly reduced.

But not all masks are created equal. Some cover the entire mouth and lip area while some have a smaller range. So there is some variability.

Regardless of the type of mask, what you can always see is the area above the nose. Some people may have their masks right above their nose, some people have it on the tip of the nose. But you can almost always see above that.

For those of you who have had our training on reading facial expressions of emotion, there’s a lot of things that go on in the upper half of the face. Clearly you can’t see the mouth region, but you can still gather things in the upper face region.

What expressions can you see?

  • The wrinkling of the disgust will give you an appearance change between the eyes at the top of the nose.
  • The lifting of the upper eyelids in fear or surprise, but especially fear, is also visible.
  • You can see the brow and eye movements of surprise
  • You can also see inner corners of the brows going up in sadness or distress.

So even though we are not able to see the entire face, there are cues we can pick up. Of course, the fact that those cues can occur need to be on our radar or they will be missed.

If the mask is a little narrower on the face even though you cannot see exactly what’s going on on the sides of the mouth, you can still see some appearance changes a bit more laterally of the lips. So you can still kind of get whether the person is smiling or whether there’s a little contempt there. Those are also some important signals we can get.

The big message is: even if someone is wearing a mask, don’t abandon the thought that we can pick up cues in the face.

Want to learn how to read facial expressions of emotion where a portion of the face is covered?

Consider our SubX Subtle Expression Training, originally funded by and developed for US government agencies. This online training will teach you how to recognize subtle expressions – facial expressions of emotion of lower intensity that can occur as fast as 1/16th of a second.

Covid-19, Remote Communication, and Body Language

We are all facing an unprecedented challenge regarding the coronavirus pandemic. Many countries around the world are on lockdown. Businesses have been forced to close and most people have been mandated to stay at home for weeks, if not months.

As a result, communication has also undoubtedly changed. Face to face meetings, interviews and negotiations are now taking place virtually or by phone. In fact, Zoom, a cloud-based video conferencing service, has seen daily users more than quadruple during the past several weeks.

How has this shift from in-person to digital communication changed the way we read people and their body language? What are some of the long-term implications of exclusive or predominant use of virtual meetings? And finally, given the loss of contextual and nonverbal cues in remote meetings, what are some tips on how to have more effective communication?

We sat down with Humintell Director and Psychologist Dr. David Matsumoto to discuss his thoughts on the questions above.

Humans Did Not Evolve To Do 2-dimensional Communication 

The recent shift to almost exclusive digital communication has undoubtedly changed the way we read people. Dr. Matsumoto suggests that this largely stems from the fact that humans did not evolve to do 2-dimensional communication, such as through a computer screen. On the contrary, we have evolved our perceptual senses to live in a 3-dimensional world and our sense of reality is grounded in that fact.

Interactions are based on being live and in person. Being live and in person with somebody and interacting with them is what we’ve evolved to do and what we’ve learned to do all of our lives.

It is true that when communicating digitally, the visual stimulation of a person’s behavior or in their facial expressions is almost the same as being face to face, especially given the level of technology that we have now. In addition, your auditory perception (hearing ability) is present, since you can clearly hear the other person’s tone of voice. But there are clearly some substantial differences between being live vs virtual, including a reduction in nonverbal communication and nonverbal behavior.

What are some of those differences?

Real-Life vs Digital Communication Differences

In a live context when we interact with others, we receive what Dr. Matsumoto calls “full nonverbal packages”. This full package not only includes the face and facial expressions, but also includes voice, gestures, body posture, as well as leg, hand and foot movements.  In-person we also have the additional ability to sense spacing between one another, hear sounds and experience various smells.

Even though we can communicate through these remote communication technologies, there is an overall reduction in the information that we can get nonverbally.

Virtually we have much less of these observable cues. You typically cannot see the whole body and therefore cannot see many of the body movements from the waist down that you can typically see live. There are more distractions: on screen, you can see yourself and focus on what you look like. People’s backgrounds are static. It’s easy to pay less attention to the person you’re speaking to.

Digitally there is also a big data reduction in the ability to detect subtle expressions of the face. 3-dimensional (3D) settings are especially important for reading faces. Expressions are appearance changes that occur because of wrinkle patterns and how light falls on those wrinkle contours. Thus, 2-dimensional (2D) perception cuts down on the number of stimuli we can get to make inferences to read facial expressions of emotion.

What other differences are there?

Context Matters

It’s clear that seeing people’s faces is different than just having audio. However, it’s also clear that even though you may be able to see people’s faces in a remote meeting, that’s different than being live.

Perhaps the biggest way digital communication has changed the way we interact with each other, has to do with context. 

In addition to the 2D and 3D differences mentioned above, Dr. Matsumoto suggests differences between virtual and in-person interactions also has to do with the priming in our minds of the setting or context we are in. In any interaction, there is contextualization that occurs in our head that primes us to be more sensitive to certain things.

From the time we are a few months old, social and cultural rules are all contextualized and transmitted through specific contexts. Everyone around the world learns rules about how to behave, how to think, how to feel, how to act from when we’re infants. Dr. Matsumoto calls this enculturation.

By the time we’re adults, our two major contexts are typically work and home. We learn to think different, act differently, and engage with people differently in those contexts. But now that work is in our home, it’s easy to be confused. We have a different mindset and all our neural networks that tie all our learning, thinking and feelings together are disjointed. There is a lack of depth of visual cues and lack of depth of cognitive processing.

Long Term Implications

In addition to the context and priming differences, being at home means a vast reduction in human contact and interaction we normally have in our daily lives, especially with a variety of people. We probably “see” our family members and friends less frequently, if at all.

The sobering truth is that the current shelters in place around the country and world although necessary, facilitate more social isolation, the effects of which are wide and pervasive. These implications are greater if social isolation is longer.

In addition, the tracking of psychological effects of the increased use of mobile technology and social media has been studied widely. Psychologist Jean Twenge wrote an excellent article for The Atlantic entitled “Have Smartphones Destroyed a Generation?”

Rates of teen depression and suicide have skyrocketed since 2011. It’s not an exaggeration to describe iGen as being on the brink of the worst mental-health crisis in decades. Much of this deterioration can be traced to their phones.

Because it is clear social isolation and withdrawal can have extreme negative impacts, Dr. Matsumoto suggests we need to ensure that we also create time to have actual in-person human interactions. This is good not only for the relationship but keeping our mental health.

Tips on How to Have More Effective Communication

Many of us are currently trying to strike a balance while working from home. What are some tips Dr. Matsumoto has for having more effective communication while communicating virtually? See below!

  • Create a context that is conducive to have work meetings

If possible, carve out a physical workspace at home and treat that space like you would a work environment. If you have a meeting, dress like you’re going to a meeting. Context matters!

  • Check your appearance

Now that we can now see ourselves on screen, it’s good to know how we appear. See what you look like before the meeting; check your camera angle. Check your background, sounds, lighting to ensure it’s appropriate.

  • Stay focused on the interaction

It’s easy to get distracted and to think that others not paying attention

  • Use nonverbal cues to confirm understanding

Use nonverbal cues – faces and voices – along with verbal content and confirm understanding. These remote meetings may take longer, and we may need to make more effort to make sure we’re all on the same page. Ask questions. Confirm understanding verbally. Not being on the same page can have dire consequences.

  • Record and re-watch

If your service allows recording of your interactions, check out how you come off. It’s a great learning experience to see yourself.

The differences between in-person and virtual communication are pervasive and we must be realistic about our expectations and understand that there will be a reduction in nonverbal communication and nonverbal behavior.

Yet, we must also understand that our current situation is temporary. With patience and understanding we will overcome these unprecedented challenges!

Body Language Decoded

Brought to you by CBC

Take a journey deep inside the intriguing world of non-verbal communication. As human beings, our bodies communicate our inner emotions and feelings in ways that can often be easily seen by others, but at other times, are barely visible.

On every continent and in every ethnicity, expressions of emotions such as happiness, surprise, anger and fear are universally recognized. These expressions are hard-wired into our facial muscles for reasons that have everything to do with human evolution and survival of the species. To the trained observer, the way people move can be more revealing than the things people say.

We all use our intuition when trying understand another person’s intentions or needs. We watch the way they walk, and how they stand. We look for their eye-contact and whether they appear comfortable or anxious. But when intuition is augmented by scientifically based observational techniques, the picture becomes much more clear.

These techniques are increasingly being used in the worlds of law enforcement, surveillance and security, as well as politics and commerce. Understanding the “non-verbals” of the people we encounter gives us an edge that can mean the difference between a sale or a no-sale, between an election victory or defeat, between safety and danger, and even between a successful or doomed relationship.

Body Language Decoded features interviews with some of the world’s leading experts in human communication including Humintell’s own Dr. David Matsumoto.