Emotion-Recognition
Emotions are at the essence of the human experience.
They influence virtually everything we do and often determine the way
we perceive the world around us. Despite their undeniable relevance,
emotions have long been ignored in technology development, limiting the
scope of human-machine interactions and undermining user experience.
Recent developments in affective computing, however, promise to change
this scenario. Ongoing efforts are pushing the boundaries of artificial
intelligence and building technology that recognizes, interprets, and
responds to human emotions. The present article will discuss recent
advancements and potential applications of emotion-aware technology. It
will also present the R&D tax credit opportunity available for
innovative companies engaged in this area of research.
The Research &
Development Tax Credit
Enacted in 1981, the Federal Research and
Development (R&D) Tax Credit allows a credit of up to 13 percent of
eligible spending for new and improved products and processes.
Qualified research must meet the following four criteria:
New or improved
products, processes, or software
Technological in nature
Elimination of
uncertainty
Process of
experimentation
Eligible costs include employee wages, cost of
supplies, cost of testing, contract research expenses, and costs
associated with developing a patent. On December 18, 2015 President
Obama signed the bill making the R&D Tax Credit permanent.
Beginning in 2016, the R&D credit can be used to offset Alternative
Minimum tax and start-up businesses can utilize the credit against
payroll taxes.
From Artificial
Intelligence to Emotional Intelligence
The emergence of emotion-recognition technology is
largely due to advancements in a branch of artificial intelligence
known as deep learning, or deep neural networks. Though its concept
dates back to the 1950s, deep learning has only recently been fully
implemented, as it relies on computational power and big data that were
not previously available. The idea is that instead of programming a
computer to perform certain tasks, we feed it with a learning algorithm
and expose it to massive amounts of data. The computer then uses this
information to teach itself, leading to unimaginable results.
This revolutionary, new avenue for technological
innovation has allowed for the creation of devices that detect and
appropriately respond to users’ emotions. They do so by gathering cues
from various sources, such as facial expressions, gestures, speech, and
changes in body temperature. Though still in its early days, this
disruptive technology promises to make its way into various industries,
completely transforming how humans and machines interact. Near-term
projections shed light on the huge potential of affective computing.
According to Research and Markets, the emotion detection and
recognition market is expected to grow from $5.66 billion in 2015 to
$22.65 billion in 2020, a compound annual growth rate of 31.9 percent.
Three Sources of
“Emotional” Data
Three sources of information stand out as drivers of
emotion-aware technology, namely, facial expressions, voice, and text.
Many of the recently developed, emotion-recognition software work with
facial expressions, which have long been used to unmask hidden
emotions. In the 1970s, preeminent psychologist Paul Ekman created a
facial-coding system that listed over 5 thousand facial muscular
movements and demonstrated their relevance to human emotion. Dr.
Ekman’s work helped shed light on the staggering complexity of facial
expressions, which often take the form of “micro expressions” and can
even go unnoticed by the human eye.
Nowadays, unprecedented access to visual data have
unleashed enormous potential for measuring nonverbal communication
through the use of algorithms that detect and analyze people’s faces.
Emotion-recognition analytics must take into account numerous,
extremely complex variables, including age and gender, ethnicity and
cultural background, eye tracking, gaze estimation, etc. It is the
utmost example of comprehensive analytics.
The second source of emotional
insight for new technology is voice. Recent technological developments
in speech and voice recognition are revolutionizing the way humans and
computers interact. While the ability to understand and respond
to voice commands is increasingly considered a basic feature, a new
frontier arises: capturing emotion through a person’s voice. The idea
is to analyze vocal modulations, such as high and low intonations, in
order to assess emotional states. By allying speech recognition
and natural language processing, this technology opens a window into
the most expressive output that our body produce - our voices.
Finally, deep learning technology is also helping
extract emotion information from text. Even though most human feelings
are conveyed with facial expression or voice intonation, a considerable
amount still manifests itself through written words. This area of
research is particularly challenging due to the context-dependency of
emotions within text, as well as the need for disambiguation and
co-referencing. Data mining in social media posts, for instance, are
increasingly considered a valuable strategy for gathering information.
Potential Applications
Powerful emotion analytics will enable apps and
devices to engage with users in a human-like manner, empathizing with
and adjusting to different emotional states. Besides creating a better
user experience, this kind of technology will give unprecedented access
to insights about customers, patients, and people in general. The most
preeminent application for emotion-recognition software so far has been
market research. The technology has been used to detect consumers’
reactions to new products and ads, thus allowing companies to go beyond
traditional consumer research methods, such as surveys and focus
groups. A deeper understanding of customers’ emotions could lead to
more satisfying business relationships, with benefits to both ends.
Despite this initial focus on market research, there
is immense potential for more diversified applications. In-car smart
cameras embedded with emotion-sensing technology could, for instance,
prevent traffic accidents due to exhaustion. They could also help
teachers assess whether students are effectively paying attention to
classes, particularly in e-learning settings. Therapeutic applications
could help detect a person’s level of stress or even clarify the
occurrence and severity of symptoms in children or people with impaired
verbal communication. On a more personal level, it could help
users keep track of their emotional wellbeing, particularly in complex,
transitional situations. Police and security applications could also be
envisioned, such as revealing inconsistencies in interrogations. Even
within market research, there is major potential for advancements, such
as using in-store cameras to measure how customers feel about their
“retail experiences” and voice-based software that gives call center
operators information about how their service is being perceived on the
other side of the line.
A growing number of companies and research
institutions throughout the country are working to make these potential
applications a reality. The following paragraphs present an overview of
their innovative activities, which could potentially be eligible for
R&D tax credits.
University Research
Massachusetts
Institute of Technology
Researchers from MIT’s Computer
Science and Artificial Intelligence Laboratory have developed a
technology that detects a person’s emotions with wireless signals. The
“EQ-Radio” measures changes in breathing and heart rhythms and
determines whether someone is excited, happy, angry, or sad. With 87
percent accuracy, the groundbreaking solution shows that
emotion-detection technology doesn’t have to rely on audiovisual cues
or on-body sensors. Instead, it can use wireless signals that reflect
off of a person’s body and back to the measuring device.
Carnegie
Mellon University & Duke University
Researchers at Carnegie Mellon
University’s Human Sensing Laboratory have created IntraFace, advanced
software for facial tracking and emotion recognition. The
state-of-the-art solution simplifies the problem of facial image
analysis, working rapidly, accurately, and with such efficiency that it
can run on most smartphones. In December 2015, the software was made
available to researchers from other institutions, in an effort to
contribute to the development of real-time facial image analysis
applications. Early adopters include researchers at Duke University,
who have already incorporated IntraFace into a screening tool for
autism.
Stanford
University
Facial recognition software
developed at Stanford University promises to help autistic patients
socialize. The application, which runs on Google Glass, scans faces in
real-time and alerts wearers about their emotions. There are over 100
autistic children currently participating in “autism glass” therapy
trials, which could eventually lead to commercialization in a couple of
years. In the words of Dennis Wall, director of the Stanford School of
Medicine's Wall Lab, “The autism glass program is meant to teach
children with autism how to understand what a face is telling them. And
we believe that when that happens they will become more socially
engaged.”
Arizona State
University
Researchers at ASU’s Motivational
Interviewing Laboratory are using emotion-recognition technology to
help improve communication between doctors and patients. Through a
combination of facial tracking and voice analysis software, they are
attempting to give practitioners insight into what makes patients
uncomfortable and how to help them be more honest and forthcoming. Next
steps include measuring both physiological responses and brain
activity.
Emotion-Recognition
Companies
Emotient: In
January 2016, Apple acquired Emotient, a San Diego, California-based
startup that uses artificial intelligence to detect emotions in
real-time. With a massive database of micro expressions, Emotient
developed an algorithm that analyzes and interprets facial patterns.
Capable of extracting more than 90 thousand data points from a single
visual input, the software classifies even the tinniest muscular
movements into different emotional categories. Founded in 2012,
Emotient has worked with major companies, such as Honda Motor Co. and
Procter & Gamble Co., to help analyze consumers’ facial expressions
when trying out products or even looking at them in a store. The
company’s software, which has also been considered for healthcare
applications, could play a strategic role in Apple’s future
developments, from its iAd platform, to the Apple Store experience, as
well as existing AI products, such as Siri.
Affectiva: Based
in Waltham, Massachusetts, MIT Media Lab spin-off Affectiva has created
emotion-recognition software that registers and assesses people’s
facial expressions. The groundbreaking solution is made possible
through a combination of computer vision and deep learning technology
that captures, categorizes, and maps images, classifying them into
different emotional categories. Affectiva aims to enable the
incorporation of emotional intelligence into different interactive
products in areas such as health, education, and robotics. As of May
2016, the company compiled a visual database of over 4.25 million
videos from people in 75 countries, which corresponded to more than 50
billion emotion-related data points. The company believes that its
system will soon be able to identify even the most complex feelings,
such as hope, inspiration, and frustration. Major companies such as
Unilever, Kellogg’s, Mars, and CBS have already used Affectiva’s
innovative technology. Its software development kit is starting to make
its way into a variety of industries, with particular focus on video
games. The company has recently raised $14 million in a Series D
funding round.
Eyeris:
Headquartered in Mountain View, California, Eyeris is the creator of
Emovu, a deep learning-based vision technology that enables everyday
devices to “understand how we feel, who we are, and how we perceive the
environment around us.” Advancing the concept of Ambient Intelligence,
Eyeris aims to add emotion intelligence to devices’ contextual
awareness, thereby enabling adaptive interfaces that guarantee utmost
customization of user experience. The company recently unveiled its
Emovu Driver Monitoring System, an in-car technology that analyzes
drivers’ facial expressions and classifies them into seven different
emotional states. Eyeris believes that the real-time monitoring of
drivers’ feelings can significantly enhance traffic safety.
EyeSee:
Operating in the U.S., Belgium, and Serbia, EyeSee has created an
online, eye-tracking and facial-coding platform. The company’s
innovative, webcam-based technology makes “implicit insights”
accessible online and allows its clients to measure the emotions evoked
by new ads, products, and websites. EyeSee’s solution performs
second-by-second analysis, classifying each webcam frame into one of
seven different categories of emotions. Besides providing data on
emotions over time, the platform calculates an overall valence score
that reflects the predominance of positive or negative feelings.
Kairos: This
Miami, Florida-based firm uses three-dimensional data gathered by its
face analysis algorithms to measure people’s feelings and interactions
with content and products. Aimed at empowering its clients with
meaningful metrics, Kairos offers both emotional analysis and facial
recognition. In 2015, the company received $300,000 in funding from the
Florida Institute and participated in the 500 Startup Distro Miami, a
preeminent startup accelerator program.
Sentio Solutions:
This Palo Alto, California-based startup has created the Feel
Wristband, a wearable device that monitors emotions and stress levels.
The company aims to fill the gap left by fitness trackers, which do not
tap into emotional metrics. Besides measuring movement, blood volume
pressure, and skin temperature, the innovative wristband gathers
information on the user’s Electrodermal Response, which is considered a
great indicator of emotional state. All of the information is processed
by Feel Wristband’s companion app and can be used to help clarify what
kind of situation or tasks affect users’ feelings the most. Sentio aims
to integrate recommendations and wellness programs to help users keep
their emotion on track.
EMOSpeech:
Founded in 2012, this Miami, Florida-based company develops enterprise
software applications for speech emotion recognition. EMOSpeech has
created a speech analytics engine that uses a continuous psychological
model to recognize different emotional states in a wide spectrum of
acoustic information. The innovative solution has been used by call
centers to analyze calls, allowing for real-time feedback to agents and
supervisors.
Beyond Verbal:
This Israeli company has developed groundbreaking technology for
voice-driven emotion analytics. By decoding vocal intonations and their
underlying emotions in real-time, Beyond Verbal aims to change the way
humans and machines interact. In over 21 years of research, the company
has gathered more than 2.5 million voice samples in 40 different
languages. In March 2016, the company unveiled an emotion analytics API
that enables the integration of emotional intelligence into any voice
driven, network-connected solution. It also recently launched the
Beyond mHealth Research Platform, which aims to use voice recognition
to detect diseases. Beyond Verbal is collaborating with hospitals,
research institutes, businesses, and universities in an effort to
identify unique biomarkers in voice samples of patients. Researchers
claim to have already found promising data on diseases such as ALS and
Parkinson’s. According to Beyond Verbal, the ultimate objective is to
create a “common API to diagnose health changes through voice data
gathered through smart cars, internet of things devices in smart homes,
and virtual assistants such as Alexa and Siri”.
Challenges Ahead
Personal privacy is undoubtedly an issue when it
comes to emotion-detection technology. Making sure that information is
used responsibly and within reasonable limits is a major challenge,
particularly due to the novelty of potential applications, which are
largely unregulated. Besides the issue of consent, there is a
considerable risk that feelings could be misinterpreted. Using such
technology for detecting lies or secretly monitoring people could lead
to unfair and biased decisions, particularly if taken as a definitive
source of information, rather than an auxiliary tool. Recently
developed algorithms are highly accurate, however, when it comes to the
level of complexity involved in emotional information, the possibility
of mistakes can never be discarded. On the one hand, it is necessary to
recognize the limitations of existing technologies and work to perfect
them. On the other, it is paramount to establish the necessary
regulations that will preserve the right to privacy and prevent abuses.
Conclusion
Emotions are everywhere, and so is technology.
Recent developments in emotion sensing and analytics promise to bring
together these two ubiquitous elements of our daily lives, creating new
opportunities for interactions between men and machine. Potential
applications pervade various industries, ranging from healthcare and
security to market research and gaming. Companies investing in the
promising field of emotion-recognition technology should take advantage
of R&D tax credits available to support their innovative efforts.
