The R&D Tax Credit Aspects of Emotion-Recognition Technology

By and

        Emotions are at the essence of the human experience. They influence virtually everything we do and often determine the way we perceive the world around us. Despite their undeniable relevance, emotions have long been ignored in technology development, limiting the scope of human-machine interactions and undermining user experience. Recent developments in affective computing, however, promise to change this scenario. Ongoing efforts are pushing the boundaries of artificial intelligence and building technology that recognizes, interprets, and responds to human emotions. The present article will discuss recent advancements and potential applications of emotion-aware technology. It will also present the R&D tax credit opportunity available for innovative companies engaged in this area of research.

The Research & Development Tax Credit

        Enacted in 1981, the Federal Research and Development (R&D) Tax Credit allows a credit of up to 13 percent of eligible spending for new and improved products and processes. Qualified research must meet the following four criteria:

  • New or improved products, processes, or software
  • Technological in nature
  • Elimination of uncertainty
  • Process of experimentation

        Eligible costs include employee wages, cost of supplies, cost of testing, contract research expenses, and costs associated with developing a patent. On December 18, 2015 President Obama signed the bill making the R&D Tax Credit permanent.  Beginning in 2016, the R&D credit can be used to offset Alternative Minimum tax and start-up businesses can utilize the credit against payroll taxes.

From Artificial Intelligence to Emotional Intelligence

        The emergence of emotion-recognition technology is largely due to advancements in a branch of artificial intelligence known as deep learning, or deep neural networks. Though its concept dates back to the 1950s, deep learning has only recently been fully implemented, as it relies on computational power and big data that were not previously available. The idea is that instead of programming a computer to perform certain tasks, we feed it with a learning algorithm and expose it to massive amounts of data. The computer then uses this information to teach itself, leading to unimaginable results.

        This revolutionary, new avenue for technological innovation has allowed for the creation of devices that detect and appropriately respond to users’ emotions. They do so by gathering cues from various sources, such as facial expressions, gestures, speech, and changes in body temperature. Though still in its early days, this disruptive technology promises to make its way into various industries, completely transforming how humans and machines interact. Near-term projections shed light on the huge potential of affective computing. According to Research and Markets, the emotion detection and recognition market is expected to grow from $5.66 billion in 2015 to $22.65 billion in 2020, a compound annual growth rate of 31.9 percent.

Three Sources of “Emotional” Data

        Three sources of information stand out as drivers of emotion-aware technology, namely, facial expressions, voice, and text. Many of the recently developed, emotion-recognition software work with facial expressions, which have long been used to unmask hidden emotions. In the 1970s, preeminent psychologist Paul Ekman created a facial-coding system that listed over 5 thousand facial muscular movements and demonstrated their relevance to human emotion. Dr. Ekman’s work helped shed light on the staggering complexity of facial expressions, which often take the form of “micro expressions” and can even go unnoticed by the human eye.

        Nowadays, unprecedented access to visual data have unleashed enormous potential for measuring nonverbal communication through the use of algorithms that detect and analyze people’s faces. Emotion-recognition analytics must take into account numerous, extremely complex variables, including age and gender, ethnicity and cultural background, eye tracking, gaze estimation, etc. It is the utmost example of comprehensive analytics.

        The second source of emotional insight for new technology is voice. Recent technological developments in speech and voice recognition are revolutionizing the way humans and computers interact.  While the ability to understand and respond to voice commands is increasingly considered a basic feature, a new frontier arises: capturing emotion through a person’s voice. The idea is to analyze vocal modulations, such as high and low intonations, in order to assess emotional states.  By allying speech recognition and natural language processing, this technology opens a window into the most expressive output that our body produce - our voices.

        Finally, deep learning technology is also helping extract emotion information from text. Even though most human feelings are conveyed with facial expression or voice intonation, a considerable amount still manifests itself through written words. This area of research is particularly challenging due to the context-dependency of emotions within text, as well as the need for disambiguation and co-referencing. Data mining in social media posts, for instance, are increasingly considered a valuable strategy for gathering information.

Potential Applications

        Powerful emotion analytics will enable apps and devices to engage with users in a human-like manner, empathizing with and adjusting to different emotional states. Besides creating a better user experience, this kind of technology will give unprecedented access to insights about customers, patients, and people in general. The most preeminent application for emotion-recognition software so far has been market research. The technology has been used to detect consumers’ reactions to new products and ads, thus allowing companies to go beyond traditional consumer research methods, such as surveys and focus groups. A deeper understanding of customers’ emotions could lead to more satisfying business relationships, with benefits to both ends.

        Despite this initial focus on market research, there is immense potential for more diversified applications. In-car smart cameras embedded with emotion-sensing technology could, for instance, prevent traffic accidents due to exhaustion. They could also help teachers assess whether students are effectively paying attention to classes, particularly in e-learning settings. Therapeutic applications could help detect a person’s level of stress or even clarify the occurrence and severity of symptoms in children or people with impaired verbal communication.  On a more personal level, it could help users keep track of their emotional wellbeing, particularly in complex, transitional situations. Police and security applications could also be envisioned, such as revealing inconsistencies in interrogations. Even within market research, there is major potential for advancements, such as using in-store cameras to measure how customers feel about their “retail experiences” and voice-based software that gives call center operators information about how their service is being perceived on the other side of the line.

        A growing number of companies and research institutions throughout the country are working to make these potential applications a reality. The following paragraphs present an overview of their innovative activities, which could potentially be eligible for R&D tax credits.

University Research

Massachusetts Institute of Technology
        Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory have developed a technology that detects a person’s emotions with wireless signals. The “EQ-Radio” measures changes in breathing and heart rhythms and determines whether someone is excited, happy, angry, or sad. With 87 percent accuracy, the groundbreaking solution shows that emotion-detection technology doesn’t have to rely on audiovisual cues or on-body sensors. Instead, it can use wireless signals that reflect off of a person’s body and back to the measuring device.   

Carnegie Mellon University & Duke University
        Researchers at Carnegie Mellon University’s Human Sensing Laboratory have created IntraFace, advanced software for facial tracking and emotion recognition. The state-of-the-art solution simplifies the problem of facial image analysis, working rapidly, accurately, and with such efficiency that it can run on most smartphones. In December 2015, the software was made available to researchers from other institutions, in an effort to contribute to the development of real-time facial image analysis applications. Early adopters include researchers at Duke University, who have already incorporated IntraFace into a screening tool for autism.  

Stanford University
        Facial recognition software developed at Stanford University promises to help autistic patients socialize. The application, which runs on Google Glass, scans faces in real-time and alerts wearers about their emotions. There are over 100 autistic children currently participating in “autism glass” therapy trials, which could eventually lead to commercialization in a couple of years. In the words of Dennis Wall, director of the Stanford School of Medicine's Wall Lab, “The autism glass program is meant to teach children with autism how to understand what a face is telling them. And we believe that when that happens they will become more socially engaged.”

Arizona State University
        Researchers at ASU’s Motivational Interviewing Laboratory are using emotion-recognition technology to help improve communication between doctors and patients. Through a combination of facial tracking and voice analysis software, they are attempting to give practitioners insight into what makes patients uncomfortable and how to help them be more honest and forthcoming. Next steps include measuring both physiological responses and brain activity.  

Emotion-Recognition Companies

Emotient: In January 2016, Apple acquired Emotient, a San Diego, California-based startup that uses artificial intelligence to detect emotions in real-time. With a massive database of micro expressions, Emotient developed an algorithm that analyzes and interprets facial patterns. Capable of extracting more than 90 thousand data points from a single visual input, the software classifies even the tinniest muscular movements into different emotional categories. Founded in 2012, Emotient has worked with major companies, such as Honda Motor Co. and Procter & Gamble Co., to help analyze consumers’ facial expressions when trying out products or even looking at them in a store. The company’s software, which has also been considered for healthcare applications, could play a strategic role in Apple’s future developments, from its iAd platform, to the Apple Store experience, as well as existing AI products, such as Siri.  

Affectiva: Based in Waltham, Massachusetts, MIT Media Lab spin-off Affectiva has created emotion-recognition software that registers and assesses people’s facial expressions. The groundbreaking solution is made possible through a combination of computer vision and deep learning technology that captures, categorizes, and maps images, classifying them into different emotional categories. Affectiva aims to enable the incorporation of emotional intelligence into different interactive products in areas such as health, education, and robotics. As of May 2016, the company compiled a visual database of over 4.25 million videos from people in 75 countries, which corresponded to more than 50 billion emotion-related data points. The company believes that its system will soon be able to identify even the most complex feelings, such as hope, inspiration, and frustration. Major companies such as Unilever, Kellogg’s, Mars, and CBS have already used Affectiva’s innovative technology. Its software development kit is starting to make its way into a variety of industries, with particular focus on video games. The company has recently raised $14 million in a Series D funding round.

Eyeris: Headquartered in Mountain View, California, Eyeris is the creator of Emovu, a deep learning-based vision technology that enables everyday devices to “understand how we feel, who we are, and how we perceive the environment around us.” Advancing the concept of Ambient Intelligence, Eyeris aims to add emotion intelligence to devices’ contextual awareness, thereby enabling adaptive interfaces that guarantee utmost customization of user experience. The company recently unveiled its Emovu Driver Monitoring System, an in-car technology that analyzes drivers’ facial expressions and classifies them into seven different emotional states. Eyeris believes that the real-time monitoring of drivers’ feelings can significantly enhance traffic safety.  

EyeSee: Operating in the U.S., Belgium, and Serbia, EyeSee has created an online, eye-tracking and facial-coding platform. The company’s innovative, webcam-based technology makes “implicit insights” accessible online and allows its clients to measure the emotions evoked by new ads, products, and websites. EyeSee’s solution performs second-by-second analysis, classifying each webcam frame into one of seven different categories of emotions. Besides providing data on emotions over time, the platform calculates an overall valence score that reflects the predominance of positive or negative feelings.

Kairos: This Miami, Florida-based firm uses three-dimensional data gathered by its face analysis algorithms to measure people’s feelings and interactions with content and products. Aimed at empowering its clients with meaningful metrics, Kairos offers both emotional analysis and facial recognition. In 2015, the company received $300,000 in funding from the Florida Institute and participated in the 500 Startup Distro Miami, a preeminent startup accelerator program.

Sentio Solutions: This Palo Alto, California-based startup has created the Feel Wristband, a wearable device that monitors emotions and stress levels. The company aims to fill the gap left by fitness trackers, which do not tap into emotional metrics. Besides measuring movement, blood volume pressure, and skin temperature, the innovative wristband gathers information on the user’s Electrodermal Response, which is considered a great indicator of emotional state. All of the information is processed by Feel Wristband’s companion app and can be used to help clarify what kind of situation or tasks affect users’ feelings the most. Sentio aims to integrate recommendations and wellness programs to help users keep their emotion on track.

EMOSpeech: Founded in 2012, this Miami, Florida-based company develops enterprise software applications for speech emotion recognition. EMOSpeech has created a speech analytics engine that uses a continuous psychological model to recognize different emotional states in a wide spectrum of acoustic information. The innovative solution has been used by call centers to analyze calls, allowing for real-time feedback to agents and supervisors.

Beyond Verbal: This Israeli company has developed groundbreaking technology for voice-driven emotion analytics. By decoding vocal intonations and their underlying emotions in real-time, Beyond Verbal aims to change the way humans and machines interact. In over 21 years of research, the company has gathered more than 2.5 million voice samples in 40 different languages. In March 2016, the company unveiled an emotion analytics API that enables the integration of emotional intelligence into any voice driven, network-connected solution. It also recently launched the Beyond mHealth Research Platform, which aims to use voice recognition to detect diseases. Beyond Verbal is collaborating with hospitals, research institutes, businesses, and universities in an effort to identify unique biomarkers in voice samples of patients. Researchers claim to have already found promising data on diseases such as ALS and Parkinson’s. According to Beyond Verbal, the ultimate objective is to create a “common API to diagnose health changes through voice data gathered through smart cars, internet of things devices in smart homes, and virtual assistants such as Alexa and Siri”.

Challenges Ahead

        Personal privacy is undoubtedly an issue when it comes to emotion-detection technology. Making sure that information is used responsibly and within reasonable limits is a major challenge, particularly due to the novelty of potential applications, which are largely unregulated. Besides the issue of consent, there is a considerable risk that feelings could be misinterpreted. Using such technology for detecting lies or secretly monitoring people could lead to unfair and biased decisions, particularly if taken as a definitive source of information, rather than an auxiliary tool. Recently developed algorithms are highly accurate, however, when it comes to the level of complexity involved in emotional information, the possibility of mistakes can never be discarded. On the one hand, it is necessary to recognize the limitations of existing technologies and work to perfect them. On the other, it is paramount to establish the necessary regulations that will preserve the right to privacy and prevent abuses.


        Emotions are everywhere, and so is technology. Recent developments in emotion sensing and analytics promise to bring together these two ubiquitous elements of our daily lives, creating new opportunities for interactions between men and machine. Potential applications pervade various industries, ranging from healthcare and security to market research and gaming. Companies investing in the promising field of emotion-recognition technology should take advantage of R&D tax credits available to support their innovative efforts.

Article Citation List



Charles R Goulding Attorney/CPA, is the President of R&D Tax Savers.

Andressa Bonafé is a Tax Analyst with R&D Tax Savers.

Similar Articles
The R&D Tax Credit Aspects of the 21st Century Cures Act
The R&D Tax Credit Aspects of Blockchain for Supply Chains
The R&D Tax Credit Aspects of Physical Security Technology
The R&D Tax Credit Aspects of Mechatronics
The R&D Tax Credit Aspects of Driverless Cars
The R&D Tax Credit Aspects of SaaS Start-Ups
The R&D Tax Credit Aspects of AI in the Insurance Industry
The R&D Tax Credit Aspects of Emerging AV Trends
Enhanced R&D Tax Credits for Specialized Co-Shared Spaces
Ethereum's Impact on Digital Contracting Creates R&D Tax Credit Opportunities
The R&D Tax Credit Aspects of Geofencing
The R&D Tax Credit Aspects of Immunology
The R&D Tax Credit Aspects of Distribution Center Automation
The R&D Tax Credit Aspects of Artificial Intelligence in Fintech
The R&D Tax Credit Aspects of Law Firm Artificial Intelligence
The R&D Tax Credit Aspects of Avionics
The R&D Tax Credit Aspects of Construction Industry IoT
The R&D Tax Aspects of Cancer Treatment
The R&D Tax Aspects of Near Field Communication
R&D Tax Aspects of DNA Identification
R&D Tax Credit Aspects of Cyber Security and Homeland Protection
The R&D Tax Credit Aspects of the Health Cloud
The Internet of Things Creates R&D Tax Credit Opportunity
The Robot Revolution and Tax Policy
Now Every Business is a Software Business
The R&D Tax Credit Opportunities for Mobile Devices
The R&D Tax Aspects of Big Data
R&D Tax Credit Fundamentals
R&D Credits for Companies Combating Superbugs
Los Angeles Tech Boom Creates Large R&D Tax Incentive Opportunities
The R&D Tax Credit Aspects of Alzheimers
The R&D Tax Aspects of Software Development
The R&D Tax Aspects of Diabetes
The R&D Tax Aspects of Brain Mapping
R&D Credit Opportunity for Smart Sensors
Rapid Robot Technology Advancements Supported by R&D Tax Credits
R&D Tax Credit Aspects of Human Body Weight