Computational Communication Science| Automated Coding of Televised Leader Displays: Detecting Nonverbal Political Behavior With Computer Vision and Deep Learning

Authors

  • Jungseock Joo University of California, Los Angeles
  • Erik P. Bucy Texas Tech University
  • Claudia Seidel University of California, Los Angeles

Keywords:

computational communication science, computer vision, deep learning, nonverbal behavior, facial expressions, gestures, 2016 presidential debates

Abstract

For decades, nonverbal communication scholars have employed manual coding as the primary research methodology for systematic content analysis of nonverbal behaviors such as facial expressions and gestures. Manual coding of visual data, however, is expensive and time consuming and therefore not suitable for studies relying on large-scale data. This article introduces a novel computational methodology that can automatically analyze visual content of human communication from visual data. Based on computer vision techniques, the method allows to automatic detection and classification of diverse facial expressions and communicative gestures that have been manually coded in traditional work. To demonstrate the new method, we develop a computational pipeline to classify fine-grained facial expressions and physical gestures and apply our technique to the first 2016 U.S. presidential debates between Donald Trump and Hillary Clinton. The results confirm that computational methods can replicate human coding with a high degree of accuracy for bodily movements and facial expressions, as well as nonverbal tics and signature displays unique to individual candidates. Automated coding should soon facilitate rapid progress in quantitative visual communication research by dramatically scaling up existing manual studies. 

Author Biographies

Jungseock Joo, University of California, Los Angeles

Assistant ProfessorUCLA

Erik P. Bucy, Texas Tech University

ProfessorTexas Tech University,USA 

Claudia Seidel, University of California, Los Angeles

Grad student

Downloads

Published

2019-09-08

Issue

Vol. 13 (2019)

Section

Special Sections

License

The International Journal of Communication is an academic journal. As such, it is dedicated to the open exchange of information. For this reason, IJoC is freely available to individuals and institutions. Copies of this journal or articles in this journal may be distributed for research or educational purposes free of charge and without permission. However, commercial use of the IJoC website or the articles contained herein is expressly prohibited without the written consent of the editor. Authors who publish in The International Journal of Communication will release their articles under the Creative Commons Attribution Non-Commercial No Derivatives (by-nc-nd) license. This license allows anyone to copy and distribute the article for non-commercial purposes provided that appropriate attribution is given. For details of the rights authors grants users of their work, see the "human-readable summary" of the license, with a link to the full license. (Note that "you" refers to a user, not an author, in the summary.) This journal utilizes the LOCKSSsystem to create a distributed archiving system among participating libraries and permits those libraries to create permanent archives of the journal for purposes of preservation and restoration. The publisher perpetually authorizes participants in the LOCKSS system to archive and restore our publication through the LOCKSS System for the benefit of all LOCKSS System participants. Specifically participating libraries may:
  • Collect and preserve currently accessible materials;
  • Use material consistent with original license terms;
  • Provide copies to other LOCKSS appliances for purposes of audit and repair.
 

Fair Use

The U.S. Copyright Act of 1976 specifies, in Section 107, the terms of the Fair Use exception: Notwithstanding the provisions of sections 106 and 106A, the fair use of a copyrighted work, including such use by reproduction in copies or phonorecords or by any other means specified by that section, for purposes such as criticism, comment, news reporting, teaching (including multiple copies for classroom use), scholarship, or research, is not an infringement of copyright. In determining whether the use made of a work in any particular case is a fair use the factors to be considered shall include:
  1. the purpose and character of the use, including whether such use is of a commercial nature or is for nonprofit educational purposes;
  2. the nature of the copyrighted work;
  3. the amount and substantiality of the portion used in relation to the copyrighted work as a whole; &
  4. the effect of the use upon the potential market for or value of the copyrighted work.

The fact that a work is unpublished shall not itself bar a finding of fair use if such finding is made upon consideration of all the above factors. In accord with these provisions, the International Journal of Communication believes in the vigorous assertion and defense of Fair Use by scholars engaged in academic research, teaching and non-commercial publishing. Thus, we view the inclusion of “quotations” from existing print, visual, audio and audio-visual texts to be appropriate examples of Fair Use, as are reproductions of visual images for the purpose of scholarly analysis. We encourage authors to obtain appropriate permissions to use materials originally produced by others, but do not require such permissions as long as the usage of such materials falls within the boundaries of Fair Use.

The International Journal of Communication encourages authors to employ fair use in their scholarly publishing wherever appropriate. Fair use is the right to use unlicensed copyrighted material (whether it is text, images, audio-visual, or other) in your own work, in some circumstances. We consult the Code of Best Practices in Fair Use for Scholarly Research in Communication, created by the International Communication Association and endorsed by the National Communication Association, and you should too. If you have any questions about whether fair use applies to your uses of copyrighted material (whether it is text, images, audio-visual, or other) in your scholarship, simply include your rationale, grounded in the Best Practices, as a supplementary document with your submission.