GailBot: An Automatic Transcription System for Conversation Analysis

Glossary:
Resources
Overview
Availability and Usage
Citation
Liability Notice
Publication Resources

Resources

News:
GailBot paper published in Dialogue and Discourse. 
News Coverage by Tufts University.
GailBot AMLAP 2020 Presentation and Abstract

Documentation:
Gailbot User Manual

Forms
GailBot App. Request Form

Overview

Take a second to listen to the following conversation:

Most speech-to-text (STT) systems would produce a transcript that looks something like this:

Example Speech to Text transcription

This transcript is useful, but falls short of truly capturing the conversation. There are a few obvious omissions. Perhaps most importantly, there is a 4.1 second silence between lines 5 and 6. This gap does some heavy lifting: it shows that Speaker 2 expected Speaker 1 to understand their joke. In addition, STT systems typically ignore laughter. On line 2, Speaker 2 laughs, an early cue that they are joking. On line 7, Speaker 1 laughs, indicating that they finally understand the joke. To really understand the sequence, what we really need is a transcript more like the one below, which marks paralanguage – everything that comes along with the words.

Example Speech to Text transcription with added markers for paralinguistic features

The system to transcribe paralinguistic features – Jeffersonian transcription – was created by Gail Jefferson. Jeffersonian transcription is slow going. On one hand, the process of re-listening to the audio to add more and more details helps the researcher understand the mechanisms underlying the interaction. On the other hand, the laborious process limits the amount of data researchers can analyze. For example, it would be close to impossible to generate enough Jeffersonian transcripts to train a deep learning language model.

Enter: GailBot. We designed GailBot to create first-pass transcriptions of some paralinguistic features (speech rate, silences, overlaps and laughter). It interfaces with existing STT algorithms, and then applies post-processing modules that insert Jeffersonian symbols. All of this is useful, but GailBot’s most valuable characteristic is that it allows researchers to create, insert, and adjust plugins, which are a standardized interface for applying customized plugins to the GailBot pipeline.

Availability and Usage

GailBot is available through its official website (https://www.gailbot.ai)

Acknowledgements

GailBot has been made possible by a small but dedicated team of researchers and research assistants in the Human Interaction Lab at Tufts University.

Faculty, post-docs, and graduate student contributors:

Muhammad Umair

Umair is a PhD student in Computer Science: Human Robot Interaction in the School of Engineering at Tufts. Umair graduated with a bachelors in Computer Science from Tufts in 2021, and was involved in various research projects throughout his undergrad.

Umair is a PhD student in Computer Science at the Tufts Human Interaction Lab, and is the lead developer for GailBot. He is interested in improving naturalistic turn-taking in spoken dialogue systems. His projects are geared towards  collecting natural interaction data using both automated and manual methods and developing data-driven AI solutions to improve turn-taking, based on inspiration from human-human interaction. Currently, he is developing models for detecting Transition Relevance Places (TRPs) to improve the timing of turn-taking in dialogue systems and exploring the use of Large Language Models (e.g, ChatGPT) for use in Spoken interaction.

Dr. Julia Mertens

Julia is a cognitive scientist whose previous research includes the terms, biophysiology, facial expression ambiguity, autism spectrum disorders, first impressions, and eye gaze percentage. Her current research focuses on the mechanisms, causes and outcomes of miscommunication. She graduated from the University of Connecticut with a Bachelors of Science in Cognitive Science and Psychology in 2015, and worked as Research Assistant at the Facial Affective and Communicative Expressions Laboratory at Emerson College from 2015 to 2017.

Julia is currently working as a Senior Scientist at Boston Fusion in Lexington, MA.

Portait of Julia Mertens by JP De Ruiter

Dr. Saul Albert

Potrait of Saul Albert by J. P. De Ruiter

Saul initiated the GailBot Project with Umair and JP while he was a postdoc at Tufts, and he is working on finding functional compromise between the demands of CA’s Jeffersonian transcription conventions and the constraints of GailBot’s automation and digital transcription formats.

Currently, Saul is an Assistant Professor of Social Science (Social Psychology) at Loughborough University’s Communication & Media division. He is working on a range of applied CA/AI projects including how disabled people and care assistants use and adapt virtual assistant technologies for ‘smart homecare’ use cases. He is also working on a book entitled ‘Conversation Analysis for Conversation Design’ with Cathy Pearl and Elizabeth Stokoe due out with Routledge in 2024/25.

Professor J.P. De Ruiter

Prof. De Ruiter is the director of the Tufts Human Interaction Lab and is the GailBot project PI.

Citation

Please cite GailBot using the following bibtex:

@article{umair2022gailbot,
  title={GailBot: An automatic transcription system for Conversation Analysis},
  author={Umair, Muhammad and Mertens, Julia Beret and Albert, Saul and de Ruiter, Jan P},
  journal={Dialogue \& Discourse},
  volume={13},
  number={1},
  pages={63--95},
  year={2022}
} 

Liability Notice

Gailbot is a tool to be used to generate specialized transcripts. However, it is not responsible for output quality. Generated transcripts are meant to be first drafts that can be manually improved. They are not meant to replace manual transcription.

GailBot may use external Speech-to-Text systems or third-party services. The development team is not responsible for any transactions between users and these services. Additionally, the development team does not guarantee the accuracy or correctness of any plugin. Plugins have been developed in good faith and we hope that they are accurate. However, users should always verify results.

By using GailBot, users agree to cite Gailbot and the Tufts Human Interaction Lab in any publications or results as a direct or indirect result of using Gailbot.

Publication Resources

This section contains resources that were included as part of various publications.

Media Files