Dive Brief:

• Algorithms are as effective as trained human evaluators at detecting red-flag language in text messages from people with serious mental illness, according to a randomized controlled trial.
• The study, details of which were published in the journal Psychiatric Services, tasked humans and natural language processing (NLP) methods with analyzing more than 7,000 text messages from 39 people.
• With one model performing comparably to human raters, the study suggests that NLP methods have the potential to enable automated tools for clinical support of people with mental illness.

Dive Insight:

Clinically trained human raters can spot warning signs such as jumping to conclusions, catastrophizing and overgeneralizing in text messages from people with serious mental illness. The written clues offer a way to pick up on changes to the mental health of patients but busy clinicians, who are trained to pick up on auditory and visual cues, can miss the red flags in text messages. 

Technology may be able to alert clinicians to warning signs. To test the idea, researchers at the University of Washington School of Medicine developed NLP methods to detect and classify cognitive distortions in text messages and compared the technology to trained human raters in a 12-week clinical trial.

Clinical annotators labeled messages for common “cognitive distortions,” such as jumping to conclusions, and the NLP classification methods were applied to the same messages. One NLP approach, described as “a tuned model that used bidirectional encoder representations from transformers,” outperformed the other algorithms and delivered comparable results to the humans.   

Justin Tauscher, the paper’s lead author and an acting assistant professor at UW Medicine, discussed the potential implications of the results in a statement to publicize the research. 

“Being able to have systems that can help support clinical decision-making I think is hugely relevant and potentially impactful for those out in the field who sometimes lack access to training, sometimes lack access to supervision or sometimes also are just tired, overworked and burned out and have a hard time staying present in all the interactions they have,” Tauscher said.