Psych_ML (AI ASSISTANTS FOR PSYCHOLOGY AND PSYCHIATRY)
Project leader: Sergey Koltsov
Project participants: Anton Surkov
The recent advancements in Large Language Models (LLMs) have opened up the practical possibility of creating AI assistants in the field of psychology and psychiatry based on LangChain technology. However, the development of such assistants is challenging due to the difficulty in obtaining high-quality data for training LLMs and creating a vector base for optimizing prompt queries. The complexity in acquiring data stems from the fact that conversations between patients and psychiatrists are typically not available in the public domain due to ethical reasons. Nevertheless, extensive discussion threads among users exist on social media, which are available for analysis. However, these discussions constitute highly noisy data, hence the need to test and adapt LLMs for the cleansing and preliminary classification of psychological data from the internet.
The project on creating an AI assistant for psychological and psychiatric care based on LLM technologies focuses on using open but highly noisy user discussions from social networks as an alternative to inaccessible therapeutic transcripts; in the new work, the authors explore the potential of four LLMs in zero-shot mode for automatic cleaning and preliminary classification of 64,404 Russian-language messages labeled by seven common disorders (depression, neurosis, paranoia, anxiety, bipolar, OCD, borderline), showing that shallow data filtering provides only a moderate increase, while fine-tuning (both standard and in NLI mode) increases accuracy by more than three times, with the NLI approach achieving maximum accuracy of ≈ 0.64, but working six times slower, which requires hypothesis optimization; at the same time, multilingual models on the source text slightly outperform English-language versions with machine translation. As a result, the first open Russian-language dataset and trained models are presented, designed to simplify annotation and become the basis for creating disorder-specific conversational agents in the field of mental health.
Publications on the project:
Koltcov S, Surkov A, Koltsova O, Ignatenko V. Using large language models for extracting and pre-annotating texts on mental health from noisy data in a low-resource language // PeerJ Computer Science, 2024 10:e2395 DOI
Project participants: Anton Surkov
The recent advancements in Large Language Models (LLMs) have opened up the practical possibility of creating AI assistants in the field of psychology and psychiatry based on LangChain technology. However, the development of such assistants is challenging due to the difficulty in obtaining high-quality data for training LLMs and creating a vector base for optimizing prompt queries. The complexity in acquiring data stems from the fact that conversations between patients and psychiatrists are typically not available in the public domain due to ethical reasons. Nevertheless, extensive discussion threads among users exist on social media, which are available for analysis. However, these discussions constitute highly noisy data, hence the need to test and adapt LLMs for the cleansing and preliminary classification of psychological data from the internet.
The project on creating an AI assistant for psychological and psychiatric care based on LLM technologies focuses on using open but highly noisy user discussions from social networks as an alternative to inaccessible therapeutic transcripts; in the new work, the authors explore the potential of four LLMs in zero-shot mode for automatic cleaning and preliminary classification of 64,404 Russian-language messages labeled by seven common disorders (depression, neurosis, paranoia, anxiety, bipolar, OCD, borderline), showing that shallow data filtering provides only a moderate increase, while fine-tuning (both standard and in NLI mode) increases accuracy by more than three times, with the NLI approach achieving maximum accuracy of ≈ 0.64, but working six times slower, which requires hypothesis optimization; at the same time, multilingual models on the source text slightly outperform English-language versions with machine translation. As a result, the first open Russian-language dataset and trained models are presented, designed to simplify annotation and become the basis for creating disorder-specific conversational agents in the field of mental health.
Publications on the project:
Koltcov S, Surkov A, Koltsova O, Ignatenko V. Using large language models for extracting and pre-annotating texts on mental health from noisy data in a low-resource language // PeerJ Computer Science, 2024 10:e2395 DOI
Have you spotted a typo?
Highlight it, click Ctrl+Enter and send us a message. Thank you for your help!
To be used only for spelling or punctuation mistakes.