An AI-based Issue Analyzing Framework for Clinical QA Workflow

Document Type

Conference Proceeding

Publication Date

10-1-2020

Publication Title

International Journal of Radiation Oncology Biology Physics

Abstract

Background: Clinical incident reporting, tracking, and risk analysis are critical parts in the quality assurance (QA) workflow. Traditional incident reporting and learning systems rely heavily on the user input to identify, report, investigate, categorize, and respond to incidents. Text information used in clinical reports has unique terminology and requires special domain knowledge to be processed correctly.

Objectives: To develop a workflow utilizing recurrent neural networks (RNN) and a word-to-vector, natural-language processing-(NLP)-based model with oncology domain knowledge, to automatically determine categories and severity level of incidents and identify risks in the clinical workflow.

Methods: A total of 3210 existing incident tickets from 16 categories entered by clinical staff over 7 years were used for development of the classification model. Ninety percent of the events were used for training and 10% for validation of the RNN-based algorithm. A subset of 355 tickets with severity level using a 0-10 scale were labeled by physicists for severity estimator model training and validation. Text from the training tickets were preprocessed with lower-casing, punctuation removal, and tokenization, and were fed into an embedding layer. In order to apply oncology domain knowledge to the embedding layer, we trained a word-to-vector model on a total of 286 million words from biomedical research articles obtained from OpenI with keyword “Oncology.” Outputs of the forward and backward RNNs were then fed into a softmax output layer for multi-label classification and a linear output layer for regression analysis of the severity level. The AI model were deployed as REST API services which served as an engine to analyze incidents entered through a web application. Immediate feedback was provided through a web interface.

Results: After 20 epochs of training, the RNN model reached accuracy of 89.5%, 85.0%, 75.2% for classification of the top three, two and one categories, respectively. root-mean-square of the severity level prediction reached 0.79 after 74 epochs. A 100-dimensional word-to-vector model was trained using corpus from biomedical articles. T-distributed stochastic neighbor embedding (t-SNE) shows that using word-to-vector model trained from biomedical articles, clinical terms are projected to closer locations in the vector space.

Conclusions: A RNN-based incident learning framework for automatic multi-label classification and severity level estimation has been developed. The trained word-to-vector model addressed the word-level label space ambiguity by identifying highly polysemous words that are unique in oncology diagnosis and treatment terminology. Performance of NLP tasks will provide more accurate prediction by using the model trained on quantity incorporating clinical context. Accuracy of the AI-based algorithm is likely to improve with a larger number of events, and more refined categorization of events used for training.

Volume

108

Issue

2

First Page

E60

This document is currently not available here.

Share

COinS