How Can We Measure Quality of Care for Children with ADHD Using Natural Language Processing?

What you need to know

• A new method using natural language processing (NLP) can accurately assess whether doctors are following guidelines for treating children with ADHD.
• The method analyzes doctors’ notes in electronic health records to check if behavioral therapy is recommended, which is the first-line treatment for young children with ADHD.
• This approach could provide a more comprehensive and efficient way to measure quality of care for ADHD compared to current methods.

Understanding ADHD and its Treatment Guidelines

Attention-Deficit/Hyperactivity Disorder (ADHD) is a common neurobehavioral condition that affects 8-10% of children in the United States. It’s typically diagnosed in preschool or elementary school years and can significantly impact a child’s academic performance if not properly managed.

The American Academy of Pediatrics (AAP) has established evidence-based guidelines for treating ADHD in children. For preschoolers aged 4-5 years, these guidelines recommend parent training in behavior management (PTBM) as the first-line treatment. This recommendation is based on stronger evidence for PTBM compared to medications in this age group. For school-aged children (6-11 years), the guidelines suggest combining PTBM with medications, as this combined approach has been shown to be more effective than either treatment alone.

Current Challenges in Measuring ADHD Care Quality

Currently, assessing the quality of ADHD care faces several challenges:

1. Limited scope of existing measures: The main quality measure used nationally, the Healthcare Effectiveness Data and Information Set (HEDIS), only captures the timing of follow-up care for children prescribed ADHD medications. This measure addresses only a narrow aspect of care and doesn’t apply to all children with ADHD.

2. Labor-intensive manual reviews: To supplement the HEDIS measure, many healthcare organizations conduct manual reviews of patient charts. However, this process is time-consuming, expensive, and often limited to a small sample of patients.

3. Difficulty accessing key information: Much of the important information about ADHD care, such as treatment recommendations, is recorded as free text in doctors’ notes. This makes it challenging to analyze systematically using traditional methods.

These limitations mean that current quality measures often fail to capture the full picture of ADHD care, particularly whether doctors are following evidence-based guidelines in their treatment recommendations.

A New Approach Using Natural Language Processing

The researchers in this study developed a novel method to address these challenges using natural language processing (NLP), a type of artificial intelligence that can analyze and understand human language. Here’s how their approach works:

1. Data extraction: The researchers extracted both structured data (like diagnosis codes) and unstructured data (doctors’ notes) from the electronic health records of children aged 4-6 years who had been diagnosed with ADHD.

2. Manual annotation: Two pediatricians reviewed and annotated a set of clinical notes, marking whether each note included a recommendation for parent training in behavior management (PTBM).

3. NLP model development: Using these annotated notes as a training set, the researchers developed and tested several NLP models. These models were designed to automatically classify whether a clinical note included a PTBM recommendation.

4. Model validation: The best-performing model was then tested on a new set of clinical notes to validate its performance.

Key Findings

The study produced several important findings:

1. Accurate classification: The best-performing NLP model (called BioClinicalBERT) was able to accurately identify whether a clinical note included a PTBM recommendation. On the test set of notes, it achieved an F1 score of 0.78, which indicates a good balance between precision (correctly identifying positive cases) and recall (finding all positive cases).

2. Low rate of PTBM recommendations: Based on the manual review, only 30.5% of children received a PTBM recommendation at their first ADHD visit. This suggests that many children may not be receiving the recommended first-line treatment.

3. Potential disparities: The study found some differences in PTBM recommendations based on race/ethnicity and insurance type. For example, 33.3% of non-Hispanic white patients received PTBM recommendations, compared to only 16.7% of non-Hispanic black patients.

4. Generalizability: The NLP model performed well not just on the first ADHD visit notes, but also on notes from follow-up visits and well-child checks. This suggests that the approach could be used to monitor ADHD care over time.

Implications and Potential Applications

This new approach to measuring quality of ADHD care has several potential benefits:

1. Comprehensive assessment: Unlike current measures that focus on narrow aspects of care, this method can assess whether doctors are following key guideline recommendations, such as recommending behavioral therapy.

2. Efficiency: Once developed, the NLP model can analyze large numbers of clinical notes much more quickly and cost-effectively than manual review.

3. Timely feedback: The automated nature of this approach could allow for more frequent assessments of care quality, potentially providing doctors and healthcare organizations with timely feedback to improve their practices.

4. Identifying disparities: By analyzing care for all patients, this method could help identify and address disparities in ADHD care based on factors like race/ethnicity or insurance type.

5. Adaptability: While this study focused on ADHD, similar approaches could potentially be developed to assess quality of care for other conditions, especially those where important aspects of care are often documented in free text notes.

Conclusions

• Natural language processing can be used to accurately assess whether doctors are following guidelines for ADHD treatment, particularly in recommending behavioral therapy.
• This approach offers a more comprehensive and efficient method for measuring quality of ADHD care compared to current practices.
• By providing more timely and detailed information about care quality, this method could help improve ADHD care and potentially reduce disparities in treatment.

While further research is needed to refine and validate this approach, it represents a promising step forward in using advanced technologies to improve healthcare quality measurement and, ultimately, patient care.