How Do Survey Data and Health Records Compare for Estimating Child Mental Health Issues?

What you need to know

• Survey data generally estimated higher rates of child mental health issues compared to health records
• Agreement between the two data sources on individual cases was low to moderate
• Conclusions about mental health patterns across age, sex and geography groups often differed between the two data sources

Comparing Two Ways to Measure Child Mental Health

Understanding how many children and youth experience mental health issues is crucial for developing effective policies and services. But measuring mental health in the population can be challenging. Two common approaches are conducting surveys and analyzing health records. This study set out to compare these two methods for estimating child and youth mental health issues and service use in Ontario, Canada.

The researchers looked at data on over 5,500 children and youth aged 4-17. For the survey data, they used the 2014 Ontario Child Health Study, which included diagnostic interviews with parents about their children’s mental health. For the health records data, they examined provincial health insurance records related to mental health diagnoses and services.

By comparing these two data sources, the study aimed to answer some key questions:

1. Do the survey and health records give similar estimates of how many kids have mental health issues and use services?
2. How well do the two sources agree on which specific children have mental health issues?
3. Do both sources show similar patterns in mental health across different groups (e.g. age, sex, urban/rural)?

Survey Data Showed Higher Rates Overall

For most mental health issues examined, the survey data indicated higher rates compared to the health records. For example:

• The survey estimated 16.4% of children/youth had either an emotional disorder (like anxiety or depression) or ADHD. Health records showed only 5.6%.

• For ADHD specifically, the survey estimate (7.6%) was over 11 times higher than health records (0.7%).

• The survey found 13.7% had used mental health services in the past 6 months, versus 7.6% in health records.

The one exception was mood disorders like depression, where health records showed a slightly higher rate (3.6%) than the survey (2.9%).

There are a few potential reasons for these differences:

1. Surveys can capture children experiencing symptoms who haven’t sought medical care yet. Health records only show those who have visited a doctor.

2. The survey used structured diagnostic interviews, while health records rely on doctors’ clinical judgments and diagnostic codes.

3. Health records may not capture all relevant service use, like school counseling.

4. There could be inaccuracies in how mental health issues are recorded in health databases.

Low Agreement on Individual Cases

Beyond the overall rates, the researchers also looked at how well the two data sources agreed on which specific children had mental health issues. They found the agreement was generally low to moderate:

• For most issues, the sensitivity (probability of being identified in both sources) was under 30%.

• Specificity (probability of not being identified in either source) was high at 97-100%.

• Kappa statistics, which measure agreement accounting for chance, ranged from 0.13 (slight agreement) for ADHD to 0.46 (moderate) for service use.

This suggests the two methods often aren’t identifying the same individual children as having mental health issues, even if the overall rates are somewhat similar.

Inconsistent Patterns Across Groups

The researchers also examined whether the survey and health records showed similar patterns in mental health issues across different groups. They looked at:

• Age (children 4-11 vs. youth 12-17)
• Sex (male vs. female)
• Geography (large urban, small/medium urban, rural)

In many cases, the two data sources disagreed on whether there were significant differences between groups:

• The survey found higher rates of mood disorders and ADHD in youth compared to children. Health records didn’t show this age difference.

• Health records showed higher rates of emotional disorders in females, while the survey didn’t find a significant sex difference.

• The survey found some urban/rural differences in emotional disorders that weren’t reflected in health records.

The only consistent finding across both sources was higher rates of ADHD and service use for males compared to females.

These inconsistencies make it difficult to draw firm conclusions about how mental health issues vary across population groups based on just one data source.

Implications for Research and Policy

This study highlights some important considerations when interpreting mental health statistics:

1. The data source matters. Survey data and health records can give quite different pictures of child mental health in a population.

2. Neither source is necessarily “right” or “wrong” - they may be capturing different aspects of mental health.

3. Caution is needed when using a single data source to examine differences across groups or geographic areas.

4. Linking and comparing multiple data sources, as done in this study, can provide more comprehensive insights.

For researchers and policymakers using mental health data, the authors recommend:

• Focusing on broader categories of mental health issues rather than specific diagnoses, as agreement tends to be better.

• Using data linkage where possible to compare multiple sources.

• Clearly acknowledging the limitations of health records data, which only capture those who have sought medical care.

• Being cautious about drawing conclusions on group differences from a single data source.

Conclusions

• Survey data and health records can give quite different estimates of child mental health issues and service use.
• Agreement between the two sources on which specific children have mental health issues is generally low.
• Patterns of mental health across age, sex and geography groups often differ between surveys and health records.
• Using multiple data sources and acknowledging their limitations is important for accurate mental health research and policymaking.

This study provides valuable insights into the strengths and limitations of different approaches to measuring child mental health. As research in this area continues, methods that combine multiple data sources may help provide a more complete picture of mental health needs in the population.