|
Abstract
|
AbstractObjectives
(1) Identify validation design and accuracy assessment standards for medical prognostic models applicable to evaluation of child abuse/neglect (CA/N) risk assessment models. (2) Assess the accuracy of the California Family Risk Assessment (CFRA) in predicting CA/N using the foregoing standards. (3) Compare the prediction accuracy of the CFRA with the prediction accuracy of coronary heart disease (CHD) prediction models. Questions addressed
(1) What validation design and accuracy assessment standards are used to evaluate medical prognostic models? (2) What is the evidence for the accuracy of the CFRA using those standards? (3) How does the accuracy of the CFRA in predicting CA/N compare with the accuracy of CHD prediction models, which are a reasonable exemplar for the CA/N prediction effort?
Method
An external validation sample of 236 California reports of CA/N from San Luis Obispo and Sutter counties, and a larger temporal validation sample of 6307 California reports from Orange, Los Angeles, and Humboldt counties were investigated and assessed with the CFRA by line child welfare staff and were followed prospectively statewide for two years to discover reported, substantiated CA/N in any California county. CFRA accuracy in predicting substantiated CA/N was assessed by calibration and discrimination. Calibration was measured as the ratio of predicted to observed cases of CA/N seen during follow-up, with a ratio of 1.0 registering perfect calibration. Discrimination was measured by the area under the receiver operating characteristic (ROC) curve (AUC), with values from.60 to.85 found typical for medical prognostic models. CHD prediction literature was reviewed to acquire values of these accuracy measures for CHD prediction models. CFRA CA/N prediction accuracy and CHD prediction accuracy were then compared.
RESULTS
Findings from external and temporal validation samples support the accuracy of CFRA prediction of CA/N. CFRA accuracy in predicting CA/N compared well with CHD prediction accuracy: (1) in the external validation sample, 43.42 CA/N cases were predicted during follow-up and 47 were observed, with consequent 7.6% deviation from perfect calibration. (2) In the temporal validation sample 857.49 CA/N cases were predicted and 801 were observed, with 7.1% deviation from perfect calibration. (3) The best performing of 20 Framingham CHD prediction models identified by systematic literature review predicted 222 CHD cases and 206 were observed, with 7.8% deviation from perfect calibration. (3) The CFRA external and temporal validation sample AUCs were.74 and.64, respectively. (4) For 26 CHD prediction cohorts found by literature review, the AUC mean and median values were.72 and.71, respectively, with a range from.60 to.84.
CONCLUSIONS/practice implications
(1) External and temporal validation results support the accuracy of the CFRA. (2) CFRA CA/N prediction accuracy parity with that for CHD prediction is encouraging, suggesting that wide use of the CFRA, properly implemented, could improve risk assessment accuracy in child protection. (3) Findings underline the importance of ensuring that no risk assessment model or method, including actuarial and consensus models and clinical judgment, is used in the field unless it has passed a test of external, or at least temporal validation. |