Abstract
Seel RT, Steyerberg EW, Malec JF, Sherer M, Macciocchi SN. Developing and evaluating
prediction models in rehabilitation populations.
This article presents a 3-part framework for developing and evaluating prediction
models in rehabilitation populations. First, a process for developing and refining
prognostic research questions and the scientific approach to prediction models is
presented. Primary components of the scientific approach include the study design
and sampling of patients, outcome measurement, selecting predictor variable(s), minimizing
methodologic sources of bias, assuring a sufficient sample size for statistical power,
and selecting an appropriate statistical model. Examples focus on prediction modeling
using samples of rehabilitation patients. Second, a brief overview for statistically
building and validating multivariable prediction models is provided, which includes
the following 7 steps: data inspection, coding of predictors, model specification,
model estimation, model performance, model validation, and model presentation. Third,
we propose a set of primary considerations for evaluating prediction model studies
using specific quality indicators as criteria to help stakeholders evaluate the quality
of a prediction model study. Lastly, we offer perspectives on the future development
and use of rehabilitation prediction models.
Key Words
List of Abbreviations:
AAN (American Academy of Neurology), CDE (common data element), EPV (events per variable), IRT (item response theory), LASSO (least absolute shrinkage and selection operator), NINDS (National Institute on Neurological Disorders and Stroke), RCT (randomized controlled trial), ROC (receiver operating curve), TBI (traumatic brain injury)To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Archives of Physical Medicine and RehabilitationAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Evans R.W. Baskin D.S. Yatsu F.M. Prognosis of neurological disorders. 2nd ed. Oxford Univ Pr, New York2000
- Modeling 5-year functional outcome in a major traumatic injury survivor cohort.Disability Rehabil. 2006; 28: 629-636
- Prognosis research: why is Dr. Lydgate still waiting?.J Clin Epidemiol. 2006; 61: 1229-1238
- Prognosis after severe TBI: a practical, evidence-based approach.in: Zasler N.D. Katz D.I. Zafonte R.D. Brain injury medicine: principles and practice. Demos, New York2007: 169-200
- Evaluating research studies that address prognosis for patients receiving physical therapy care: a clinical update.Phys Ther. 2007; 87: 1527-1535
- A predictive screening index for posttraumatic stress disorder and depression following traumatic injury.J Consult Clin Psychol. 2008; 76: 923-932
- Towards a common language for functioning, disability and health: ICF—The International Classification of Functioning, Disability and Health.World Health Organization, Geneva2002
- Clinical prediction models: a practical approach to development, validation, and updating.Springer, New York2009
- Clinical versus actuarial judgment.Science. 1989; 243: 1668-1674
- Clinical versus statistical prediction: a theoretical analysis and review of the evidence.Univ of Minnesota Pr, Minneapolis1954
- Prediction of the need for intensive care in patients who come to the emergency department with acute chest pain.N Engl J Med. 1996; 334: 1498-1504
- Psychological science can improve diagnostic decisions.Psychol Sci Public Interest. 2000; 1: 1-26
- 50 years of successful predictive model building should be enough: lessons for philosophy of science.Philos Sci. 2002; 69: S197-S208
- Using simple heuristics to target macrolide prescription in children with community-acquired pneumonia.Arch Pediatr. 2002; 156: 1005-1008
- IMPACT database of traumatic brain injury: design and description.J Neurotrauma. 2007; 24: 239-250
- Prognosis and clinical trial design in traumatic brain injury: the IMPACT study.J Neurotrauma. 2007; 24: 232-238
- The influence of enrollment criteria on recruitment and outcome distribution in traumatic brain injury studies: results from the impact study.J Neurotrauma. 2009; 26: 1069-1075
- Clinical practice guideline process manual.2011 ed. American Academy of Neurology, St. Paul2011
- Methodological evaluation of observational research-observational studies of risk factors of chronic diseases.Agency for Healthcare Research and Quality, Rockville2011 (Publication No. 11-EHC008-EF)
- GRADE guidelines 2.J Clin Epidemiol. 2011; 64: 395-400
- GRADE guidelines 4.J Clin Epidemiol. 2011; 64: 407-415
- Understanding prognosis to improve rehabilitation: the example of lateral elbow pain.Arch Phys Med Rehabil. 1996; 77: 586-593
- Outcome assessment in traumatic brain injury clinical trials and prognostic studies.J Head Trauma Rehabil. 2010; 25: 92-98
- Reporting and methods in clinical prediction research: a systematic review.PLoS Med. 2012; 9 (2012 May 22. [Epub ahead of print]): e1001221
- Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis.Springer, New York2001
- From Meehl (1954) to fast and frugal heuristics (and back): new insights into how to bridge the clinical-actuarial divide.Theory and Psychology. 2008; 18: 443-464
- Clinical practice guidelines we can trust: standards for developing trustworthy clinical practice guidelines (CPGs).Institute of Medicine of the National Academies, Washington (DC)2011
- Applied multiple regression/correlation analysis for the behavioral sciences.3rd ed. Routledge, New York2003
- Experimental and quasi-experimental designs for generalized causal inference.Wadsworth, Belfast2002
- Dichotomizing continuous predictors in multiple regression: a bad idea.Stat Med. 2006; 25: 127-141
- The research question.in: Foundations of clinical research: applications to practic. 2nd ed. Prentice Hall Health, Upper Saddle River2000: 113-136
- Evaluation of the quality of prognosis studies and systematic reviews.Ann Intern Med. 2006; 144: 427-437
- Scale development: theory and applications.2nd ed. Sage Publications, Thousand Oaks2003
- Health measurement scales: a practical guide to their development and use.in: 3rd ed. Oxford Univ Pr, Oxford2003
- Introduction to classical and modern test theory.Harcourt Brace, Orlando1986
- The use of raw scores from ordinal scales: time to end malpractice?.J Rehabil Med. 2012; 44: 97-98
- Constructing measures: an item response modeling approach.Erlbaum, Mahwah2005
- Applying the Rasch model: fundamental measurement in the human sciences.2nd ed. Lawrence Erlbaum, Mahwah2007
- Improving measurement methods in rehabilitation: core concepts and recommendations for scale development.Arch Phys Med Rehabil. 2012; 93: S154-S163
- Common data elements for research on traumatic brain injury and psychological health: current status and future development.Arch Phys Med Rehabil. 2010; 91: 1692-1696
- Incorporation of the International Spinal Cord Injury Data Set elements into the National Institute of Neurological Disorders and Stroke Common Data Elements.Spinal Cord. 2011; 49: 60-64
- What you see may not be what you get: a brief, non-technical introduction to overfitting in regression-type models.Psychosom Med. 2004; 66: 411-421
- Prognostic modeling with logistic regression analysis: a comparison of selection and estimation methods in small data sets.Stat Med. 2000; 19: 1059-1079
- Simplifying a prognostic model: a simulation study based on clinical data.Stat Med. 2002; 21: 3803-3822
- Determinants of the best significance level in forward stepwise logistic regression.Comm Stat Sim Comp. 1997; 26: 559-575
- Applied logistic regression.2nd ed. Wiley, New York2000
- What do we mean by validating a prognostic model?.Stat Med. 2000; 19: 453-473
- Users' guides to the medical literature.JAMA. 1994; 272: 234-237
- Assessing the generalizability of prognostic information.Ann Intern Med. 1999; 130: 515-524
- Development and validation of a prediction model with missing predictor data: a practical approach.J Clin Epidemiol. 2011; 63: 205-214
- Statistical analysis with missing data.in: 2nd ed. Wiley, Hoboken2002
- Statistical approaches to the univariate prognostic analysis of the IMPACT database on traumatic brain injury.J Neurotrauma. 2007; 24: 251-258
- Improving the robustness of fractional polynomial models by preliminary covariate transformation: a pragmatic approach.Comput Stat Data Anal. 2007; 51: 4240-4253
- Regression modeling strategies for improved prognostic prediction.Stat Med. 1984; 3: 143-152
- Regression methods in biostatistics: linear, logistic, survival, and repeated measures models.Springer, New York2005
- Predictive value of statistical models.Stat Med. 1990; 9: 1303-1325
- Regression and shrinkage via the Lasso.J R Stat Soc Series B Stat Methodol. 1996; 58: 267-288
- Assessing the performance of prediction models: a framework for traditional and novel measures.Epidemiology. 2010; 21: 128-138
- Internal validation of predictive models: efficiency of some procedures for logistic regression analysis.J Clin Epidemiol. 2001; 54: 774-781
- Internal and external validation of predictive models: a simulation study of bias and precision in small samples.J Clin Epidemiol. 2003; 56: 441-447
- Prognostic modeling with logistic regression analysis: in search of a sensible strategy in small data sets.Med Decis Making. 2001; 21: 45-56
- Counseling men with prostate cancer: a nomogram for predicting the presence of small, moderately differentiated, confined tumors.J Urol. 2003; 170: 1792-1797
Article info
Footnotes
Supported in part by the National Institute on Disability and Rehabilitation Research, U.S. Department of Education (grant no. H133G070171).
No commercial party having a direct financial interest in the results of the research supporting this article has or will confer a benefit on the authors or on any organization with which the authors are associated.
Reprints are not available from the author
Identification
Copyright
© 2012 American Congress of Rehabilitation Medicine. Published by Elsevier Inc. All rights reserved.
