Advertisement

Expert Panel Survey to Update the American Congress of Rehabilitation Medicine Definition of Mild Traumatic Brain Injury

Open AccessPublished:October 06, 2020DOI:https://doi.org/10.1016/j.apmr.2020.08.022

      Abstract

      Objective

      As part of an initiative led by the Brain Injury Special Interest Group Mild Traumatic Brain Injury (TBI) Task Force of the American Congress of Rehabilitation Medicine (ACRM) to update the 1993 ACRM definition of mild TBI, the present study aimed to characterize current expert opinion on diagnostic considerations.

      Design

      Cross-sectional web-based survey.

      Setting

      Not applicable.

      Participants

      An international, interdisciplinary group of clinician-scientists (N=31) with expertise in mild TBI completed the survey by invitation between May and July 2019 (100% completion rate).

      Interventions

      Not applicable.

      Main Outcome Measures

      Ratings of agreement with statements related to the diagnosis of mild TBI and ratings of the importance of various clinical signs, symptoms, test findings, and contextual factors for increasing the likelihood that the individual sustained a mild TBI, on a scale ranging from 1 (“not at all important”) to 10 (“extremely important”).

      Results

      Men (n=25; 81%) and Americans (n=21; 68%) were over-represented in the sample. The survey revealed areas of expert agreement (eg, acute symptoms are diagnostically useful) and disagreement (eg, whether mild TBI with abnormal structural neuroimaging should be considered the same diagnostic entity as “concussion”). Observable signs were generally rated as more diagnostically important than subjective symptoms (Wilcoxon signed ranks test, Z=3.77; P<.001; r=0.68). Diagnostic importance ratings for individual symptoms varied widely, with some common postconcussion symptoms (eg, fatigue) rated as unhelpful (<75% of respondents indicated at least 5 out of 10 importance). Certain acute test findings (eg, cognitive and balance impairments) and contextual factors (eg, absence of confounds) were consistently rated as highly important for increasing the likelihood of a mild TBI diagnosis (≥75% of respondents indicated at least 7 out of 10).

      Conclusions

      The expert survey findings identified several potential revisions to consider when updating the ACRM mild TBI definition, including preferentially weighing observable signs in a probabilistic framework, incorporating symptoms and test findings, and adding differential diagnosis considerations.

      Keywords

      List of abbreviations:

      ACRM (American Congress of Rehabilitation Medicine), CT (computed tomography), IQR (interquartile range), MRI (magnetic resonance imaging), TBI (traumatic brain injury), WHO (World Health Organization)
      The American Congress of Rehabilitation Medicine (ACRM) sponsored the publication of a definition of mild traumatic brain injury (TBI) in 1993.
      Head Injury Interdisciplinary Special Interest Group of the American Congress of Rehabilitation
      Definition of mild traumatic brain injury.
      This work was conducted by the Mild Traumatic Brain Injury Committee of the Head Injury Interdisciplinary Special Interest Group from the ACRM. For the past 25 years, that definition, reprinted in table 1, has been widely used in clinical practice and research. In early 2019, the ACRM Brain Injury Special Interest Group Mild TBI Task Force began to undertake an update of the 1993 ACRM definition of mild TBI.
      Table 1ACRM’s 1993 definition of mild traumatic brain injury
      A traumatically induced physiological disruption of brain function, as manifested by at least 1 of the following:
      • 1.
        any loss of consciousness;
      • 2.
        any loss of memory for events immediately before or after the accident;
      • 3.
        any alteration in mental state at the time of the accident (e.g., feeling dazed, disoriented, or confused); and
      • 4.
        focal neurological deficit(s) that may or may not be transient; but where the severity of the injury does not exceed the following:
      • loss of consciousness of approximately 30 minutes or less;
      • after 30 minutes, an initial Glasgow Coma Scale (GCS) of 13-15; and
      • posttraumatic amnesia (PTA) not greater than 24 hours.
      Numerous definitions of mild TBI have been published since the ACRM definition in 1993, including definitions by the Centers for Disease Control and Prevention in 2003,
      Centers for Disease Control and Prevention
      Report to Congress on mild traumatic brain injury in the United States: steps to prevent a serious public health problem.
      the World Health Organization (WHO) Collaborating Centre Task Force on Mild Traumatic Brain Injury in 2005,
      • Holm L.
      • Cassidy J.D.
      • Carroll L.J.
      • Borg J.
      Summary of the WHO Collaborating Centre for Neurotrauma Task Force on Mild Traumatic Brain Injury.
      the Department of Veterans Affairs and the Department of Defense in 2009 and 2016,
      Veterans Affairs/Department of Defense
      Management of concussion-mild traumatic brain injury.
      the Demographics and Clinical Assessment Working Group of the International and Interagency Initiative toward Common Data Elements for Research on Traumatic Brain Injury and Psychological Health in 2010,
      • Menon D.K.
      • Schwab K.
      • Wright D.W.
      • Maas A.I.
      Position statement: definition of traumatic brain injury.
      and the Ontario Neurotrauma Foundation in 2018.
      Ontario Neurotrauma Foundation
      Guidelines for concussion/mild traumatic brain injury & persistent symptoms.
      The Concussion in Sport Group has also proposed
      • Aubry M.
      • Cantu R.
      • Dvorak J.
      • et al.
      Summary and agreement statement of the 1st International Symposium on Concussion in Sport, Vienna 2001.
      and updated
      • McCrory P.
      • Meeuwisse W.
      • Dvorak J.
      • et al.
      Consensus statement on concussion in sport—the 5th international conference on concussion in sport held in Berlin, October 2016.
      a definition of sport-related concussion.
      These case definitions vary regarding which signs and symptoms must be present to diagnose mild TBI, but certain diagnostic features are consistent across these definitions (table 2). For example, all agree that a brief loss of consciousness is sufficient but not necessary to diagnose mild TBI. However, important discrepancies exist between definitions (see table 2), including how alteration in mental status is defined, whether or not structural changes to the brain belong within the classification of mild TBI, and whether subjective “postconcussion” symptoms (eg, headache, dizziness, concentration difficulty) can provide sufficient diagnostic evidence of mild TBI. There is also inconsistent use and meaning of the terms “concussion” and mild TBI. These 2 labels are sometimes considered synonymous or as subordinate or superordinate terms, with concussion representing a subset of mild TBI without structural brain injury.
      Table 2Comparison of threshold criteria for mild TBI diagnosis across organization and expert group case definitions
      ACRM

      1993
      CDC 2003WHO

      2005
      CDE

      2010
      VA/DoD 2016CISG 2017ONF 2018
      Trauma-related intracranial lesion on conventional CT or MRI can be presentYes
      Implied, but not explicitly stated, that this feature is considered sufficient evidence of mild TBI.
      YesYesYesNo
      Exclusion criterion.
      No
      Exclusion criterion.
      Yes
      Exclusion for “concussion” but not mild TBI.
      Focal neurologic deficitYesYes
      Implied, but not explicitly stated, that this feature is considered sufficient evidence of mild TBI.
      Yes
      Must rule out alternative explanations for acute alteration in brain function.
      Yes
      Must rule out alternative explanations for acute alteration in brain function.
      YesYes
      Implied, but not explicitly stated, that this feature is considered sufficient evidence of mild TBI.
      ,
      Must rule out alternative explanations for acute alteration in brain function.
      Yes
      Exclusion for “concussion” but not mild TBI.
      Loss of consciousnessYesYesYes
      Must rule out alternative explanations for acute alteration in brain function.
      YesYesYes
      Implied, but not explicitly stated, that this feature is considered sufficient evidence of mild TBI.
      ,
      Must rule out alternative explanations for acute alteration in brain function.
      Yes
      Decreased consciousnessYes
      Implied, but not explicitly stated, that this feature is considered sufficient evidence of mild TBI.
      YesYes
      Implied, but not explicitly stated, that this feature is considered sufficient evidence of mild TBI.
      ,
      Must rule out alternative explanations for acute alteration in brain function.
      YesYesYes
      Implied, but not explicitly stated, that this feature is considered sufficient evidence of mild TBI.
      ,
      Must rule out alternative explanations for acute alteration in brain function.
      Yes
      Retrograde amnesiaYesYesNoYesYes?Yes
      Post-traumatic amnesiaYesYesYes
      Must rule out alternative explanations for acute alteration in brain function.
      Yes
      Must rule out alternative explanations for acute alteration in brain function.
      YesYes
      Implied, but not explicitly stated, that this feature is considered sufficient evidence of mild TBI.
      ,
      Must rule out alternative explanations for acute alteration in brain function.
      Yes
      Confusion/disorientation (objectively assessed, including GCS<15)YesǂYesYes
      Must rule out alternative explanations for acute alteration in brain function.
      Yes
      Must rule out alternative explanations for acute alteration in brain function.
      YesYesǂ,
      Must rule out alternative explanations for acute alteration in brain function.
      Yes
      Confusion/disorientation (subjective)YesYes?Yes
      Implied, but not explicitly stated, that this feature is considered sufficient evidence of mild TBI.
      ,
      Must rule out alternative explanations for acute alteration in brain function.
      Yes
      Implied, but not explicitly stated, that this feature is considered sufficient evidence of mild TBI.
      Yes
      Implied, but not explicitly stated, that this feature is considered sufficient evidence of mild TBI.
      ,
      Must rule out alternative explanations for acute alteration in brain function.
      ?
      Dazed (subjective)YesNoNo?YesYes
      Implied, but not explicitly stated, that this feature is considered sufficient evidence of mild TBI.
      ,
      Must rule out alternative explanations for acute alteration in brain function.
      ?
      Difficulty thinking/slowed thinking (subjective)?NoNoYes
      Unclear if difficulty thinking/slowed thinking is a sign (observable behavior) or symptom (self-reported).
      Yes
      Unclear if difficulty thinking/slowed thinking is a sign (observable behavior) or symptom (self-reported).
      Yes
      Must rule out alternative explanations for acute alteration in brain function.
      Yes
      Unclear if difficulty thinking/slowed thinking is a sign (observable behavior) or symptom (self-reported).
      Physical symptomsNoNoNoNoNoYes
      Must rule out alternative explanations for acute alteration in brain function.
      Yes
      Cognitive or emotional symptomsNoNoNoNoNoYes
      Must rule out alternative explanations for acute alteration in brain function.
      No
      NOTE. “Yes” in a cell indicates that the presence of the clinical feature in that row is sufficient to rule-in a diagnosis of mild TBI, according to the case definition for that column. All case definitions specify or imply that any alteration in consciousness or mental status has an abrupt onset (“at the time of injury” or “immediately following the event”). Onset of subjective symptoms may be delayed by “minutes to hours” (CISG 2017). No case definitions specify a minimum duration for signs or symptoms.
      Abbreviations: ?, unclear whether or not this feature is considered sufficient evidence of mild TBI; CDC, Centers for Disease Control and Prevention; CDE, Demographics and Clinical Assessment Working Group of the International and Interagency Initiative toward Common Data Elements for Research on Traumatic Brain Injury and Psychological Health; CISG, Concussion in Sport Group; ONF, Ontario Neurotrauma Foundation; VA/DoD, Department of Veterans Affairs and the Department of Defense; WHO, World Health Organization Collaborating Centre Task Force on Mild Traumatic Brain Injury.
      Implied, but not explicitly stated, that this feature is considered sufficient evidence of mild TBI.
      Exclusion criterion.
      Exclusion for “concussion” but not mild TBI.
      § Must rule out alternative explanations for acute alteration in brain function.
      Unclear if difficulty thinking/slowed thinking is a sign (observable behavior) or symptom (self-reported).
      The lack of a universal case definition for mild TBI has been problematic for research and clinical practice. For example, synthesizing studies (eg, for meta-analysis) or comparing across studies that used different case definitions may be misleading, because some definitions capture more or less homogenous samples or more or less severely injured patients than others.
      • Kristman V.L.
      • Borg J.
      • Godbolt A.K.
      • et al.
      Methodological issues and research recommendations for prognosis after mild traumatic brain injury: results of the international collaboration on mild traumatic brain injury prognosis.
      Clinically, a patient who reports feeling “dazed” after a head trauma might be diagnosed as having sustained a mild TBI by a clinician who uses the ACRM case definition but not by a clinician who uses the WHO case definition.
      • Ruff R.M.
      • Iverson G.L.
      • Barth J.T.
      • Bush S.S.
      • Broshek D.K.
      Recommendations for diagnosing a mild traumatic brain injury: a National Academy of Neuropsychology education paper.
      The present study aimed to characterize current expert opinion on issues related to the definition of mild TBI. Clinical conditions can be defined by an expert consensus approach when pathologic, laboratory, and imaging confirmation of disease is not available,
      American Academy of Neurology
      Clinical Practice Guideline Process Manual.
      ,
      Committee on the Development of a Consensus Case Definition for Chronic Multisymptom Illness in 1990-1991 Gulf War Veterans, Board on the Health of Select Populations, Institute of Medicine
      Chronic multisymptom illness in Gulf War veterans: case definitions reexamined.
      as in the case of mild TBI. Expert ratings, combined with a synthesis of the best available scientific evidence, have proven useful for identifying important diagnostic features
      • Baguley I.J.
      • Perkes I.E.
      • Fernandez-Ortega J.-F.
      • Rabinstein A.A.
      • Dolce G.
      • Hendricks H.T.
      Paroxysmal sympathetic hyperactivity after acquired brain injury: consensus on conceptual definition, nomenclature, and diagnostic criteria.
      and incorporating them into a consensus-based operational case definition.
      • England J.D.
      • Gronseth G.S.
      • Franklin G.
      • et al.
      Distal symmetric polyneuropathy: a definition for clinical research - report of the American Academy of Neurology, the American Association of Electrodiagnostic Medicine, and the American Academy of Physical Medicine and Rehabilitation.
      Findings from the present study could demonstrate that achieving expert consensus on a definition of mild TBI is feasible or futile.
      We assembled an international, interdisciplinary group of clinician-scientists with expertise in mild TBI and invited them to rate (1) their views on areas of discrepancy among existing mild TBI case definitions and possible new directions for an updated definition, and (2) the diagnostic importance of various signs, symptoms, test findings, and contextual factors. We included a broader range of potential diagnostic features than have been included in previous case definitions because considerable evidence now exists concerning the diagnostic utility of serum biomarkers,
      • Begaz T.
      • Kyriacou D.N.
      • Segal J.
      • Bazarian J.J.
      Serum biochemical markers for post-concussion syndrome in patients with mild traumatic brain injury.
      ,
      • OʼConnell B.
      • Kelly Á.M.
      • Mockler D.
      • et al.
      Use of blood biomarkers in the assessment of sports-related concussion-a systematic review in the context of their biological significance.
      clinical assessment tools,
      • Echemendia R.J.
      • Broglio S.P.
      • Davis G.A.
      • et al.
      What tests and measures should be added to the SCAT3 and related tests to improve their reliability, sensitivity and/or specificity in sideline concussion diagnosis? A systematic review.
      • Yengo-Kahn A.M.
      • Hale A.T.
      • Zalneraitis B.H.
      • Zuckerman S.L.
      • Sills A.K.
      • Solomon G.S.
      The Sport Concussion Assessment Tool: a systematic review.
      • Patricios J.
      • Fuller G.W.
      • Ellenbogen R.
      • et al.
      What are the critical elements of sideline screening that can be used to establish the diagnosis of concussion? A systematic review.
      and observable behaviors (eg, motor incoordination
      • Echemendia R.J.
      • Broglio S.P.
      • Davis G.A.
      • et al.
      What tests and measures should be added to the SCAT3 and related tests to improve their reliability, sensitivity and/or specificity in sideline concussion diagnosis? A systematic review.
      ). These factors might improve the sensitivity, specificity, and reliability of mild TBI diagnostic methods.

      Methods

      Between January and March of 2019, a Working Group from the ACRM Mild TBI Task Force (co-led by NDS and GLI) identified potential Expert Panel members for the updated case definition project by scanning author lists of mild TBI agreement statements and clinical practice guidelines published within the last 5 years,
      Ontario Neurotrauma Foundation
      Guidelines for concussion/mild traumatic brain injury & persistent symptoms.
      ,
      • McCrory P.
      • Meeuwisse W.
      • Dvorak J.
      • et al.
      Consensus statement on concussion in sport—the 5th international conference on concussion in sport held in Berlin, October 2016.
      ,
      • Lumba-Brown A.
      Centers for Disease Control and prevention guideline on the diagnosis and management of mild traumatic brain injury among children.
      Ontario Neurotrauma Foundation
      Guidelines Diagnosing and Managing Pediatric Concussion.
      • Broglio S.P.
      • Cantu R.C.
      • Gioia Ga
      • et al.
      National Athletic Trainers’ Association position statement: management of sport concussion.
      • Collins M.W.
      • Kontos A.P.
      • Okonkwo D.O.
      • et al.
      Statements of agreement from the Targeted Evaluation and Active Management (TEAM) approaches to treating concussion meeting held in Pittsburgh, October 15-16, 2015.
      Veterans Affairs/Department of Defense
      Management of concussion-mild traumatic brain injury (MTBI) clinical practice guidelines.
      • Silverberg N.D.
      • Iaccarino M.A.
      • Panenka W.J.
      • et al.
      Management of concussion and mild traumatic brain injury: a synthesis of practice guidelines.
      asking for nominations from Working Group members, and conducting literature searches for high-impact articles. The target sample size was 20 to 40 individuals to ensure reliability of group judgments while limiting coordination difficulties and diminishing returns with an excessively large Expert Panel.
      • Vogel C.
      • Zwolinsky S.
      • Griffiths C.
      • Hobbs M.
      • Henderson E.
      • Wilkins E.
      A Delphi study to build consensus on the definition and use of big data in obesity research.
      ,
      • Murphy M.
      • Black N.
      • Lamping D.
      • et al.
      Consensus development methods, and their use in clinical guideline development.
      The Working Group identified 32 individuals as mild TBI experts based on the following criteria: (1) demonstrates specialized knowledge in mild TBI based on academic publication history (eg, lead or senior author of relevant studies) or previous participation in mild TBI clinical practice guideline development; (2) has clinical experience diagnosing patients with mild TBI; and (3) contributes to the diversity of the sample with respect to their clinical discipline and specialty, practice setting, geographic location, or sex. The Working Group invited 32 individuals to participate on the Expert Panel, and 31 agreed. These 31 experts were sent an initial electronic survey in May 2019 using the Qualtrics platform,a hosted by The University of British Columbia, and were asked to complete it by June 25, 2019. A reminder e-mail was sent on June 19, 2019. Experts who did not complete the survey by June 25, 2019 were prompted once more by e-mail to complete it by July 2, 2019. The 1 expert who did not respond to the invitation (and follow-up e-mail) to join the Expert Panel was not sent a survey.
      The 80-item survey was created by the Working Group using an iterative process of drafting and revising. In the first section of the survey, respondents were prompted to enter their demographics, clinical practice characteristics, and contributions to research (table 3). In the second section, they were presented with a series of statements (shown in table 4) addressing points of controversy. Items in this section of the survey were based on observed discrepancies between previous case definitions (see table 2). However, we included a few additional statements regarding possible new directions that could be considered for an updated case definition. Respondents were asked to rate their agreement with each statement on a scale from 1 (strongly disagree) to 5 (strongly agree), with a neutral midpoint. They were also invited to add narrative comments to each statement and response. The statements were balanced with respect to positive (should) or negative (should not) wording, and respondents were explicitly told that the statements do not necessarily reflect the views of the ACRM Mild TBI Task Force. Note that this section of the survey contained additional items for project planning purposes that are not reported here. The third section was modeled based on a previous study
      • Baguley I.J.
      • Perkes I.E.
      • Fernandez-Ortega J.-F.
      • Rabinstein A.A.
      • Dolce G.
      • Hendricks H.T.
      Paroxysmal sympathetic hyperactivity after acquired brain injury: consensus on conceptual definition, nomenclature, and diagnostic criteria.
      that aimed to establish expert consensus-based diagnostic criteria. In this section, respondents were asked to rate a variety of signs, symptoms, test findings, and contextual factors (grouped under those subheadings), and asked to rate “how important each factor is in increasing the likelihood that the person sustained a mild TBI,” on a scale from 1 (not at all important) to 10 (extremely important). They could decline to provide a rating and instead indicate “I don’t know.” The survey items in the third section were taken from existing diagnostic criteria (see table 2), standardized symptom inventories,
      • Alla S.
      • Sullivan S.J.
      • Hale L.
      • McCrory P.
      Self-report scales/checklists for the measurement of concussion symptoms: a systematic review.
      and promising tests and assessment procedures identified from rapid literature reviews being conducted in parallel with this survey by the Working Group for the broader case definition project.
      Table 3Demographic characteristics of the expert consensus group
      Demographic VariableSurvey ResponseFrequencyPercent
      SpecialtyPhysical medicine and rehabilitation825.8
      Neurology619.4
      Neuropsychology516.1
      Emergency medicine412.9
      Neurosurgery39.7
      Sports medicine26.5
      Critical care medicine13.2
      Behavioral neurology and neuropsychiatry13.2
      Athletic trainer13.2
      Primary work locationAcademic medical center/hospital2064.5
      University412.9
      Other412.9
      Community hospital13.2
      Military medical center/treatment facility13.2
      Research facility13.2
      Veterans Health Administration hospital00.0
      Secondary work locationNone (primary location only)1135.5
      Academic medical center/hospital722.6
      Research facility722.6
      University39.7
      Veterans Health Administration hospital39.7
      Community hospital00.0
      Military medical center/treatment facility00.0
      Other00.0
      Geographic locationUnited States–Northeast1032.3
      United States–South412.9
      United States–West412.9
      Europe412.9
      Canada412.9
      United States–Midwest39.7
      Australia26.5
      SexMale2580.7
      Female619.4
      Years in independent practice (postlicensure)>152787.1
      10-1539.7
      5-1013.2
      1-500.0
      0 (trainee)00.0
      Academic contributions (peer-reviewed publications)≥300412.9
      200-299722.6
      100-1991135.5
      30-99722.6
      10-2926.5
      1-900.0
      000.0
      Patient population (yes/no)Athletes2683.9
      Civilians2683.9
      Military service members or veterans1445.2
      Adults2787.1
      Adolescents2271.0
      Children1651.6
      Table 4Expert ratings on possible changes to the ACRM diagnostic criteria for mild TBI
      StatementMean ± SDMedianModeRangeIQR
      Diagnostic criteria for mTBI should specify that alternative explanations for altered mental status (eg, acute stress reaction, substance/alcohol use) must be ruled out.4.65 ± 0.66552-54-5
      Rapid onset postconcussion symptoms (eg, headache, dizziness, feeling like “in a fog”) after head/neck trauma should indicate at least a possible mTBI.4.45±1.03551-54-5
      Diagnostic criteria for mTBI should incorporate levels of certainty (eg, possible, probable, and definite categories) rather than be binary (diagnosis present or absent).4.35±0.99551-54-5
      Diagnostic criteria should distinguish between mTBI without neuroimaging evidence of intracranial trauma
      Includes: epidural hematoma, subdural hematoma, subarachnoid hemorrhage, midline shift supratentorial, cisternal compression, contusion, intracerebral hemorrhage, intraventricular hemorrhage, diffuse axonal injury, brain edema, and encephalomalacia (Haacke et al, J Magn Reson Imaging 2010;32:516-43 and Broglio et al, J Neurotrauma 2018;35:2776-83.). Excludes: skull fracture and nonspecific white matter lesions (eg, MRI T2 hyperintensities).
      (also known as uncomplicated mTBI of commotio cerebri) and mTBI with neuroimaging evidence of intracranial trauma
      Includes: epidural hematoma, subdural hematoma, subarachnoid hemorrhage, midline shift supratentorial, cisternal compression, contusion, intracerebral hemorrhage, intraventricular hemorrhage, diffuse axonal injury, brain edema, and encephalomalacia (Haacke et al, J Magn Reson Imaging 2010;32:516-43 and Broglio et al, J Neurotrauma 2018;35:2776-83.). Excludes: skull fracture and nonspecific white matter lesions (eg, MRI T2 hyperintensities).
      (also known as complicated mTBI or contusion cerebri).
      4.23±1.12551-54-5
      Diagnostic criteria for mTBI should include a maximum timeframe for the onset of symptoms (eg, headache, dizziness, feeling like “in a fog”).4.06±1.03452-54-5
      Most current definitions of mTBI include loss of consciousness and posttraumatic amnesia as observable signs. Diagnostic criteria for mTBI should be expanded to include other observable signs (eg, blank/vacant look or motor incoordination).4.06±1.24451-54-5
      Neuroimaging evidence of intracranial trauma
      Includes: epidural hematoma, subdural hematoma, subarachnoid hemorrhage, midline shift supratentorial, cisternal compression, contusion, intracerebral hemorrhage, intraventricular hemorrhage, diffuse axonal injury, brain edema, and encephalomalacia (Haacke et al, J Magn Reson Imaging 2010;32:516-43 and Broglio et al, J Neurotrauma 2018;35:2776-83.). Excludes: skull fracture and nonspecific white matter lesions (eg, MRI T2 hyperintensities).
      implies a more severe form of TBI, not an mTBI.
      3.26±1.37341-52-4
      The terms “concussion” and “mild traumatic brain injury” can be used synonymously.3.19±1.4032, 41-52-4
      Diagnostic criteria for mTBI should include a minimum duration of symptoms (eg, headache, dizziness, feeling like “in a fog”).2.97±1.25321-52-4
      Some definitions of mTBI are based on observable signs only (eg, loss of consciousness and posttraumatic confusion) and do not include subjectively experienced symptoms (eg, headache, dizziness, feeling like “in a fog”). In my opinion, symptoms are not necessary or sufficient to diagnose mTBI, and therefore should be omitted from diagnostic criteria.1.84±1.13211-51-2
      NOTE. Each statement was rated on a scale from 1 (strongly disagree) to 5 (strongly agree). Statements were worded positively (should) or negatively (should not) at random.
      Includes: epidural hematoma, subdural hematoma, subarachnoid hemorrhage, midline shift supratentorial, cisternal compression, contusion, intracerebral hemorrhage, intraventricular hemorrhage, diffuse axonal injury, brain edema, and encephalomalacia (Haacke et al, J Magn Reson Imaging 2010;32:516-43 and Broglio et al, J Neurotrauma 2018;35:2776-83.). Excludes: skull fracture and nonspecific white matter lesions (eg, MRI T2 hyperintensities).
      Respondents were assured that their data would be anonymized prior to analysis and reported in aggregate form. Respondents consented to having their responses shared publicly in aggregate form. The University of British Columbia Behavioural Research Ethics Board deemed the present study exempt from requiring ethics board review and approval.

      Statistical analysis

      Frequency and central tendency statistics were reported for the first section of the survey. For the second section, consistent agreement was defined for the present study as 75% or more of the sample reporting that they somewhat or strongly agreed, and consistent disagreement was defined as 75% or more of the sample reporting that they somewhat or strongly disagreed. In contrast, agreement was considered ambiguous or inconsistent when the interquartile range included an agreement midpoint rating of 3. For the third section of the survey, we considered items as diagnostically important if 75% or more of respondents gave a rating of at least 7 out of 10, and diagnostically useful if 75% or more of respondents gave a rating of at least 5 out of 10. Given the skewed distributions of most diagnostic features, we focused our interpretation on median ratings. We used the Wilcoxon signed ranks test to compare across items and effect size r to measure the effect size of those differences. The coefficient r was calculated by dividing the test statistic (Z) by the square root of the sample size
      • Fritz C.O.
      • Morris P.E.
      • Richler J.J.
      Effect size estimates: current use, calculations, and interpretation.
      and was interpreted as small (r=0.10), medium (r=0.30), or large (r=0.50).
      • Cohen J.
      Statistical power analysis for the behavioral sciences. 2nd ed.

      Results

      The survey completion rate was 100% (31 out of 31). Respondents took an average (median) of 28.4 minutes (interquartile range [IQR], 21.5-48.8min) to complete the survey. No data were missing (ie, no unanswered items). However, some respondents used the “I don’t know” response option for certain diagnostic importance ratings.
      The characteristics of the sample are reported in table 3. Most respondents were in independent clinical practice (postlicensure) for more than 10 years (96.8%), and the majority had published at least 100 peer-reviewed articles (71.0%). The sample represented a variety of specialties involved in the clinical care of diverse mild TBI subpopulations. The majority of respondents worked primarily at an academic medical center, hospital, or university (77.4%) in the United States (67.7%). Men were over-represented (81%).

      Opinions on areas of discrepancy among existing mild TBI case definitions and possible new directions for an updated definition

      Agreement ratings with statements relating to discrepancies among definitions and potentially controversial topics are shown in table 4, from highest to lowest agreement. There was consistent agreement or disagreement for the majority of items. In contrast, agreement was ambiguous or inconsistent for select items (eg, whether neuroimaging evidence of intracranial trauma implies a more severe form of TBI and whether diagnostic criteria for mild TBI should include a minimum duration of symptoms).
      Agreement was consistent that diagnostic criteria for mild TBI should include a maximum timeframe for the onset of symptoms. Respondents were also asked to recommend a specific time cutoff from the following options: 1 hour, 12 hours, 24 hours, 3 days, 7 days, or “I do not recommend a maximum timeframe.” Their responses are shown in table 5. The most common response (n=12, 38.7%) was 3 days. Most participants favored a maximum timeframe of 3 days or less (cumulative percentage, 77.4%). However, some preferred a longer timeframe (n=2, 6.5%) or declined to recommend a specific timeframe (n=5, 16.1%).
      Table 5Expert recommendations for the minimum duration of symptoms and maximum timeframe for the onset of symptoms
      Diagnostic CriteriaRecommendationFrequencyPercentCumulative Percent
      Minimum duration of symptoms (eg, headache, dizziness, feeling like “in a fog”)10 seconds26.456.45
      30 seconds39.6816.13
      5 minutes516.1332.26
      15 minutes412.9045.16
      I do not recommend a minimum duration1754.84100.00
      Maximum timeframe for the onset of symptoms (eg, headache, dizziness, feeling like “in a fog”)1 hour39.689.68
      12 hours39.6819.35
      24 hours619.3538.71
      3 days1238.7177.42
      7 days26.4583.87
      I do not recommend a maximum timeframe516.13100.00

      Diagnostic importance ratings

      Ratings of diagnostic importance are shown in Fig 1, Fig 2, Fig 3, Fig 4. Items that were endorsed as diagnostically important by the majority of respondents (ie, ≥75% gave a rating of ≥7 out of 10) are denoted with the unique symbol ǂ. Loss of consciousness, disorientation, confused or inappropriate behavior, seizure with clonic movements, focal neurologic deficits, and tonic posturing all had a median rating of 10 out of 10. With the exception of facial injury (median, 4; IQR, 2-6) and scalp injury (median, 4; IQR, 2-8), all other signs were rated as diagnostically useful, with median scores of 7 to 8. Diagnostic importance ratings for subjective symptoms (see fig 2) ranged widely. Six symptoms (feeling confused, feeling disoriented, feeling dazed, balance problems, difficulty remembering, and dizziness) were rated as diagnostically important, whereas pressure in head (median, 6; IQR, 5-8), don’t feel right (median, 6; IQR, 5-8), seeing stars (median, 6; IQR, 4-9), drowsiness (median, 6; IQR, 4-8), nervous or anxious (median, 6; IQR, 5-7), fatigue (median, 6; IQR, 4-7), and neck pain (median, 5; IQR, 3-7) were not rated as important or helpful. Overall, observable signs were rated as more diagnostically important than symptoms (Wilcoxon signed ranks test, Z=3.77, P<.001, r=0.68). Limiting this comparison to the top 5-highest rated items in each category yielded a similar finding, with observable signs being rated as more diagnostically important than symptoms (Wilcoxon signed ranks test, Z=4.05, P<.001, r=0.73).
      Figure thumbnail gr1
      Fig 1Perceived diagnostic importance of observable signs. o indicates outlier data point (>1.5 times the IQR), ∗ indicates extreme outlier data point (>3 times the IQR), and ǂ indicates that 75% or more of the respondents rated 7 or higher.
      Figure thumbnail gr2
      Fig 2Perceived diagnostic importance of subjective symptoms. o indicates outlier data point (>1.5 times the IQR), and ǂ indicates that 75% or more of the respondents rated 7 or higher.
      Figure thumbnail gr3
      Fig 3Perceived diagnostic importance of acute test findings. o indicates outlier data point (>1.5 times the IQR), ∗ indicates extreme outlier data point (>3 times the IQR), and ǂ indicates that 75% or more of the respondents rated 7 or higher. Abbreviations: GFAP, glial fibrillary acidic protein; S100B, S100 calcium binding protein β; UCH-L1, ubiquitin carboxy-terminal hydrolase L1; VOM, vestibulo-oculomotor.
      Figure thumbnail gr4
      Fig 4Perceived diagnostic importance of contextual factors. o indicates outlier data point (>1.5 times the IQR), and ǂ indicates that 75% or more of the respondents rated 7 or higher.
      Regarding acute test findings (see fig 3), a trauma-related intracranial abnormality on computed tomography (CT) or magnetic resonance imaging (MRI) was among the most strongly endorsed (median, 10; IQR, 8.5-10). However, the distribution was very skewed, with many participants rating it as very diagnostic of mild TBI (median, 10), whereas 5 (16.1%) respondents rated positive CT or MRI findings as not diagnostically important (<7 out of 10). Of these 5 participants, 4 somewhat or strongly agreed with the statement (see table 4) that “neuroimaging evidence of intracranial trauma implies a more severe form of TBI, not a mild TBI,” suggesting that they viewed positive CT or MRI findings as a rule-out rather than a rule-in test finding. Acute findings of cognitive impairment (median, 9; IQR, 8-10), balance impairment (median, 9; IQR, 7-10), vestibulo-oculomotor impairment (median, 8; IQR, 7-10), and a focal neurologic deficit (median, 9; IQR, 8-10) were rated as diagnostically important. Exertion intolerance and 2 of the serum biomarkers queried (ubiquitin C-terminal hydrolase L1 and glial fibrillary acidic protein) were rated as diagnostically useful.
      Of the contextual factors (see fig 4), an absence of alternative explanations for acutely altered mental status (median, 9; IQR, 7-10) was the only item rated as diagnostically important. All other contextual factors were rated as diagnostically useful.

      Discussion

      Since the ACRM developed a case definition of mild TBI in 1993, many alternative definitions have been published using variable methodological rigor and with key differences that result in discrepant diagnostic classifications. The current survey study aimed to characterize expert opinion regarding existing case definitions and to identify potentially helpful approaches and diagnostic features that will inform a Delphi consensus-driven refinement of the 1993 ACRM case definition of mild TBI. Surveying our interdisciplinary, international group of experts revealed areas of agreement but also ongoing controversy.
      One area of agreement was the view that some individuals with mild TBI initially present with only subjective symptoms and that such cases should be captured by diagnostic criteria. The original ACRM definition did not include symptoms but vaguely acknowledged circumstances in which it is “appropriate to consider symptomatology that, when linked to a traumatic head injury, can suggest the existence of a mild traumatic brain injury.”1(p86) Subsequent definitions have varied on whether acute symptoms such as headaches, dizziness, and subjective cognitive difficulties can be sufficient to diagnose a mild TBI and under what circumstances symptoms alone can be considered diagnostic. The expert panel was clear that acute symptoms are diagnostically important. The expert panel did not view that setting a threshold for the minimum duration of symptoms would be feasible but might have if asked about a minimum duration for particular symptoms, such as feeling dazed.
      • Walker W.C.
      • Cifu D.X.
      • Hudak A.M.
      • Goldberg G.
      • Kunz R.D.
      • Sima A.P.
      Structured interview for mild traumatic brain injury after military blast: inter-rater agreement and development of diagnostic algorithm.
      Most experts expressed that symptom onset after mild TBI could be delayed but could generally only be causally attributable to mild TBI if the symptoms appear within a few days of the injury.
      Consistent with case definitions for other health conditions,
      • England J.D.
      • Gronseth G.S.
      • Franklin G.
      • et al.
      Distal symmetric polyneuropathy: a definition for clinical research - report of the American Academy of Neurology, the American Association of Electrodiagnostic Medicine, and the American Academy of Physical Medicine and Rehabilitation.
      the expert panel considered observable signs to be more important than subjective symptoms. Most existing case definitions for mild TBI do not distinguish between observable signs (eg, inability to answer orientation questions) and subjective symptoms (eg, feeling “confused”), and in no previous definition are signs given disproportionate weight. The present findings suggest that a revised case definition should consider both of these issues. Disproportionate weighting of diagnostic features is possible in a probabilistic framework, which the expert panel generally supported. Probabilistic diagnostic criteria exist for a number of other neurologic conditions
      • Baguley I.J.
      • Perkes I.E.
      • Fernandez-Ortega J.-F.
      • Rabinstein A.A.
      • Dolce G.
      • Hendricks H.T.
      Paroxysmal sympathetic hyperactivity after acquired brain injury: consensus on conceptual definition, nomenclature, and diagnostic criteria.
      ,
      • Rascovsky K.
      • Hodges J.R.
      • Knopman D.
      • et al.
      Sensitivity of revised diagnostic criteria for the behavioural variant of frontotemporal dementia.
      ,
      • Stern B.J.
      • Royal W.
      • Gelfand J.M.
      • et al.
      Definition and consensus diagnostic criteria for neurosarcoidosis: from the Neurosarcoidosis Consortium Consensus Group.
      and have been considered for mild TBI.
      • Garcia G.-G.P.
      • Lavieri M.S.
      • Jiang R.
      • Mcallister T.W.
      • Mccrea M.A.
      • Broglio S.P.
      A data-driven approach to unlikely, possible, probable, and definite acute concussion assessment.
      An important area of ongoing controversy is whether head trauma resulting in structural brain injury visible on CT or MRI should be considered part of the mild TBI spectrum or a categorically different (ie, more severe) injury. This controversy has also manifested as disagreement on whether "concussion” and mild TBI are synonymous diagnostic labels, or whether the former should be reserved for patients without structural brain injury on CT or MRI. Existing case definitions and expert opinions (based on the present survey findings) are divided on these points, which may reflect a divide between the civilian (nonsport) mild TBI and the sport concussion
      • McCrory P.
      • Meeuwisse W.
      • Dvorak J.
      • et al.
      Consensus statement on concussion in sport—the 5th international conference on concussion in sport held in Berlin, October 2016.
      and United States military medicine
      • McAllister T.
      • McCrea M.
      Long-term cognitive and neuropsychiatric consequences of repetitive concussion and head-impact exposure.
      communities. Increasingly sensitive neuroimaging modalities may create a moving threshold such that more patients who present with a clinically “mild” TBI will have abnormal scans. Future advances in blood-based and imaging biomarkers
      • Yue J.K.
      • Yuh E.L.
      • Korley F.K.
      • et al.
      Association between plasma GFAP concentrations and MRI abnormalities in patients with CT-negative traumatic brain injury in the TRACK-TBI cohort: a prospective multicentre study.
      ,
      • Manley G.T.
      • Mac Donald C.L.
      • Markowitz A.J.
      • et al.
      The Traumatic Brain Injury Endpoints Development (TED) initiative: progress on a public-private regulatory collaboration to accelerate diagnosis and treatment of traumatic brain injury.
      may be able to not only improve the identification of mild TBI (eg, when clinical signs and symptoms are equivocal), but also refine the grading of TBI severity across the disease spectrum, dissolving the traditional boundary between mild versus moderate-to-severe TBI
      • Meares S.
      • Shores E.A.
      • Taylor A.J.
      • et al.
      The prospective course of postconcussion syndrome: the role of mild traumatic brain injury.
      and rendering this nosologic issue moot.
      The overall degree of consistency between expert ratings on the present survey suggests that achieving consensus on an updated mild TBI case definition, the next phase of this initiative, may be challenging but feasible. Consider that when queried about 10 intentionally controversial statements in the second section of the survey, the IQR of responses included agree and strongly agree or disagree and strongly disagree for 7 statements. Across the diagnostic importance ratings of observable signs, symptoms, test findings, and contextual factors, most interquartile ranges spanned 2 to 3 points out of 10, although the range was broad. It is unclear whether some variability in responding resulted from confusing wording, varying subjective interpretations of “diagnostic importance,” and other sources of measurement error rather than reflecting true variability in respondent’s beliefs.

      Study limitations

      A strength of the current study was the 100% survey completion rate. However, as with all survey research, the study only provides a snapshot of respondents’ views at one point in time. The survey items did not undergo rigorous psychometric evaluation or pilot testing. Certain areas may have been insufficiently queried. For example, respondents were asked about the diagnostic importance of the impact force, but not about other potentially relevant biomechanical variables. The survey did not tie diagnostic importance ratings of self-reported symptoms to a specific timeframe. This may have contributed to variability in survey responses. It is possible that survey respondents would have rated certain vsymptoms as more or less diagnostically useful at different time points. Previous research suggests that the diagnostic value of current symptom reporting generally lessens over time since injury.
      • Bernard C.O.
      • Ponsford J.A.
      • McKinlay A.
      • McKenzie D.
      • Krieser D.
      Predictors of post-concussive symptoms in young children: Injury versus non-injury related factors.
      • Mclean S.A.
      • Kirsch N.L.
      • Tan-Schriner C.U.
      • et al.
      Health status, not head injury, predicts concussion symptoms after minor injury.
      • Hoge C.W.
      • McHurk D.
      • Thomas J.L.
      • Cox A.L.
      • Engel C.C.
      • Castro C.A.
      Mild traumatic brain injury in U.S. soldiers returning from Iraq.
      • Huang J.
      • Gates A.J.
      • Sinatra R.
      • Barabási A.L.
      Historical comparison of gender inequality in scientific careers across countries and disciplines.
      Another important limitation of the study was sampling bias. Our aim was to recruit leading clinician-scientists in the field of mild TBI for the Expert Panel. We attempted to operationalize eligibility criteria for the Expert Panel, but judgments about who is an “expert” are of course partially subjective and susceptible to bias. The Working Group’s perception of expertise likely resulted in sampling bias, as the great majority of Expert Panel members were men (81%) and a majority were from the United States (68%). Sex and gender imbalances in cumulative publication impact
      • Witteman H.O.
      • Hendricks M.
      • Straus S.
      • Tannenbaum C.
      Are gender gaps due to evaluations of the applicant or the science? A natural experiment at a national funding agency.
      owing to systemic inequities, for example in access to research funding
      • Bohren M.A.
      • Javadi D.
      • Vogel J.P.
      Gender balance in WHO panels for guidelines published from 2008 to 2018.
      and participation in clinical practice guideline development,
      • Nielsen M.W.
      • Alegria S.
      • Börjeson L.
      • et al.
      Gender diversity leads to etter science.
      may have compounded our sampling bias. Teams with too few women may be less collaborative, less effective at solving problems, and less likely to consider diverse viewpoints. Because only 19% of survey respondents were from outside of North America, cultural bias may have also affected our findings. Emergency medicine and primary care physicians, who most often make the initial mild TBI diagnosis, were also under-represented in the survey sample. Future steps of the ACRM Brain Injury Special Interest Group Mild TBI Task Force’s endeavor to update of the 1993 ACRM definition of mild TBI should strategically improve diversity on the Expert Panel and to involve stakeholders from under-represented groups.

      Conclusions

      The findings from this survey clarify where a group of experts, identified by the ACRM Brain Injury Special Interest Group Mild TBI Task Force, stand on controversial issues related to the diagnosis of mild TBI. The present study also identified observable signs, symptoms, test findings, and contextual factors of perceived diagnostic importance for mild TBI, many of which have not been considered in previous diagnostic frameworks. Together with the latest empirical evidence, the survey findings will help guide an updated ACRM case definition of mild TBI. Specifically, the survey findings confirm the importance of elements unique to certain diagnostic criteria that should be integrated into an updated case definition, such as the WHO’s requirement to rule out confounds for confusion and posttraumatic amnesia,
      • Holm L.
      • Cassidy J.D.
      • Carroll L.J.
      • Borg J.
      Summary of the WHO Collaborating Centre for Neurotrauma Task Force on Mild Traumatic Brain Injury.
      the Demographics and Clinical Assessment Working Group of the International and Interagency Initiative toward Common Data Elements’ specification that structural neuroimaging findings can rule-in a mild TBI diagnosis,
      • Menon D.K.
      • Schwab K.
      • Wright D.W.
      • Maas A.I.
      Position statement: definition of traumatic brain injury.
      and the Concussion in Sport Group’s proposition that symptoms following a head impact, without associated observable signs, can warrant a mild TBI diagnosis under some circumstances.
      • McCrory P.
      • Meeuwisse W.
      • Dvorak J.
      • et al.
      Consensus statement on concussion in sport—the 5th international conference on concussion in sport held in Berlin, October 2016.
      The survey findings also suggest that, unlike any previous case definition of mild TBI, an updated definition should (1) consider a probabilistic framework that weighs observable signs more than subjective symptoms and (2) incorporate objective cognitive, balance, and vestibular-oculomotor test findings, if research evidence now supports this. The present study findings also help to clarify current expert opinion on the lowest threshold for diagnosing a mild TBI, and therefore, for diagnosing TBI in general. Finally, where research evidence is lacking, the present study’s characterization of expert opinion can offer guidance with an updated case definition of mild TBI. For example, in the absence of evidence that certain symptoms rated as unhelpful in this survey study (eg, drowsiness or feeling anxious) can discriminate between people with versus those without mild TBI, these symptoms should not be included in an updated case definition, despite their appearance on standardized symptom checklists used in research and clinical practice. Our working group’s next steps are to finalize the aforementioned rapid literature reviews, draft an updated case definition based on evidence syntheses and the present survey findings, and begin Delphi voting with the 31 members of the Expert Consensus Panel.

      Supplier

      • a.
        Qualtrics; Qualtrics, LLC.

      Acknowledgment

      We thank Nathan Huebschmann for assistance with data analysis and preparing tables and figures.

      References

        • Head Injury Interdisciplinary Special Interest Group of the American Congress of Rehabilitation
        Definition of mild traumatic brain injury.
        J Head Trauma Rehabil. 1993; 8: 86-87
        • Centers for Disease Control and Prevention
        Report to Congress on mild traumatic brain injury in the United States: steps to prevent a serious public health problem.
        (Available at:) (Accessed January 14, 2019)
        • Holm L.
        • Cassidy J.D.
        • Carroll L.J.
        • Borg J.
        Summary of the WHO Collaborating Centre for Neurotrauma Task Force on Mild Traumatic Brain Injury.
        J Rehabil Med. 2005; 37: 137-141
        • Veterans Affairs/Department of Defense
        Management of concussion-mild traumatic brain injury.
        2009
        • Menon D.K.
        • Schwab K.
        • Wright D.W.
        • Maas A.I.
        Position statement: definition of traumatic brain injury.
        Arch Phys Med Rehabil. 2010; 91: 1637-1640
        • Ontario Neurotrauma Foundation
        Guidelines for concussion/mild traumatic brain injury & persistent symptoms.
        (Available at:) (Accessed January 14, 2019)
        • Aubry M.
        • Cantu R.
        • Dvorak J.
        • et al.
        Summary and agreement statement of the 1st International Symposium on Concussion in Sport, Vienna 2001.
        Clin J Sport Med. 2002; 12: 6-11
        • McCrory P.
        • Meeuwisse W.
        • Dvorak J.
        • et al.
        Consensus statement on concussion in sport—the 5th international conference on concussion in sport held in Berlin, October 2016.
        Br J Sports Med. 2017; 51: 838-847
        • Kristman V.L.
        • Borg J.
        • Godbolt A.K.
        • et al.
        Methodological issues and research recommendations for prognosis after mild traumatic brain injury: results of the international collaboration on mild traumatic brain injury prognosis.
        Arch Phys Med Rehabil. 2014; 95: S265-S277
        • Ruff R.M.
        • Iverson G.L.
        • Barth J.T.
        • Bush S.S.
        • Broshek D.K.
        Recommendations for diagnosing a mild traumatic brain injury: a National Academy of Neuropsychology education paper.
        Arch Clin Neuropsychol. 2009; 24: 3-10
        • American Academy of Neurology
        Clinical Practice Guideline Process Manual.
        2011 (https://www.aan.com/siteassets/home-page/policy-and-guidelines/guidelines/about-guidelines/11guidelinedevmanual_v408_web.pdf. Accessed October 20, 2020)
        • Committee on the Development of a Consensus Case Definition for Chronic Multisymptom Illness in 1990-1991 Gulf War Veterans, Board on the Health of Select Populations, Institute of Medicine
        Chronic multisymptom illness in Gulf War veterans: case definitions reexamined.
        National Academies Press (US), Washington, DC2014
        • Baguley I.J.
        • Perkes I.E.
        • Fernandez-Ortega J.-F.
        • Rabinstein A.A.
        • Dolce G.
        • Hendricks H.T.
        Paroxysmal sympathetic hyperactivity after acquired brain injury: consensus on conceptual definition, nomenclature, and diagnostic criteria.
        J Neurotrauma. 2014; 31: 1515-1520
        • England J.D.
        • Gronseth G.S.
        • Franklin G.
        • et al.
        Distal symmetric polyneuropathy: a definition for clinical research - report of the American Academy of Neurology, the American Association of Electrodiagnostic Medicine, and the American Academy of Physical Medicine and Rehabilitation.
        Neurology. 2005; 64: 199-207
        • Begaz T.
        • Kyriacou D.N.
        • Segal J.
        • Bazarian J.J.
        Serum biochemical markers for post-concussion syndrome in patients with mild traumatic brain injury.
        J Neurotrauma. 2006; 23: 1201-1210
        • OʼConnell B.
        • Kelly Á.M.
        • Mockler D.
        • et al.
        Use of blood biomarkers in the assessment of sports-related concussion-a systematic review in the context of their biological significance.
        Clin J Sport Med. 2018; 28: 561-571
        • Echemendia R.J.
        • Broglio S.P.
        • Davis G.A.
        • et al.
        What tests and measures should be added to the SCAT3 and related tests to improve their reliability, sensitivity and/or specificity in sideline concussion diagnosis? A systematic review.
        Br J Sports Med. 2017; 51: 895-901
        • Yengo-Kahn A.M.
        • Hale A.T.
        • Zalneraitis B.H.
        • Zuckerman S.L.
        • Sills A.K.
        • Solomon G.S.
        The Sport Concussion Assessment Tool: a systematic review.
        Neurosurg Focus. 2016; 40: E6
        • Patricios J.
        • Fuller G.W.
        • Ellenbogen R.
        • et al.
        What are the critical elements of sideline screening that can be used to establish the diagnosis of concussion? A systematic review.
        Br J Sports Med. 2017; 51: 888-894
        • Lumba-Brown A.
        Centers for Disease Control and prevention guideline on the diagnosis and management of mild traumatic brain injury among children.
        JAMA Pediatr. 2018; 172e182853
        • Ontario Neurotrauma Foundation
        Guidelines Diagnosing and Managing Pediatric Concussion.
        (Available at:) (Accessed December 3, 2019)
        • Broglio S.P.
        • Cantu R.C.
        • Gioia Ga
        • et al.
        National Athletic Trainers’ Association position statement: management of sport concussion.
        J Athl Train. 2014; 49: 245-265
        • Collins M.W.
        • Kontos A.P.
        • Okonkwo D.O.
        • et al.
        Statements of agreement from the Targeted Evaluation and Active Management (TEAM) approaches to treating concussion meeting held in Pittsburgh, October 15-16, 2015.
        Neurosurgery. 2016; 79: 912-929
        • Veterans Affairs/Department of Defense
        Management of concussion-mild traumatic brain injury (MTBI) clinical practice guidelines.
        (Available at:) (Accessed January 14, 2019)
        • Silverberg N.D.
        • Iaccarino M.A.
        • Panenka W.J.
        • et al.
        Management of concussion and mild traumatic brain injury: a synthesis of practice guidelines.
        Arch Phys Med Rehabil. 2020; 101: 382-393
        • Vogel C.
        • Zwolinsky S.
        • Griffiths C.
        • Hobbs M.
        • Henderson E.
        • Wilkins E.
        A Delphi study to build consensus on the definition and use of big data in obesity research.
        Int J Obes. 2019; 43: 2573-2586
        • Murphy M.
        • Black N.
        • Lamping D.
        • et al.
        Consensus development methods, and their use in clinical guideline development.
        Heal Technol Assess. 1998; 2: 1-88
        • Alla S.
        • Sullivan S.J.
        • Hale L.
        • McCrory P.
        Self-report scales/checklists for the measurement of concussion symptoms: a systematic review.
        Br J Sports Med. 2009; 43: i3-12
        • Fritz C.O.
        • Morris P.E.
        • Richler J.J.
        Effect size estimates: current use, calculations, and interpretation.
        J Exp Psychol Gen. 2012; 141: 2-18
        • Cohen J.
        Statistical power analysis for the behavioral sciences. 2nd ed.
        Lawrence Erlbaum Associates, Hillsdale, NJ1988
        • Walker W.C.
        • Cifu D.X.
        • Hudak A.M.
        • Goldberg G.
        • Kunz R.D.
        • Sima A.P.
        Structured interview for mild traumatic brain injury after military blast: inter-rater agreement and development of diagnostic algorithm.
        J Neurotrauma. 2015; 32: 464-473
        • Rascovsky K.
        • Hodges J.R.
        • Knopman D.
        • et al.
        Sensitivity of revised diagnostic criteria for the behavioural variant of frontotemporal dementia.
        Brain. 2011; 134: 2456-2477
        • Stern B.J.
        • Royal W.
        • Gelfand J.M.
        • et al.
        Definition and consensus diagnostic criteria for neurosarcoidosis: from the Neurosarcoidosis Consortium Consensus Group.
        JAMA Neurol. 2018; 75: 1546-1553
        • Garcia G.-G.P.
        • Lavieri M.S.
        • Jiang R.
        • Mcallister T.W.
        • Mccrea M.A.
        • Broglio S.P.
        A data-driven approach to unlikely, possible, probable, and definite acute concussion assessment.
        J Neurotrauma. 2019; 36: 1571-1583
        • McAllister T.
        • McCrea M.
        Long-term cognitive and neuropsychiatric consequences of repetitive concussion and head-impact exposure.
        J Athl Train. 2017; 52: 309-317
        • Yue J.K.
        • Yuh E.L.
        • Korley F.K.
        • et al.
        Association between plasma GFAP concentrations and MRI abnormalities in patients with CT-negative traumatic brain injury in the TRACK-TBI cohort: a prospective multicentre study.
        Lancet Neurol. 2019; 18: 953-961
        • Manley G.T.
        • Mac Donald C.L.
        • Markowitz A.J.
        • et al.
        The Traumatic Brain Injury Endpoints Development (TED) initiative: progress on a public-private regulatory collaboration to accelerate diagnosis and treatment of traumatic brain injury.
        J Neurotrauma. 2017; 34: 2721-2730
        • Meares S.
        • Shores E.A.
        • Taylor A.J.
        • et al.
        The prospective course of postconcussion syndrome: the role of mild traumatic brain injury.
        Neuropsychology. 2011; 25: 454-465
        • Bernard C.O.
        • Ponsford J.A.
        • McKinlay A.
        • McKenzie D.
        • Krieser D.
        Predictors of post-concussive symptoms in young children: Injury versus non-injury related factors.
        J Int Neuropsychol Soc. 2016; 22: 793-803
        • Mclean S.A.
        • Kirsch N.L.
        • Tan-Schriner C.U.
        • et al.
        Health status, not head injury, predicts concussion symptoms after minor injury.
        Am J Emerg Med. 2009; 27: 182-190
        • Hoge C.W.
        • McHurk D.
        • Thomas J.L.
        • Cox A.L.
        • Engel C.C.
        • Castro C.A.
        Mild traumatic brain injury in U.S. soldiers returning from Iraq.
        N Engl J Med. 2008; 358: 453-463
        • Huang J.
        • Gates A.J.
        • Sinatra R.
        • Barabási A.L.
        Historical comparison of gender inequality in scientific careers across countries and disciplines.
        Proc Natl Acad Sci U S A. 2020; 117: 4609-4616
        • Witteman H.O.
        • Hendricks M.
        • Straus S.
        • Tannenbaum C.
        Are gender gaps due to evaluations of the applicant or the science? A natural experiment at a national funding agency.
        Lancet. 2019; 393: P531-P540
        • Bohren M.A.
        • Javadi D.
        • Vogel J.P.
        Gender balance in WHO panels for guidelines published from 2008 to 2018.
        Bull World Health Organ. 2019; 97: 477-485
        • Nielsen M.W.
        • Alegria S.
        • Börjeson L.
        • et al.
        Gender diversity leads to etter science.
        Proc Natl Acad Sci U S A. 2017; 114: 1740-1742