If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Corresponding author Sean R Smith, MD, Department of Physical Medicine and Rehabilitation, University of Michigan, 325 E Eisenhower Pkwy, Ste 100, Ann Arbor, MI 48108.
George Mason University, Fairfax, VirginiaDepartment of Physical Medicine and Rehabilitation, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania
To develop an item response theory (IRT)–calibrated, patient-reported outcome measure (the PROMIS Cancer Function Brief 3D Profile) of physical function, including associations with fatigue and social participation, in cancer rehabilitation patients.
Design
Large-scale field testing, graded response model IRT analyses, and multivariate regression analysis.
Setting
Six cancer rehabilitation clinics associated with cancer centers across the United States.
Participants
Adults (N=616) treated in outpatient cancer rehabilitation medicine clinics.
Intervention
Not applicable.
Main Outcome Measures: The PROMIS(r) Cancer Function 3D Profile (including existing items from PROMIS(r) item banks).
Results
A total of 616 patients completed 21 items in the initial item pool. Nine items were removed because of comparatively lower information that they provide according to the IRT item calibrations, low item-total correlations, or bimodal distributions. The remaining items generated a 12-item short form. Regression analyses determined that the items were responsive to and representative of the patient population across trait ranges and multiple domains and subdomains of function.
Conclusions
This psychometric investigation supports the use of the PROMIS Cancer Function Brief 3D Profile for evaluating function in outpatient cancer rehabilitation patients.
A significant portion of the estimated 17 million cancer survivors in the United States will experience impaired function as a result of their disease and its treatment.
There are many causes for this deficiency, but one that is potentially addressable in the near term is the lack of a scalable and accessible means of assessing physical and social function to align care with patients’ symptoms and goals.
Implementing personalized pathways for cancer follow-up care in the United States: proceedings from an American Cancer Society–American Society of Clinical Oncology summit.
Accordingly, it is essential to develop psychometrically sound measurement tools that accurately reflect real-world clinical outcomes, are not burdensome to the patient, are cost effective, and do not add measurably to the workload of an already overstretched workforce. Such an instrument could harmonize assessment across multiple institutions and improve the personalization of treatment. Unfortunately, existing measures of function in cancer patients fall short in a number of ways. For example, the PROMIS–Cancer Bank v1.1–Physical Function consists of 45 questions, making it burdensome for patients and of unclear value to providers. Other measures such as the Functional Assessment of Cancer Therapy–General ask patients for information not directly modifiable with rehabilitation, such as nausea and whether family members have accepted a patient’s illness. In addition, many measures do not capture both physical function and related domains and subdomains amenable to rehabilitation, such as fatigue.
The Cancer Rehabilitation Medicine Metrics Consortium (CRMMC) was formed to address the need for pragmatic, easy-to-administer outcome measures relevant to the delivery of cancer rehabilitation service.
The group’s goal was to create an instrument with high applicability across the range of outpatient cancer rehabilitation medicine practices. To meet these criteria, the CRMMC considered extant legacy and item response theory (IRT)-modeled patient reported outcome measures (PROs). Because multiple institutions in the CRMMC were advancing Patient Reported Outcome Measurement Information System (PROMIS)-based monitoring strategies, the decision was made to use PROMIS tools. Advantageous features of PROMIS for use in cancer rehabilitation settings include its applicability to outpatient populations, familiarity among oncology providers, availability in the public domain, and options for either computer adaptive testing (CAT), generic short form (SF), or profiles consisting of multiple SFs.
However, neither CAT nor existing SF approaches to item selection and administration were deemed an optimal match for the CRMMC’s goals for a number of reasons. Subdomain coverage is not specified in the PROMIS CAT and SF algorithms; thus patients may complete CAT without responding to the items of greatest relevance to cancer rehabilitation providers. The CRMMC therefore elected to create a novel PROMIS measure that included subdomain coverage (eg, fine motor function and fatigue) in trait ranges that are pertinent to the assessment and treatment of disability across cancer populations.
The CRMMC completed a modified Delphi process to select the domains for assessment and candidate items within each domain. In particular, the physical function, fatigue, and social roles ability domains were chosen given their empirical association with disablement experienced by cancer survivors.
Psychological symptoms, social outcomes, socioeconomic attainment, and health behaviors among survivors of childhood cancer: current state of the literature.
Participants in this process were the physician authors (SS, MV, DZ, SS, LG, MH, AC) of this manuscript and 2 more from institutions with robust cancer rehabilitation programs but who did not participate in data collection. The IRT item information properties for each bank were reviewed, and the CRMMC identified discriminating items in the target trait ranges. Those believed to be most relevant to the management of disability among patients with cancer were selected, resulting in 21 candidate items for clinical testing.
Although the Delphi method is an accepted and successful consensus process for developing functional assessment tools,
The comprehensive ICF core sets for spinal cord injury from the perspective of physical therapists: a worldwide validation study using the Delphi technique.
Content validity of the comprehensive international classification of functioning, disability and health (ICF) core set for low back pain from the perspective of physicians: a Delphi survey.
the measurement properties of such instruments remain unknowable without clinical testing. This study reports the results of the empirical, multisite process used to field test the 21 candidate items to create a low-burden 12-item assessment tool.
Methods
Study sample
A convenience sample was recruited among outpatient cancer rehabilitation clinics at 6 participating institutions. Inclusion criteria were having a history of cancer and age of 18 years or older. There were no limitations regarding demographic characteristics, disease type, treatment, stage, or duration of survivorship. Patients with cognitive or communication limitations that could interfere with the ability to complete the questionnaire, including lack of English proficiency, were excluded.
The institutional research boards of each participating site approved the study, and patients were required to provide written consent if required by the Institutional Research Boards where the data were collected. The University of Michigan was the coordinating site and was responsible for monitoring data quality, securely storing the data, and coordinating data use agreements.
Data collection and management
Data were collected from patients and their treating physiatrists at initial consultations or follow-up visits. Patients completed the study questionnaire in the waiting area or examination room prior to seeing the physiatrist. Patients were assigned a study identification number and those with multiple clinic appointments during the data collection period were invited to complete the questionnaire at each visit.
Demographic and clinical data including age, sex, type of cancer, treatment history, and disease stage were abstracted from the participant’s electronic health records and recorded either during or immediately after patient visits. Additional characteristics were recorded given their association with disablement, including body mass index (BMI) and the presence of bone or central nervous system metastases, polyneuropathy, and noncancer comorbidities that might affect function (musculoskeletal, neurologic, cardiopulmonary, psychiatric).
The 21 items were administered using computer-interfaced tablets or paper forms, depending on site resources. Patient and clinician responses were captured in a REDCapa database.
Investigators who collected responses via paper forms manually entered results into the database; tablet-based responses were automatically recorded. Data were exported as Excelb files and merged in STATA v.15.0c after collection was completed.
Candidate SF items
Among the candidate items, 11 assessed physical function with a combination of mobility and activities of daily living questions, 5 assessed fatigue, and 5 assessed ability to participate in social roles and activities items (appendix 1). The items spanned a wide trait range, appeared broadly applicable across diverse cancer populations, and had excellent IRT item information characteristics. Items were selected from the entire PROMIS item bank and not specific SFs; they are named based on existing PROMIS IDs.
Before inclusion, the reading level of candidate items was analyzed using The Flesch-Kincaid Grade Level formula. In addtion, the approximate metabolic equivalent level of each physical function item was calculated using the Compendium of Physical Activities
to confirm that the pool of selected items captured a reasonable range of performance intensity.
Pain and distress numerical rating scales
Participants rated their pain and distress on 11-point numeric rating scales. Returning patients also reported a perceived general rating of change item. The distress scale was adapted from the National Comprehensive Cancer Network’s Distress Thermometer.
scales. Physicians also completed a perceived rate of change item for returning patients.
Statistical analysis
Descriptive statistics were used to characterize the study population’s demographic and clinical characteristics. Means and standard deviations (SDs) were calculated for continuous variables and proportions for binary and categorical variables.
Item information characteristics
The development and calibration of PROMIS item banks used both general and diseased populations.
Psychometric evaluation and calibration of health-related quality of life item banks: plans for the Patient-Reported Outcomes Measurement Information System (PROMIS).
PROMIS measures are reported on a T score metric (mean, 50; SD, 10) and anchored to the US general population to enable easy interpretation, comparison, and generalizability across normal and diseased populations. The PROMIS item banks were calibrated using the graded response model. We used the item parameter calibrations published from the PROMIS item bank to obtain the item information characteristic curves. In addition, we ran one model for each of the 3 constructs for IRT scoring using published item parameters from the item banks and scored those who responded to at least 50% of the items. Densities of the IRT scores were plotted. The flexMIRT v2.0 softwared was used for the IRT scoring. Item information curves and density plots were compared to characterize trait coverage and informational yield in the target population.
Anchor-based analyses
Linear, logistic, and ordinal logistic regression models were constructed to assess which of the candidate items discriminated as expected across anchoring variables. Participants’ item responses were included as indicator variables in the models. Dependent variables included KPS, ECOG, BMI, presence of active cancer, and presence of distant metastases. The regression modeling approach was linear for KPS, ordinal logistic for ECOG and BMI, and logistic for presence of active cancer and distant metastases. Models were adjusted for correlation at the participant-level because multiple participants provided more than one data point. Conventional goodness-of-fit statistics were estimated. All analyses were performed in STATA.
Final item selection
The IRT information curves were reviewed for trait coverage and information content. Lesser performing, redundant items, and those with low information content were culled. In a second selection step, CRMMC members reviewed the output from the regression analyses to identify which candidate items strongly associated with related clinical constructs (eg, physical function with KPS). The group also reviewed which items had a stepwise increase in the coefficients of the ordered response options (never [5], rarely [4], sometimes [3], often [2], and always [1]) and which had the broadest coefficient span. The CRMMC weighted the KPS and ECOG models more heavily in decision making because these constructs are more strongly correlated with physical function than BMI or the presence of active disease or metastases. Consensus was reached through group discussion regarding which items to retain in the final assessment tool.
Results
Study sample
A total of 616 unique patients were enrolled across the 6 sites, comprising 1009 assessment points. The PRO typically took patients less than 5 minutes to complete. Patient demographic and clinical characteristics are listed in table 1. Breast cancer was the most common diagnosis (42.5% of the sample), with women comprising two-thirds of the study participants. The average age was 59.4 years (SD, 14.1y). Just over half of the participants had no evidence of disease (56.9%) at the time of evaluation. Nearly all received surgery, chemotherapy, or radiation therapy prior to assessment. Overall, the sample had a high level of physical performance. Only 20% required an assistive device for ambulation and two-thirds had ECOG performance scores of 0 (fully active, able to carry on all predisease performance) or 1 (restricted in physically strenuous activity). Table 2 includes performance site characteristics and breaks down enrollment by cancer diagnosis.
We plotted item information curves using the item parameters provided by PROMIS item banks as well as the score density plots (fig 1). Score summary statistics are presented in table 3. For each item, the x-axis of the item information curves shows trait ranges and the y-axis shows the amount of information at each point of the trait. For physical function items (n=1004), PFC45, PFA16, PFA17, and PFB18 had low peak of information (y-axis). PFA23, PFA21, and PFB12 had peak of information of 3 or more and trait range between –2 and 1, indicating that these items captured lower levels of function. PFB24, PFC12, and PFA4 had a trait range between –1 and 2 and were therefore better at capturing higher levels of function.
Fig 1Score distribution of the sample and the item informative curves (ICCs).
All 5 fatigue (n=996) items had similar levels of peak information. FATIMP30 and FATEXP22 captured higher levels of fatigue than others, although in general, the items had similar trait ranges and measured severe symptoms better than mild symptoms. Only 5.3% of the respondents scored below the population average.
Finally, items regarding the ability to participate in social roles and activities (n=982) revealed that 76.2% of respondents scored lower than the population average. All except SRPPER16 had high peak of information levels, and all items had a similar trait range, favoring lower levels of social participation.
Items were culled based on IRT analysis if they provided a lesser degree of information or if the level of information provided was considerably inconsistent across the trait range (eg, if an item performed poorly at the low end of a trait but better at the high end). The CRMMC prioritized having some items that performed better at higher or lower trait ranges as long as the item did not perform exceedingly poorly at one end. When items provided comparable levels of information and trait coverage, the CRMMC examined their subdomain coverage. Items were prioritized for inclusion if they were relevant to an important subdomain that did not overlap with other items or if the activity described in the stem was clinically relevant. For example, the ability to reach into a high cupboard addresses upper extremity and shoulder function, a highly relevant construct to survivors of breast and head and neck cancers. Items that were presumptively culled on the basis of their IRT information curves included PFB12, PFA17, PFB18, PFC12, FATEXP22, SRPPER16, and SRPPER46. Items potentially fit for inclusion in the final SF based on IRT analysis and clinical judgment, before regression analysis, were PFA23, PFA21, PFB24, PFC45, PFA16, PFA42, PFC12, FATIMP30, FATIMP51, FATIMP18, FATIMP9, SRPPER11, SRPPER18, and SRPPER23.
Anchor-based assessments and validity
Review of the regression models was used principally as a “tie breaker” when 2 items offered comparable information and subdomain coverage relevant to cancer survivors. Output from the regression models is listed in table 4.
Table 4Results of regression analyses
Item
R2
Pseudo R2
Prob(F)
Response Option Coefficient
2
3
4
5
PFA23—Are you able to go for a walk of at least 15 minutes?
SRPPER46—I have trouble doing all of the activities with friends that I want to do
KPS
0.30
0.00
7.60
12.54
20.07
22.68
ECOG
0.13
0.00
–1.26
–2.27
–3.33
–4.18
BMI
0.00
0.05
–0.02
–0.01
–0.23
–0.61
Active Cancer
0.02
0.00
0.51
0.58
0.35
0.21
Metastatic
0.04
0.00
0.48
0.66
0.43
0.07
NOTE. Items are named by the PROMIS item IDs. Items were scored based on participant responses compared to KPS, ECOG, BMI, presence of active cancer, and presence of metastatic cancer. R2 was calculated with KPS because of it being a linear scale of 11 possible responses. Pseudo-R2 was calculated for the other responses due to… Coefficients related the relationship between function (higher numbers indicate higher patient-reported function) and the variables.
KPS and ECOG had strong associations between the 21 candidate items, after removing PFB18 (“Are you able to shave your face or apply make-up?”), as reflected in R2 values of 0.29 to 0.40 for KPS and pseudo R2 values of 0.12 to 0.21. Pseudo R2 values were estimated with ordinal logistic regression and were therefore lower than the R2 values estimated with linear regression. The Prob (F), the probability that the null hypothesis is true for the model, was <.001 for all models. With few exceptions, coefficient P values for the response options were also <.001.
Although the regression model output was principally used to refine SF item selection, the high R2 and pseudo R2 values of the physical function and social role ability items in the KPS and ECOG models support the convergent validity of these domains in the final SF.
BMI had the lowest association with participants’ responses. Although the Prob (F) was <.001 for all models, pseudo R2 values ranged from <.001 to .2, excepting PFA21 (“Are you able to go up and down stairs at a normal pace?”), which had a pseudo R2 value of 0.24.
Presence of active or metastatic cancer had more variable P values and response coefficients were inconsistently associated with increasing trait severity for some items. Pseudo R2 values for these models were generally low (<.001-0.04).
Final item selection
In the event that multiple items offered comparable information and had similar regression model output, the group used a consensus process to determine the collection of high performing items that optimized subdomain coverage, provided clinically actionable information, and were not redundant. For example, both PFA21 (“Are you able to go up and down stairs at a normal pace?”) and PFB24 (“Are you able to run a short distance, such as to catch a bus?”) performed similarly on IRT and regression analyses, captured a high trait range, and associated strongly with performance status. PFB24 was ultimately selected because stair climbing is a key subdomain of physical function and has clear rehabilitation implications. Table 5 lists the final 12-item SF.
Table 5Final, 12-item SF
Domain
Item
Response Categories
Physical function
Are you able to go for a walk of at least 15 min?
Without any difficulty (5), with a little difficulty (4), with some difficulty (3), with much difficulty (2), unable to do (1)
Are you able to go up and down stairs at a normal pace?
Are you able to sit on and get up from the toilet?
Are you able to dress yourself, including tying shoelaces and buttoning your clothes?
Are you able to carry a laundry basket up a flight of stairs?
Does your health now limit you in doing heavy work around the house like scrubbing floors, or lifting or moving heavy furniture?
Not at all (5), a little bit (4), somewhat (3), quite a lot (2), cannot do (1)
Fatigue
In the past 7 d, how often did you have trouble finishing things because of your fatigue?
Never (1), rarely (2), sometimes (3), often (4), always (5)
In the past 7 d, how often did your fatigue make it difficult to plan activities ahead of time?
In the past 7 d, how often did you have to limit your social activities because of your fatigue?
Social participation
I have trouble doing all of my regular leisure activities with others.
Never (5), rarely (4), sometimes (3), often (2), always (1)
I have trouble doing all of the work that is really important to me (including work at home).
I have trouble doing all of the family activities that I want to do.
IRT modeled instruments can be scored using model estimates or through simplified score conversion tables. The former require that item responses be stored on a computing platform -programmed with the IRT model output. The CRMMC believed that this requirement would severely constrain use of the SF and, therefore, chose to work with the Outcomes and Measurement Science Hub within the Department of Medical Social Sciences, Northwestern Feinberg School of Medicine, to develop a score conversion chart using official PROMIS calibrations (appendix 2)
Discussion
To the best of our knowledge, the PROMIS Cancer Function Brief 3D Profile is the first specialized SF created from PROMIS item banks to meet the unique assessment and clinical decision-making needs of rehabilitation providers who treat cancer patients. We believe that the development of the PROMIS Cancer Function Brief 3D Profile is important, as it has high information density and requires minimal time for patients to complete.
Specific and consistent subdomain coverage in PRO assessments is required to individualize management and ensure that providers receive reliable information regarding a patient’s function. Extant PROMIS CAT and SF tools either do not provide consistent (CAT) or specific (generic SFs) subdomain coverage, whereas the PROMIS Cancer Function Brief 3D Profile does. In addition, within a given domain such as fatigue, scores from this measure are comparable to those derived using PROMIS CATs and SFs, as all include IRT-calibrated items on a common unidimensional continuum. Because domain scores are comparable, data collected can track the trajectory of a patient’s longitudinal progress while providing information needed by diverse provider groups, primary and specialist, to inform clinical decision-making.
The resulting 12-item SF met the goals the CRMMC set out to accomplish. It assesses key subdomains in trait ranges relevant to cancer survivors seen in outpatient clinics. Its developmental process was comprehensive, evidence-based, and used expert opinion to leverage vetted and highly applicable PROMIS items into a useful, short PRO tool. Furthermore, given its low respondent burden, many patients were enrolled and a high percentage of patients completed the items. Because administration and data capture did not significantly interfere with the workflow of study physicians, the PROMIS Cancer Function Brief 3D Profile is feasible to collect in busy outpatient clinics. Given the different performance sites and modes of collecting data for this study, the feasibility of this profile is likely generalizable to most practice settings.
The benefits of this process are borne out in the study results, wherein the selected PROMIS items performed well vis-à-vis anchors (KPS and ECOG) and relevant clinical characteristics. In addition, it is the only SF that we know of to be developed and assessed in multiple cancer rehabilitation medicine programs, which minimizes sample bias and ensures utility in various outpatient practices. Although a large proportion of the study cohort were women with breast cancer, this weighting is consistent with supportive care clinical trials and representative of cancer rehabilitation care worldwide.
Study limitations
Limitations include that data were collected from clinics affiliated with large cancer centers. Therefore, the PROMIS Cancer Function Brief 3D Profile’s value in the community oncology setting is less clear. Furthermore, the tool was developed to err on the side of low respondent burden to increase the likelihood that the instrument would be administered, completed, and incorporated in clinical decision-making. Despite efforts to ensure comprehensive subdomain coverage, it is possible that functional information may be sacrificed for brevity. In addition, information was not collected regarding participants’ race, ethnicity, educational level, or socioeconomic status, so future PROMIS Cancer Function Brief 3D Profile assessments should evaluate its performance across minority and vulnerable subgroups. The fact that nonphysician rehabilitation service providers and patient representatives did not participate in the development of this tool is also a limitation. However, the instrument was constructed to support clinical assessment and decision-making by rehabilitation physicians. The CRMMC is hopeful that physical therapists, occupational therapists, and other providers will trial this tool to determine its validity in other settings.
In the future, comparison of this SF with existing measures may further support its role in the evaluation of functional status in cancer patients. Future investigations should also explore its use toward triaging patients to cancer rehabilitation care.
Conclusions
The results of IRT analysis, multivariate regression analyses, and expert evaluation of the items that were field-tested produced a new 12-item assessment tool measuring physical function, fatigue, and social participation in patients with a history of cancer. The results support use of the PROMIS Cancer Function Brief 3D Profile for evaluating function in patients with a history of cancer.
Suppliers
a.
REDCap; Vanderbilt University.
b.
Excel; Microsoft Corp.
c.
STAT, v 15.0; STATACorp LLC.
d.
flexMIRT, v 2.0; Vector Psychometric Group.
Acknowledgments
The authors thank Drs. Eric Wisotzky, MD, and Chris Custodio, MD, for their contributions to conceptualizing the assessment tool.
Supplementary Data
Appendix 1The Original 21 PROMIS Items Administered to Patients
Domain
Item
Response Categories
Physical function
Are you able to go for a walk of at least 15 min? (PFA23)
Without any difficulty (5), with a little difficulty (4), with some difficulty (3), with much difficulty (2), unable to do (1)
Are you able to go up and down stairs at a normal pace? (PFA21)
Are you able to run a short distance, such as to catch a bus? (PFB24)
Are you able to sit on and get up from the toilet? (PFC45)
Are you able to dress yourself, includ ing tying shoelaces and buttoning your clothes? (PFA16)
Are you able to carry a laundry basket up a flight of stairs? (PFA42)
Are you able to make a bed, including spreading and tucking in bed sheets? (PFB12)
Are you able to reach into a high cupboard? (PFA17)
Are you able to shave your face or apply make-up (PFB18)
Does your health now limit you in doing two hours of physical labor? (PFC12)
Not at all (5), a little bit (4), somewhat (3), quite a lot (2), cannot do (1)
Does your health now limit you in doing heavy work around the house like scrubbing floors, or lifting or moving heavy furniture? (PFA4)
Fatigue
In the past 7 d, how often were you too tired to think clearly? (FATIMP30)
Never (5), rarely (4), sometimes (3), often (2), always (1)
In the past 7 d, how often did you have trouble finishing things because of your fatigue? (FATIMP51)
In the past 7 d, how often did you have to limit your social activities because of your fatigue? (FATIMP18)
In the past 7 d, how often were you bothered by your fatigue? (FATEXP22)
In the past 7 d, how often did your fatigue make it difficult to plan activities ahead of time? (FATIMP9)
Social participation
I have trouble doing all of my regular leisure activities with others. (SRPPER11)
Never (5), rarely (4), sometimes (3), often (2), always (1)
I have to do my work for shorter periods of time than usual. (SRPPER16)
I have trouble doing all of the family activities that I want to do. (SRPPER18)
I have trouble doing all of the work that is really important to me (including work at home). (SRPPER23)
I have trouble doing all of the activities with friends that I want to do. (SRPPER46)
Implementing personalized pathways for cancer follow-up care in the United States: proceedings from an American Cancer Society–American Society of Clinical Oncology summit.
Psychological symptoms, social outcomes, socioeconomic attainment, and health behaviors among survivors of childhood cancer: current state of the literature.
The comprehensive ICF core sets for spinal cord injury from the perspective of physical therapists: a worldwide validation study using the Delphi technique.
Content validity of the comprehensive international classification of functioning, disability and health (ICF) core set for low back pain from the perspective of physicians: a Delphi survey.
Psychometric evaluation and calibration of health-related quality of life item banks: plans for the Patient-Reported Outcomes Measurement Information System (PROMIS).
00133-7&id=gr1a.jpg){kind=link}
00133-7&id=gr1b.jpg){kind=link}
00133-7&id=gr1c.jpg){kind=link}
00133-7&id=gr1d.jpg){kind=link}