Advertisement

Participatory Action Research Designs in Applied Disability and Rehabilitation Science: Protecting Against Threats to Social Validity

Published:November 22, 2012DOI:https://doi.org/10.1016/j.apmr.2012.07.033

      Abstract

      Researchers and disability advocates have been debating consumer involvement in disability and rehabilitation science since at least 1972. Despite the length of this debate, much confusion remains. Consumer involvement may represent a spirit of democracy or even empowerment, but as a tool of science, it is necessary to understand how to judge its application. To realize consumer involvement as a design element in science, researchers need a framework for understanding how it can contribute to the scientific process. The thesis of this article is that a primary scientific function of consumer involvement is to reduce threats to the social validity of research, the extent to which those expected to use or benefit from research products judge them as useful and actually use them. Social validity has traditionally not been treated with the same rigor as concerns for internal and external validity. This article presents a framework that describes 7 threats to social validity and explains how 15 forms of consumer involvement protect against those threats. We also suggest procedures for reporting and reviewing consumer involvement in proposals and manuscripts. This framework offers tools familiar to all scientists for identifying threats to the quality of research, and for judging the effectiveness of strategies for protecting against those threats. It may also enhance the standing of consumer involvement strategies as tools for protecting research quality by organizing them in a way that allows for systematic criticism of their effectiveness and subsequent improvement.

      Keywords

      List of abbreviations:

      CEI (Community Engagement Initiative), CIL (center for independent living), PAR (participatory action research)
      The purpose of applied research in disability and rehabilitation is to develop empirically derived solutions to problems experienced by people with disabilities.
      • Gray D.
      Disability and rehabilitation research from policy to program: a personal perspective.
      National Institute on Disability and Rehabilitation Research
      PAR sourcebook: forging collaborative partnerships in the study of disability.
      National Institute on Disability and Rehabilitation Research
      The new paradigm on disability: research issues and approaches.
      Researchers apply a wide range of scientific methods to develop solutions that may include mechanical and electrical technologies, medical and pharmacologic treatments, and behavioral and social technologies (including laws, policies, programs, and treatment techniques). To be considered truly successful, findings from the research must actually be used.
      Over the past 30 years, researchers and advocates have debated the role of people with disabilities in the conduct of disability and rehabilitation science.
      • Brown S.C.
      Methodological paradigms that shape disability research.
      • DeJong G.
      Independent living: from social movement to analytic paradigm.

      National Institute on Disability and Rehabilitation Research (2005). Long-range plan for fiscal years 2005-2009. Available at: http://www.ed.gov/legislation/FedRegister/other/2006-1/021506d.pdf. Accessed February 16, 2012.

      • Fenton J.
      • Batavia A.
      • Roody D.S.
      Proposed policy statement for NIDRR on constituency-oriented research and dissemination (CORD).
      • Nelson G.
      • Ochocka J.
      • Griffin K.
      • Lord J.
      Nothing about me, without me: participatory action research with self-help/mutual and organization for psychiatric survivors.
      • Tewey B.P.
      Building participatory action research partnerships on disability and rehabilitation research.
      • White G.W.
      Consumer participation in disability research: the golden rule as a guide for ethical practice.
      • White G.W.
      • Nary D.E.
      • Froehlich A.K.
      Consumers as collaborators in research and action.
      This debate has tended to focus on the concepts of participatory action research (PAR).
      • Campbell M.
      Continuum of participatory research: models, applications and implementation mechanisms.
      • White G.W.
      • Suchowierska M.
      • Campbell M.
      Developing and systematically implementing Participatory Action Research.
      The broader fields of medicine and health focus on concepts of patient-centered outcomes research and community-based participatory research.
      Task Force on the Principles of Community Engagement
      Principles of community engagements.
      While these discussions have been useful, researchers need a structured framework to realize consumer involvement as a design element in science.
      • Viswanathan M.
      • Ammerman A.
      • Eng E.
      • et al.
      Community-based participatory research: assessing the evidence.
      The thesis of this article is that a primary scientific function of consumer involvement is to reduce threats to the social validity of research, the extent to which potential adopters of research products judge them as useful and actually use them. This article briefly reviews the history of consumer involvement in disability and rehabilitation research, introduces and defines the concept of social validity, compares social validity with internal and external validity, and describes threats to social validity. Importantly, it presents a framework for judging the extent to which different forms of consumer involvement protect against threats to the social validity of research. It also reviews several forms of consumer involvement as a means of explicating the scientific function of consumer involvement.

      History of consumer involvement

      The idea of involving end users in research is neither new nor unique to disability and rehabilitation science. The agricultural research and extension service has long involved a partnership among farmers, ranchers, agricultural workers, field agents, and researchers. Rogers and Shoemaker
      • Rogers E.M.
      • Shoemaker E.F.
      Communication of innovations: a cross-cultural approach.
      report on the lessons learned by agricultural researchers in the 1920s when they did not involve farmers in the development and dissemination of hybrid seed corn. Early on, it was tested in the Southwestern United States. Despite the increased yield in corn, farmers quickly rejected it. It seems that the corn meal produced from processing the corn was not suitable for making tortillas. Research and development had indeed produced a more efficient variety of corn, but it produced flour that had limited value to intended consumers. From this and similar experiences, agricultural researchers began to consider consumer concerns in research, development, and dissemination.
      A recognized parallel in disability and rehabilitation science is the abandonment of assistive technology by people with disabilities.
      • Batavia A.I.
      • Hammer G.S.
      Toward the development of consumer-based criteria for the evaluation of assistive devices.
      • Burke R.K.
      Disuse of assistive technology by vocational rehabilitation and independent living clients in North Carolina.
      • Phillips B.
      • Zhao H.
      Predictors of assistive technology abandonment.
      • Riemer-Reiss M.L.
      • Wacker R.R.
      Factors associated with assistive technology discontinuance among individuals with disabilities.
      • Verza R.
      • Carvalho M.L.
      • Battaglia M.A.
      • Uccelli M.M.
      An interdisciplinary approach to evaluating the need for assistive technology reduces equipment abandonment.
      There are numerous reports of elegant assistive devices that had been demonstrated as effective and reliable by engineers, but which were discarded because they did not meet consumers' needs. This has led rehabilitation engineers to call for the involvement of consumers in the development of assistive technologies.
      • Newell A.
      User-centered design and disability.
      The involvement of consumers in disability and rehabilitation research began at least by the early 1970s.

      Seekins T. Consumer involvement in rehabilitation research over the past thirty years. In: Annual conference of the National Association of Rehabilitation Research and Training Centers; 1992; Washington (DC).

      For example, Remmes
      • Remmes H.S.
      Consumer involvement in rehabilitation.
      presented a paper at the 1972 meeting of the National Association of Rehabilitation Research and Training Centers in which he stated that people with disabilities want to participate in any decision-making process that affects their lives, and suggested that consumers could become a positive force in rehabilitation research.
      In the late 1970s, the emergence of the independent living movement, with its roots in civil rights and consumerism, asserted that people with disabilities should control all aspects of their lives, from medical care to employment choices.
      • Bassman R.
      Whose reality is it anyway? Consumers/survivors/ex-patients can speak for themselves.
      • Humphries K.
      • Rappaport J.
      Research self-help/mutual and groups and organizations: many roads, one journey.
      By extension, this included controlling research that used resources targeted at issues of importance to them.
      The literature on consumer involvement in disability and rehabilitation science has grown steadily since then.
      • Campbell M.
      • Seekins T.
      Whatever happened to PAR? Methodological and administrative challenges of implementing participatory research in applied rehabilitation science.
      A particularly important contribution to the debate was the identification of PAR as a means of conveying a spirit of such involvement and possible methods for achieving it.
      • Whyte W.F.
      Participatory action research.
      Still, confusion remains about the nature and purpose of PAR; some argue that it is a philosophy, others that it is a method, and still others that it is a grab-bag of procedures, and, of course, some argue for its use and others against it. Given the importance of consumer involvement to the funding and practice of disability and rehabilitation science, efforts to clarify its purpose and characteristics are warranted.
      • DeJong G.
      Independent living: from social movement to analytic paradigm.

      Social validity

      Wolf
      • Wolf M.M.
      Social validity: the case for subjective measurement or how applied behavior analysis is finding its heart.
      defined social validity as the extent to which potential adopters of research results and products judge them as useful and actually use them. Social validity involves judgments of the importance of research goals, the acceptability of procedures, and the significance of impact by those expected to use its results or to benefit from them. Wolf
      • Wolf M.M.
      Social validity: the case for subjective measurement or how applied behavior analysis is finding its heart.
      argued that when behavioral researchers attend to the social validity of their research, the probability that their research would be supported by the public or be used to solve problems increases.

       Threats to social validity

      A primary concern in the scientific study of behavioral and social phenomenon, as well as medicine and engineering, has been the internal and external validity of the research results.
      To incorporate consumer involvement as a design element in science, researchers need tools for assessing threats to social validity and assessing the effectiveness of strategies for protecting against those threats. Such tools are needed by those who review research proposals in the design stage (eg, human subject protection committees), by those who formally review proposals to make funding recommendations, and by journal editors and peer reviewers when reviewing research reports for publication. They are also useful for policy-makers, program managers, and professional service providers who are considering the implementation of a finding, and for consumers judging whether to adopt a new approach.

       Framework for assessing protection against threats to social validity

      Campbell and Stanley's
      • Campbell D.T.
      • Stanley J.C.
      Experimental and quasi-experimental designs for research.
      classic description of and rules for judging internal and external validity of experimental and quasi-experimental research designs provides a framework for organizing and evaluating social validity. Table 1 lists 15 commonly used consumer involvement procedures and assesses the degree to which they can help protect research against 7 distinct threats to social validity. First, several threats to social validity are explained with selected examples of how consumer involvement helps protect against the threat. Next, several forms of consumer involvement are described and their method of protection explained.
      Table 1Levels of protection against threats to social validity offered by various forms of consumer involvement
      Forms of Consumer InvolvementIrrelevance of Issue or Problem SelectedLack of Clarity About Important GoalsMisunderstanding of Acceptability of Research MethodsMisunderstanding About Range of Acceptable Intervention Approaches or ProceduresFailure to Consider Adopter Criteria for SuccessMisinterpretation of ResultsGenerality of Findings in Real-Life Applications
      Preinvolvement
       Theory-derived data mining or methods researchNoneNoneNoneNoneNoneNoneNone
       People with disabilities as data pointsNoneNoneNoneNoneNoneNoneNone
       Advisory committeesLowLowMediumLowLowLowLow
      True PAR
       Emancipatory researchHighHighHighHighHighHighLow
       Organizational developmentHighHighHighHighHighHighMedium
       Community development action researchHighHighMediumMediumHighMediumMedium
      Quasi-PAR
       Focus groupsMediumMediumMediumMediumMediumMediumLow
       Researcher translationMediumMediumLowLowLowLowLow
       Agenda setting surveysHighHighLowMediumMediumNoneHigh
       Best practices researchLowMediumNoneLowLowLowMedium-high
       Research teamsMediumHighHighMediumHighLowMedium
       Researcher with disabilityHighHighHighMediumMedium-highLow-mediumMedium
       Consumer consultantsMediumMediumHighMedium-highMedium-highMediumMedium
       Product championsNANANAMediumMediumHighHigh
       Partners in advocacyHighHighMediumHighHighHighHigh
      Abbreviation: NA, not applicable.

       Threats to the social validity of applied research

      As with threats posed to internal and external validity of research designs, there are numerous threats that challenge the social validity of applied research. These include threats posed by: (1) selecting irrelevant topics for research, (2) a lack of clarity about important consumer goals, (3) misunderstanding the acceptability of research methods, (4) misunderstanding the range of intervention acceptability, (5) ignoring criteria that potential adopters would use to judge the significance of outcomes and impacts, (6) misinterpreting results, and (7) lacking generality of findings in real-life application. Table 1 lists these threats and summarizes the degree to which different forms of consumer involvement may help protect against those threats. The following sections describe the column headings for table 1.

       Threats posed by selecting irrelevant issues

      The first threat to social validity involves selecting an issue or problem for research that lacks importance or relevance to a constituency. Many researchers are interested primarily in theoretic, methodologic, or measurement issues. For research with such a focus, the actual content of study may be arbitrary except to the extent that it provides a heuristic mechanism for developing or testing methods and measures. It may be as useful to such researchers who are studying time-sampling procedures to study the sitting behavior of an office secretary as to examine the study behavior of a child with a learning disability. In fact, for some purposes, the selection of a secretary's sitting behavior may be more convenient and provide a better opportunity to study the issue (ie, observation procedures) of interest to them.
      Several forms of consumer involvement can protect against the threat posed by selecting irrelevant issues or problems to address. For example, Seekins et al
      • Seekins T.
      • Smith N.
      • McCleary T.
      • Clay J.
      • Walsh J.
      Secondary disability prevention: involving consumers in the development of policy and program options.
      first conducted an extensive literature review to identify a range of secondary conditions that might be experienced by adults with disabilities related to mobility impairments. This review led to the identification of 26 conditions. Next, they conducted a series of focus groups and an open-ended survey of adults with disabilities. This led to the identification of 14 additional conditions they had not found in the literature. Later research led to the finding that 10 of the top 15 conditions were those identified by consumers but not found in the literature review.
      • Seekins T.
      • Clay J.
      • Ravesloot C.
      A descriptive study of secondary conditions reported by a population of adults with physical disabilities served by three independent living centers in a rural state.
      Had consumers not been involved in the early stage of this research, it is unlikely that the consumer-relevant issues would have been selected for research and development.

       Threats posed by lack of clarity of important goals

      Researchers can select important issues for study but miss the mark by focusing on socially unimportant goals. Clark et al
      • Clark H.B.
      • Greene B.F.
      • Macrae J.W.
      • McNees M.P.
      • Davis J.L.
      • Risley T.R.
      A parent advice package for family shopping trips: development and evaluation.
      present a series of studies that outline procedures for increasing the likelihood that important issues and goals will be addressed. In the first study, they reported canvassing parents to determine child-rearing problems in need of solutions. These parents indicated that shopping with young children often posed significant problems. Next, they surveyed parents about desired goals in shopping, and the acceptability of using rewards and punishers. Third, they worked with 2 families to develop and test procedures (eg, instructions and contingencies) for achieving satisfactory shopping trips. This led to 3 separate procedures, which were experimentally evaluated, and the rejection of a fourth approach, because it appeared to be incompatible with parental practices. Extended evaluations under experimental controls replicated the effectiveness and consumer satisfaction with the intervention. The advice package was later commercially disseminated.
      • Green B.F.
      • Clark H.B.
      • Risley T.R.
      Shopping with children: advice for parents.

       Threats posed by choosing unacceptable methods

      A third threat to social validity involves the use of research methods that would be deemed unacceptable by those who participate in the research or to those who might adopt the resulting products. Santelli et al
      • Santelli B.
      • Singer G.H.
      • DiVenere N.
      • Ginsberg C.
      • Powers L.
      Participatory action research: reflections on critical incidents in a PAR project.
      describe a partnership between researchers and consumers—in this case, parents of children with disabilities—in which the participants negotiated a clear delineation of tasks. The researchers agreed to give direction on research design and methods, take primary responsibility for developing study instruments, implement the intervention, run statistical analyses, interpret and summarize the data, and publish the results in peer-reviewed journals. The consumers agreed to provide information about parent-to-parent organizations, suggest modification to the research design and research methods and therefore the study would be more comfortable for parents, recruit participants, and write about the findings in a way that would be clearly understood by other parents.

       Threats posed by misunderstanding the range of acceptable interventions

      Innovation is a hallmark of applied research. Designing an innovative, socially valid intervention is tricky business, however. Researchers must consider not only the potential effectiveness of procedures and the acceptability of research methods but must also consider whether the innovation itself will be acceptable to potential adopters. Research in community development and appropriate technology suggest a number of dimensions along which consumers or participants in research might inform the choices that affect the characteristics of an innovation.
      • Fawcett S.B.
      • Mathews R.M.
      • Fletcher R.K.
      Some promising dimensions for behavioral community psychology.
      • Schumacher E.F.
      Small is beautiful: economies as if people mattered.
      These include dimensions such as simplicity, cost, contextual compatibility, and flexibility. In general, consumers or potential adopters can provide information about the type and form of an intervention that may be most attractive and most feasible.
      Consumers can be involved in the development and selection of strategies and intervention approaches in several beneficial ways. For example, Klatt
      • Klatt K.P.
      • White G.W.
      • Gard M.J.
      Do they get it? Exploring the value of research and the ability of participants with disabilities to identify key components of a research article.
      and White
      • White G.W.
      • Klatt K.
      • Gard M.
      • Sucwoierska M.
      • Wyatt D.
      Empowerment through research: a primer to guide understanding and use of research to make a difference.
      and colleagues worked with a group of consumers to develop a research primer for better understanding disability research. However, during the initial meeting with about 20 consumers recruited from a center for independent living (CIL), it became apparent that these individuals were not very interested in understanding how the research process works. Rather, they wanted to know more about how disability research could actually help improve their lives. As a result of these discussions the researchers changed their project objectives to accommodate the consumers' interests and request. The resulting product
      • Klatt K.P.
      • White G.W.
      • Gard M.J.
      Do they get it? Exploring the value of research and the ability of participants with disabilities to identify key components of a research article.
      was a consumer-directed booklet that shows consumers a step-by-step approach on how to conduct their own research to make positive community change for people with disabilities.

       Failure to consider adopter criteria for success

      A fifth threat to the social validity of research is failure to understand the criteria that potential adopters and beneficiaries use to judge the significance of the impact expected from the research. Researchers can focus on important problems and do so in a way that is acceptable, but fail to understand the criteria that would convince potential adopters to believe that the use of the research results would be meaningful to them. Many researchers who use statistical analyses to detect differences fail to consider the clinical or practical significance of results.
      • Kirk R.E.
      Practical significance: a concept whose time has come.
      • Rutledge T.
      • Loh C.
      Effect sizes and statistical testing in the determination of clinical significance in behavioral medicine research.
      While statistical differences can be particularly important in testing theory, because they can help detect small differences, small differences are usually of less interest to practitioners who are looking for information or techniques that make a meaningful difference. Potential consumers can help focus attention on the criteria for practical significance.
      • Santelli B.
      • Singer G.H.
      • DiVenere N.
      • Ginsberg C.
      • Powers L.E.
      Participatory action research: reflections on critical incidents in a PAR project.

       Threats posed by misinterpreting results

      A sixth threat to the social validity involves misinterpreting results. This can occur when researchers either draw conclusions that are biased or fail to notice implications. Involving people with disabilities in the discussion of research findings can emphasize such practical implications. Similarly, consumers might raise issues for further systematic investigation that researchers working alone might miss. Consumers might also help identify subtle limitations in study methods and procedures that researchers might miss as well.

      Generality of findings in real-life application

      The contributions by consumers or potential users of research are considered to be data in a research project or study. As such, they are subject to threats similar to those posed to the collection of other data. One of these is the threat posed by selection bias. Any consumer or potential user has his or her individual perspective on a range of issues. These may not reflect a representative sample of a researcher's intended constituency. A classic example involves the conflict between the medical and disability-rights models in disability and rehabilitation research. For example, there has been much debate about whether resources should be allocated to finding a cure for various ailments that lead to disability or finding better ways to promote community integration and participation of those experiencing disability (ie, the independent living model).

      Justice for All. Available at: http://www.jfanow.org/cgi/getin.pl?1R. Accessed June 13, 2012.

      • Shapiro J.P.
      No pity: people with disabilities forging a new civil rights movement.
      • White G.W.
      • Simpson J.L.
      • Gonda C.
      • Ravesloot C.R.
      • Coble Z.
      From independence to interdependence: a conceptual model for better understanding community participation of centers for independent living consumers.
      Many more subtle biases exist, including biases that would lead informants to want to please the researchers, thus undermining the intent of consumer involvement.
      A special case of this threat involves the generality of findings and recommendations from one cultural group to another. Clay
      • Clay J.A.
      Native American independent living.
      analyzed cultural variables in considering the application of independent living interventions and procedures to American Indians. She identified aspects of Independent Living philosophy that were both compatible and incompatible with American Indian culture. Involving representatives from culturally diverse groups can help protect against assumptions that an approach that is effective in one context will work in every cultural context.

      Strategies of consumer involvement and how these protect against threats to social validity

      As experimental designs protect against threats to internal and external validity, consumer involvement in research can also protect against threats to social validity. Following Campbell,
      • Campbell D.T.
      • Stanley J.C.
      Experimental and quasi-experimental designs for research.
      White,
      • White G.W.
      • Suchowierska M.
      • Campbell M.
      Developing and systematically implementing Participatory Action Research.
      and colleagues, there are 3 levels of consumer involvement in research: pre-PAR designs, true PAR designs, and quasi-PAR designs. The far left column of table 1 displays these 3 levels of consumer involvement and the forms of consumer involvement within each level. These are each briefly discussed below.

       Preinvolvement

      Consumer involvement is a process not an outcome. Research with no or minimal amounts of direct consumer involvement may still have social validity. This may be referred to as preinvolvement. Three forms of preinvolvement are described below.

       Consumers as subjects of research

      White et al
      • White G.W.
      • Suchowierska M.
      • Campbell M.
      Developing and systematically implementing Participatory Action Research.
      describe the role of consumers who simply provide data and refer to this as nonparticipatory subject-based research. In this case, the authors describe no implementing mechanisms for PAR, because consumers or potential users have no other role than providing data (eg, answering a questionnaire). The approach, as described here, provides no protection against threats to social validity.

       Theory-derived data mining or methods research

      This form of research involves analysis of already collected data, such as U.S. census data. A critical aspect of this type of research is that the questions addressed are derived from a theory or conceptual system. If, for example, questions of interest were derived from requests made by policy-makers, program managers, or advocates, then that would reflect consumer involvement in selecting the issue. Theory-derived data mining appears to fall into the category of nonparticipatory research described by White et al.
      • White G.W.
      • Suchowierska M.
      • Campbell M.
      Developing and systematically implementing Participatory Action Research.

       Advisory committees

      White
      • White G.W.
      • Suchowierska M.
      • Campbell M.
      Developing and systematically implementing Participatory Action Research.
      describes advisory committees as an implementing mechanism for user-sensitive PAR research. By this, the author means that involvement in research is typically limited to the initial stages of research and that the extent of involvement even at this stage is limited in degree. Presumably, consumers might help select issues for study, suggest potential audiences and users, and offer suggestions about strategies. The researchers are free to accept, modify, or even reject this advice. Consumers have control only to the extent that they can refuse to participate, or are willing to comment on the research and the responsiveness of the researchers to others in the networks they represent.
      Many research centers describe consumer involvement strategies that involve consumer participation on advisory boards. These advisory boards may serve to guide the research of a large program or focus on a single project. The composition and processes of such advisory boards vary dramatically and may address programmatic, political, research, dissemination, as well as consumer issues. Further, the preparation and experience of members may vary substantially. Their awareness of research procedures and design, and their frequency of meeting may also vary. Typical advisory committees help set direction or consent to research activities, and may periodically review progress. Depending on the openness of researchers, this form of consumer involvement can provide low to moderate levels of protection against threats to social validity. Researchers might be expected to report the issues (ie, threats to social validity) and contributions addressed by an advisory committee. Major recommendations rejected by researchers should be reported, along with a rationale for doing so.

       True PAR

      True experimental designs use randomized assignment to control for threats to internal and external validity. True PAR involves total control of the research by those affected. As with random assignment, this protects against threats to social validity, but not necessarily to the generality of findings.

       Emancipatory research

      White
      • White G.W.
      • Suchowierska M.
      • Campbell M.
      Developing and systematically implementing Participatory Action Research.
      describes emancipatory research as full participation and complete control over the research process. Emancipatory research is based on the understanding that emerged from efforts such as those of the Mondragon in Spain.
      • Whyte W.F.
      Social inventions for solving human problems.
      That community took charge of its development by learning and employing empirically grounded scientific methods to local economic development. The intent of such research is to focus almost exclusively on the issues of importance to the members of the community or even smaller subgroups (eg, a business). It is emancipatory in the sense that the participants control the selection of issues for study, the methods used, and the evaluation of success, set free from influence by others who are not members of that community.
      In this vein, the National Institute on Disability and Rehabilitation Research funded the Community Research for Assistive Technology project (grant no. H133A010702) to explore the potential of true PAR. This project examined disparities in assistive technology use in employment.
      • Yeager P.
      • Kaye H.S.
      • Reed M.
      • Doe T.M.
      Assistive technology and employment: experiences of Californians with disabilities.
      • Kaye H.S.
      • Yeager P.
      • Reed M.
      Disparities in usage of assistive technology among people with disabilities.
      The work was conducted by the California Foundation for Independent Living Centers.

       Organizational quality improvement

      Research and development can be done within an organizational context as well as a community. The difference between continuous quality improvement and emancipatory research involves the structure, authority, and even ownership of resources. Owners, managers, and supervisors typically have a greater say in setting an organization's applied research agenda. Still, the values, goals, resources, and customs of the organization are very likely to be reflected in the applied research and development activities that are targeted at its own improvement. Here, it is important to distinguish between organizational development and research that is focused on developing or improving products and services that it offers to its consumers. Threats to the social validity of the latter form of product-oriented research are not protected by organizational developmental procedures. Rather, specified consumers must be involved to do so.

       Community development action research

      Community development action research is similar to emancipatory research. It is usually open and democratic, and it is typically directed by a group of community leaders who engage an external consultant to orchestrate a research and development process. The range of issues and processes are generally outlined by a steering committee. Community members are typically engaged as stakeholders to help define specific issues, generate debate, select alternatives, implement alternatives, and judge their success. The larger and more diverse the community, the more consumer involvement must be achieved through representation rather than direct participation. Still, many design features can be considered, such as the extent to which a new approach (ie, intervention, policy, or reorganization of relations) fits with local customs and resources.
      Drum et al
      • Drum C.E.
      • Berardinelli M.
      • Pickett-Cooper P.
      • et al.
      Community action guide: version 2.0.
      • Drum C.E.
      • Goff T.
      • Horner-Johnson W.
      • Pobutsky A.
      • Ritacco B.
      • Weaver A.
      Community action guide: a process for improved community accessibility.
      • Drum C.E.
      • Krahn G.
      • Horner-Johnson W.
      • et al.
      The Oregon community engagement initiative: a multi-case study of a disability coalition development process.
      designed a community development approach to address local access barriers. This methodology, called the Community Engagement Initiative (CEI), builds on Hahn's
      • Hahn H.
      The potential impact of disability studies on political science (as well as vice versa).
      theory on the institutional creation of disability and more traditional community development. The CEI process engages disability advocates in promoting community dialogue to resolve accessibility problems. The main steps in the CEI method include: (1) a facilitated town hall meeting involving persons with disabilities and their families in which they identify barriers to participating in their community, (2) a facilitated meeting between representatives from the community's leadership and town hall participants to review community assets and barriers, and (3) a barrier mobilization process. The CEI methodology has achieved promising results in resolving a range of access barriers.

       Quasi-PAR

      As with experimental designs in applied field settings, most consumer involvement designs combine several procedures in order to address specific concerns at different stages of research. Ten procedures are subsequently outlined. Others are possible.

       Focus groups

      The technical use of focus groups has grown in the past several years.
      • Byers P.Y.
      • Zeller R.A.
      • Byers B.D.
      Focus group methods.
      • Kroll T.
      • Barbour R.
      • Harris J.
      Using focus groups in disability research.
      In disability and rehabilitation research, focus groups typically refer to one or a few small groups of individuals whom the researchers select for reasons of convenience, representation of a constituency, or potential use of results from a line of inquiry. Focus groups are seen as temporary project advisory groups to help address specific or narrow questions. For example, Seekins et al
      • Seekins T.
      • Smith N.
      • McCleary T.
      • Clay J.
      • Walsh J.
      Secondary disability prevention: involving consumers in the development of policy and program options.
      reported using focus groups of individuals with disabilities related to physical and sensory impairments to help identify and define secondary conditions experienced by adults with such conditions. As mentioned, these focus groups helped identify 14 conditions not identified in the literature review. Further research with larger numbers of participants using survey methods built evidence of the generality of these consumer identified issues. As such, focus groups provide modest protection against threats posed to the issues they address. Confirmation through multiple groups or surveys enhances these protections.

       Researcher translation from network participation

      Popular publications by people with disabilities about issues of importance to them offer insight into issues of importance, acceptability of methods, and criteria that people with disabilities might use to assess the significance of impact of a program.

      American Association of People with Disabilities. 2002. Available at: http://www.aapd.com/. Accessed June 13, 2012.

      Mouth. Voice of the disability nation. Available at: http://www.mouthmag.com/. Accessed June 13, 2012.

      The Ragged Edge Online. Available at: http://ragged-edge-mag.com/. Accessed May 11, 2012.

      National Alliance on Mentally Illness. Available at: http://www.nami.org/advocacy.html. Accessed June 13, 2012.

      Alliance for Equality of Blind Canadians. Available at: http://www.nfbae.ca/. Accessed June 13, 2012.

      People First Ltd. People first and other self advocacy links. 2002. Available at: http://www.peoplefirst.org.uk/pflinks.html. Accessed October 1, 2012.

      Reviews of this literature are equivalent to reviews of technical literature. Similarly, researchers can participate in conferences that emphasize perspectives, goals, and standards that are held by people with disabilities (eg, national or state conferences of such organizations, such as the Association of Programs for Rural Independent Living, National Council on Independent Living, and Society for Disability Studies, meetings of state traumatic brain injury associations, state and national meetings of the National Alliance for the Mentally Ill, and other similar groups).
      Participation in consumer-run organizations can also provide timely and critical information. With experience, researchers can learn to translate both direct and subtle calls for different forms of research on various topics of importance. As with focus groups, these forms of consumer involvement provide low to moderate protection against threats to social validity to the topics addressed. Focus groups or surveys that confirm such interpretations enhance protection.

       Agenda setting surveys

      Perhaps one of the most widely used procedures for consumer involvement in disability and rehabilitation research is the concerns report method.
      • Schriner K.F.
      • Fawcett S.B.
      Development and validation of a community concerns report method.
      • Fawcett S.B.
      • Seekins T.
      • Whang P.L.
      • Muiu C.
      • Suarez de Balcazar Y.
      Involving consumers in decision-making.
      The concerns report method has been demonstrated to be a particularly useful method of building a broad, consumer-oriented research agenda. It has been used to set research agendas reflecting interests of people with disabilities living in rural communities, vocational rehabilitation counselors, CILs, individuals with intellectual and developmental disabilities, individuals with psychiatric disabilities, and Peruvians with disabilities.
      • Jackson K.
      • Seekins T.
      • Offner R.
      Involving consumers and service providers in shaping rural rehabilitation agenda.
      • Arnold N.
      • Seekins T.
      • Nelson R.
      A comparison of vocational rehabilitation counselors: urban and rural differences.
      • Jones M.
      • Petty R.
      • Bolles C.
      • Matthews R.M.
      Independent living: a survey of program and service needs.
      • Youngbauer J.G.
      • Budde J.
      Needs conference for underserved consumers with mental retardation.
      • Snyder J.
      • Temple L.
      • Youngbauer J.G.
      Needs conference for underserved consumers with psychiatric disabilities.

      White GW, Chapman V, Jay A, Branstetter A, Mayo L, Isola J. Using the consumer concerns report method to identify key issues for Perúvians with disabilities. Proceedings of the 128th Annual Meeting of the American Public Health Association, Disability Forum; 5107.0: November 15, 2000 - Board 8 Abstract #8663. Boston, MA.

      The concerns report method combines several different procedures of consumer involvement. Researchers use literature reviews and focus groups to generate a concerns menu, often composed of several hundred issues. Next, additional consumer representatives use these menus to select and develop items for a concerns survey. Surveys are then distributed among a large, defined constituency represented by the participants in the focus groups. Resulting data are summarized as potential strengths and problems, and subsequent focus groups discuss these results to explore the nature of the most important issues. Such structured discussions often recommend studies that will build on or protect strengths and solve problems. This method provides exceptional protection against threats to social validity posed by irrelevance of issues selected and lack of clarity about important consumer goals. It also provides reasonable protection against threats posed by misunderstanding of the range of acceptable interventions, and ignorance of criteria for success. Because the process does not typically provide for discussion of research methods, it does not provide direct information about the acceptability of research methods. Finally, if the concerns survey data are collected from a representative sample, they also provide some protection against threats posed to the generality of findings and products.

       Best practices research

      From a social perspective, the purpose of a great deal of applied research is to identify strategies for improving the quality of life for people experiencing similar conditions (ie, to identify replicable solutions to common problems). One approach to this task is to conduct best practices studies. This means observing what people and organizations are currently doing to address a common issue and comparing the outcomes of those approaches.
      As these programs continue to be used, it may be presumed that they address important problems in an acceptable way, and offer some degree of effectiveness and user satisfaction. Further, there is a presumption that they are compatible with local culture and resources. (Some best practices research projects focus primarily on these latter dimensions.) As such, users have chosen to implement and maintain these practices. From a behavioral perspective, the choice to adopt an innovation is seen as a function of antecedent and consequent events—the contingencies of reinforcement act to sustain the practice. This constitutes a type of PAR or the results of it, regardless of the origin of the idea.
      An important aspect of best practices research is to match the characteristics of target settings with needs to those with successful practices. Potential users in a target setting should at least have acknowledged the problem they face, if they have not initiated the request for information about alternative practices. Researchers must also be able to show similarities and differences between settings that might function to facilitate or impede use. For example, Clay
      • Clay J.A.
      Native American independent living.
      compared characteristics of the independent living (IL) philosophy and service model as it fits into the general social context and, more specifically, into American Indian tribal context. The author identified features of compatibility and incompatibility that might influence tribal adoption of IL programs. One point of incompatibility, for example, involved the role of advocacy. Advocacy (both for individuals and people with disabilities in general) is a cornerstone of the IL model. Conversely, within many tribal cultures, individual issues are expected to be subjugate to issues of tribal sovereignty: what is best for the tribe as a whole. Researchers and program developers who take such issues into account protect their work from threats posed by a lack of understanding of the acceptability of research methods and misunderstandings about the range of acceptable interventions.

       Research teams

      One of the most intensive forms of consumer involvement occurs when researchers and representatives of potential users collaborate on a research project. This process involves a dynamic and ongoing interplay between research collaborators, or what White et al
      • White G.W.
      • Nary D.E.
      • Froehlich A.K.
      Consumers as collaborators in research and action.
      identify as consumer empowered teams. The consumer collaborators function as members of the research team during the course of the project and may be paid as members of the project. This form of consumer involvement may be most effective in the design and testing of an innovative program or treatment technique. For example, Balcazar et al
      • Balcazar F.
      • Seekins T.
      • Fawcett S.B.
      • Hopkins B.L.
      Effects of training on advocacy behaviors, and generalization to community engagements and outcomes.
      reported a project in which members of a local consumer advocacy group worked hand in hand with researchers on the development and evaluation of a consumer advocacy system. In such instances, the role consumers play often fades away and they become collaborating members of the research team. It provides good protection against threats posed by misunderstanding of the range of acceptable intervention approaches and failure to consider adopter criteria for judging success.

       People with disabilities as researchers

      There are many researchers in the field of applied disability and rehabilitation science who experience disability themselves. The participation of these research leaders in the design, conduct, and use of research may be seen as a form of PAR. In this case, one or more individuals with personal experience with disability and research experience have significant control over the research process and the allocation of resources in that process: an approximation of the White et al
      • White G.W.
      • Suchowierska M.
      • Campbell M.
      Developing and systematically implementing Participatory Action Research.
      definition of true-PAR. These researchers with disabilities may pursue issues they have seen as important in their personal experience, as well as ones that relate to their professional interest. For example, one of the authors
      • White G.W.
      • Thomson R.J.
      • Nary D.E.
      An empirical analysis of the effects of a self-administered advocacy letter training program.
      had problems in dealing with his insurance company paying for a wheelchair and spent numerous hours in trying to get the equipment paid for, as specified in the insurance plan. As a result of this experience, White et al
      • White G.W.
      • Thomson R.J.
      • Nary D.E.
      An empirical analysis of the effects of a self-administered advocacy letter training program.
      created a guide for others with disabilities on how to successfully advocate for their rights through writing action letters that could articulate their disability concerns and serve as a paper trail to monitor response or nonresponse of the targeted individual or organization. This guide, later renamed the action letter portfolio, was created, evaluated, and the results later published for others to replicate.
      This form of consumer involvement provides good protection against all threats to social validity except those that involve the generality of findings. Specifically, researchers with disabilities represent a small and potentially select sample of the population. While their extensive personal experience with disability, and their expertise as researchers, provides them with insights into issues of importance and the subtleties of selecting important goals, they may be less familiar with aspects of the different contexts in which results are to be applied. Such researchers often use additional forms of consumer involvement to protect against biases and use their own experience to augment that information, and thus, further enhancing the social validity of their work.

       Consumer consultants

      Some research programs engage technical experts to assist in tasks such as experimental design, measurement, or statistical analysis. Similarly, some experts can serve as consultants on both consumer involvement strategies and social validity. Experts may consult on both the process and the content of research. While expertise may inform researchers about both, experts may have their own biases. Multiple consultants enhance protection but may begin to approximate focus groups comprised of experts. The benefits of expert consumer consultations include efficiency, experience in the content area and research process, familiarity with the subtleties and interrelations among a broad range of issues, deeper understanding of the interaction of research and content, and awareness of linkages to other areas of research and other researchers working on related topics.

       Product champions and disseminators

      After research and development of a particular innovation, adopters may recognize the particular value of the research product. If convinced of the research effectiveness, these constituents may then work to help disseminate it by providing testimonials or more intense training and technical assistance in its use. While this approach does not involve consumers in the design and conduct of the research, those who choose to support the research outcomes are applying similar judgments to the evaluation of the results. As such, they are also likely to display a favorable opinion of the social validity of the research product. Of course, these product champions may face actual or apparent conflicts of interest if they are paid for this role.

       Partners in advocacy

      It may be more proper to say that researchers become partners with consumers in advocacy, because the consumer representatives are more often the leaders in such efforts: they choose the issues, develop the strategies, and may call on researchers as members of a strategic effort. Still, consumer-initiated advocacy efforts steer the issue agenda and articulate goals. Research that addresses issues and goals in this manner may be said to enjoy some protection from threats to its social validity posed by those concerns. For example, the National Council on Independent Living initiated an advocacy effort to increase funding for CILs, the Drive for 75. Their goal was to increase funding for CILs by $75 million. Innes et al
      • Innes W.
      • Enders A.
      • Seekins T.
      • Merrit D.
      • Kirshenbaum A.
      • Arnold N.
      Geographic analysis of independent living center services: urban and rural distribution equity.
      conducted an independent study to examine the distribution of CILs around the nation and estimated that it would take approximately $72 million to achieve minimal but universal access to their services in all counties of the United States. These research data, showing independent confirmation of the National Council on Independent Living's estimate, were used as one of the advocacy points in the Drive for 75. It is important to note, however, that research conducted to address an advocacy issue may not always support the position of advocates.

      Discussion

      This article describes a method to organize and understand consumer involvement in research within a scientific framework. Specifically, we present the case that involving those who are expected to use and benefit from the products of research is a design element that protects research findings from threats to its social validity. This framework offers tools familiar to all scientists for identifying and managing threats to the quality of research, and for judging the effectiveness of the strategies for protecting against those threats. It also organizes the various procedures in a way that allows for systematic criticism of their effectiveness and subsequent improvement. Finally, it provides tools for improving the design of research from the beginning.
      Peer review of research proposals and reports of research findings are a critical step in the advancement of science. Yet, there are few guidelines for reporting consumer involvement procedures in proposals or publications. For example, style guides such as the American Psychological Association's Publications Manual
      American Psychological Association
      Publication manual of the American Psychological Association.
      offer no advice on reporting consumer involvement nor do such standards as the Consolidated Standards of Reporting Trials checklist.
      Just as funding agencies and journal editors are concerned about the methods used to protect against threat to internal and external validity, they might also develop expectations that researchers address how they plan to protect against threats to social validity in their research proposals and manuscripts. Proposal reviewers should consider how the researchers selected the problem to address, the goals to achieve, the methods used, and the criteria for judging success. Researchers may build the case for the social validity of their work in several ways. As with any other aspect of research, the reviewer must judge the extent to which it meets standards. Judging the effectiveness of the procedures to maximize social validity is no exception. The framework proposed here offers a tool that can be consistently applied by reviewers with diverse backgrounds, such as those often found on review panels.
      Stylistically, aspects of consumer involvement may be reported in different ways. First, for example, if consumer support for a line of research comes from established consumer literature, or interpolation of information gathered through conferences or other meetings, it might best be cited in the introduction of a manuscript. Similarly, data that support the importance of the issues, such as that collected from consumer surveys, should be presented in the introduction. Otherwise, the consumer-involvement procedures may be best incorporated into the methods section of a manuscript. For example, the characteristics of participants in a focus group might be reported in the section traditionally labeled subjects. The actual operation of the focus group might be described in the procedures section. Third, limitations to social validity of the research should be articulated in the discussion section, just as limitations to the reliability and validity of the data would be discussed there. Finally, if no process was used to protect against threats to the social validity of the research, an author might be expected to acknowledge this limitation and provide a clear rationale for its absence in the discussion section.

      Conclusions

      This article presents a framework for judging the effectiveness of strategies of consumer involvement in protecting applied research studies against threats to its social validity. It is not an exhaustive list or a cookbook. Rather, it offers a tool to those who are designing or reviewing research. Reviewers of research must rely on their experience and insights to make judgments of the merit of particular research methods and results. This applies to judgments about the internal, external, and social validity of applied research.
      Consumer involvement may represent a spirit of democracy or even empowerment, but as a tool of science, it is necessary to understand how to judge its application. To advance this tool, however, it is important for researchers to report their efforts to protect against threats to social validity and for their peers to critique their methods.

      References

        • Gray D.
        Disability and rehabilitation research from policy to program: a personal perspective.
        Am Psychol. 1990; 45: 751-756
        • National Institute on Disability and Rehabilitation Research
        PAR sourcebook: forging collaborative partnerships in the study of disability.
        National Institute on Disability and Rehabilitation Research, U.S. Department of Education, Washington (DC)1995
        • National Institute on Disability and Rehabilitation Research
        The new paradigm on disability: research issues and approaches.
        National Institute on Disability and Rehabilitation Research, U.S. Department of Education, Washington (DC)2001
        • Brown S.C.
        Methodological paradigms that shape disability research.
        in: Albrecht G.L. Seelman K.D. Bury M. Handbook of disability studies. Sage Publications, Thousand Oaks2001: 145-170
        • DeJong G.
        Independent living: from social movement to analytic paradigm.
        Arch Phys Med Rehabil. 1979; 60: 435-446
      1. National Institute on Disability and Rehabilitation Research (2005). Long-range plan for fiscal years 2005-2009. Available at: http://www.ed.gov/legislation/FedRegister/other/2006-1/021506d.pdf. Accessed February 16, 2012.

        • Fenton J.
        • Batavia A.
        • Roody D.S.
        Proposed policy statement for NIDRR on constituency-oriented research and dissemination (CORD).
        National Institute on Disability and Rehabilitation, Washington (DC)1993
        • Nelson G.
        • Ochocka J.
        • Griffin K.
        • Lord J.
        Nothing about me, without me: participatory action research with self-help/mutual and organization for psychiatric survivors.
        Am J Community Psychol. 1998; 26: 881-912
        • Tewey B.P.
        Building participatory action research partnerships on disability and rehabilitation research.
        Conwall Inc, McClean1997
        • White G.W.
        Consumer participation in disability research: the golden rule as a guide for ethical practice.
        Rehabil Psychol. 2002; 47: 438-446
        • White G.W.
        • Nary D.E.
        • Froehlich A.K.
        Consumers as collaborators in research and action.
        J Prev Interv Community. 2001; 21: 15-34
        • Campbell M.
        Continuum of participatory research: models, applications and implementation mechanisms.
        Annual Meeting of the National Association of Rehabilitation Research and Training Centers, Washington (DC)2001
        • White G.W.
        • Suchowierska M.
        • Campbell M.
        Developing and systematically implementing Participatory Action Research.
        Arch Phys Med Rehabil. 2004; 85: S3-S12
        • Task Force on the Principles of Community Engagement
        Principles of community engagements.
        2nd ed. NIH, 2011 (Report No. 11–7782)
        • Viswanathan M.
        • Ammerman A.
        • Eng E.
        • et al.
        Community-based participatory research: assessing the evidence.
        Evidence Report/Technology Assessment No. 99 (Prepared by RTI-University of North Carolina Evidence-based Practice Center under Contract No. 290-02-0016). AHRQ Publication 04-E022-2. Agency for Healthcare Research and Quality, RockvilleJuly 2004
        • Rogers E.M.
        • Shoemaker E.F.
        Communication of innovations: a cross-cultural approach.
        2nd ed. The Free Pr, New York1971
        • Batavia A.I.
        • Hammer G.S.
        Toward the development of consumer-based criteria for the evaluation of assistive devices.
        J Rehabil Res Dev. 1990; 27: 425
        • Burke R.K.
        Disuse of assistive technology by vocational rehabilitation and independent living clients in North Carolina.
        Department of Rehabilitation Studies, East Carolina University, Greenville1999
        • Phillips B.
        • Zhao H.
        Predictors of assistive technology abandonment.
        Assist Technol. 1993; 5: 36-45
        • Riemer-Reiss M.L.
        • Wacker R.R.
        Factors associated with assistive technology discontinuance among individuals with disabilities.
        J Rehabil. 2000; 66: 44-50
        • Verza R.
        • Carvalho M.L.
        • Battaglia M.A.
        • Uccelli M.M.
        An interdisciplinary approach to evaluating the need for assistive technology reduces equipment abandonment.
        Mult Scler. 2006; 12: 88-93
        • Newell A.
        User-centered design and disability.
        in: National Institute on Disability and Rehabilitation Research, editor. PAR Sourcebook: forging collaborative partnerships on the study of disability. National Institute on Disability and Rehabilitation Research, U.S. Department of Education, Washington (DC)2001: 101-104
      2. Seekins T. Consumer involvement in rehabilitation research over the past thirty years. In: Annual conference of the National Association of Rehabilitation Research and Training Centers; 1992; Washington (DC).

        • Remmes H.S.
        Consumer involvement in rehabilitation.
        Rehabil Rec. 1972; 1: 33-35
        • Bassman R.
        Whose reality is it anyway? Consumers/survivors/ex-patients can speak for themselves.
        J Humanist Psychol. 2001; 4: 11-35
        • Humphries K.
        • Rappaport J.
        Research self-help/mutual and groups and organizations: many roads, one journey.
        Appl Prev Psychol. 1994; 3: 217-231
        • Campbell M.
        • Seekins T.
        Whatever happened to PAR? Methodological and administrative challenges of implementing participatory research in applied rehabilitation science.
        in: National Institute on Disability and Rehabilitation Research, editor. PAR sourcebook: forging collaborative partnerships in the study of disability. National Institute on Disability and Rehabilitation Research, U.S. Department of Education, Washington (DC)2001: 51-53
        • Whyte W.F.
        Participatory action research.
        Sage Publications, Newbury Park1991
        • DeJong G.
        Independent living: from social movement to analytic paradigm.
        Arch Phys Med Rehabil. 1979; 60: 435-446
        • Wolf M.M.
        Social validity: the case for subjective measurement or how applied behavior analysis is finding its heart.
        J Appl Behav Anal. 1978; 11: 203-214
      3. Leong F.T. Austin J.T. The psychology research handbook: a guide for graduate students and research. Sage Publishing, Thousand Oaks2006
        • Campbell D.T.
        • Stanley J.C.
        Experimental and quasi-experimental designs for research.
        Rand McNally College Publishing, Chicago1963
        • Seekins T.
        • Smith N.
        • McCleary T.
        • Clay J.
        • Walsh J.
        Secondary disability prevention: involving consumers in the development of policy and program options.
        J Dis Pol Stud. 1991; 1: 21-35
        • Seekins T.
        • Clay J.
        • Ravesloot C.
        A descriptive study of secondary conditions reported by a population of adults with physical disabilities served by three independent living centers in a rural state.
        J Rehabil. 1994; 60: 47-51
        • Clark H.B.
        • Greene B.F.
        • Macrae J.W.
        • McNees M.P.
        • Davis J.L.
        • Risley T.R.
        A parent advice package for family shopping trips: development and evaluation.
        J Appl Behav Anal. 1977; 10: 605-624
        • Green B.F.
        • Clark H.B.
        • Risley T.R.
        Shopping with children: advice for parents.
        Academic Therapy Publications, San Rafael1977
        • Santelli B.
        • Singer G.H.
        • DiVenere N.
        • Ginsberg C.
        • Powers L.
        Participatory action research: reflections on critical incidents in a PAR project.
        J Assoc Pers Sever Handi. 1998; 23: 211-222
        • Fawcett S.B.
        • Mathews R.M.
        • Fletcher R.K.
        Some promising dimensions for behavioral community psychology.
        J Appl Behav Anal. 1980; 3: 319-342
        • Schumacher E.F.
        Small is beautiful: economies as if people mattered.
        Harper and Row, New York1973
        • Klatt K.P.
        • White G.W.
        • Gard M.J.
        Do they get it? Exploring the value of research and the ability of participants with disabilities to identify key components of a research article.
        J Dis Pol Stud. 2003; 14: 2-6
        • White G.W.
        • Klatt K.
        • Gard M.
        • Sucwoierska M.
        • Wyatt D.
        Empowerment through research: a primer to guide understanding and use of research to make a difference.
        Univ. of Kansas, Lawrence2005
        • Kirk R.E.
        Practical significance: a concept whose time has come.
        Ed Psych Meas. 1996; 56: 746-759
        • Rutledge T.
        • Loh C.
        Effect sizes and statistical testing in the determination of clinical significance in behavioral medicine research.
        Ann Behav Med. 2004; 42: 138-145
        • Santelli B.
        • Singer G.H.
        • DiVenere N.
        • Ginsberg C.
        • Powers L.E.
        Participatory action research: reflections on critical incidents in a PAR project.
        J Assoc Pers Sev Handi. 1998; 23: 211-222
      4. Justice for All. Available at: http://www.jfanow.org/cgi/getin.pl?1R. Accessed June 13, 2012.

        • Shapiro J.P.
        No pity: people with disabilities forging a new civil rights movement.
        Random House, New York1994
        • White G.W.
        • Simpson J.L.
        • Gonda C.
        • Ravesloot C.R.
        • Coble Z.
        From independence to interdependence: a conceptual model for better understanding community participation of centers for independent living consumers.
        J Dis Pol Stud. 2010; 20: 233-240
        • Clay J.A.
        Native American independent living.
        Rur Spec Ed Q. 1992; 11: 41-50
        • Whyte W.F.
        Social inventions for solving human problems.
        Am Soc Rev. 1982; 47: 1-13
        • Yeager P.
        • Kaye H.S.
        • Reed M.
        • Doe T.M.
        Assistive technology and employment: experiences of Californians with disabilities.
        Work. 2006; 27: 333-344
        • Kaye H.S.
        • Yeager P.
        • Reed M.
        Disparities in usage of assistive technology among people with disabilities.
        Assist Technol. 2008; 20: 194-203
        • Drum C.E.
        • Berardinelli M.
        • Pickett-Cooper P.
        • et al.
        Community action guide: version 2.0.
        Oregon Health and Science Univ, Portland2007
        • Drum C.E.
        • Goff T.
        • Horner-Johnson W.
        • Pobutsky A.
        • Ritacco B.
        • Weaver A.
        Community action guide: a process for improved community accessibility.
        Oregon Health and Science Univ, Portland2002
        • Drum C.E.
        • Krahn G.
        • Horner-Johnson W.
        • et al.
        The Oregon community engagement initiative: a multi-case study of a disability coalition development process.
        Community Devel. 2009; 40: 3-19
        • Hahn H.
        The potential impact of disability studies on political science (as well as vice versa).
        Pol Stud J. 1993; 21: 740-751
        • Byers P.Y.
        • Zeller R.A.
        • Byers B.D.
        Focus group methods.
        in: Wiederman M.W. Whitley Jr., B.E. Handbook for conducting research on human sexuality. Lawrence Erlbaum Associates, Mahwah2002: 173-193
        • Kroll T.
        • Barbour R.
        • Harris J.
        Using focus groups in disability research.
        Qual Health Res. 2007; 17: 690-698
        • Seekins T.
        • Smith N.
        • McCleary T.
        • Clay J.
        • Walsh J.
        Secondary disability prevention: involving consumers in the development of policy and program options.
        J Dis Pol Stud. 1991; 1: 21-35
      5. American Association of People with Disabilities. 2002. Available at: http://www.aapd.com/. Accessed June 13, 2012.

      6. Mouth. Voice of the disability nation. Available at: http://www.mouthmag.com/. Accessed June 13, 2012.

      7. The Ragged Edge Online. Available at: http://ragged-edge-mag.com/. Accessed May 11, 2012.

      8. National Alliance on Mentally Illness. Available at: http://www.nami.org/advocacy.html. Accessed June 13, 2012.

      9. Alliance for Equality of Blind Canadians. Available at: http://www.nfbae.ca/. Accessed June 13, 2012.

      10. People First Ltd. People first and other self advocacy links. 2002. Available at: http://www.peoplefirst.org.uk/pflinks.html. Accessed October 1, 2012.

        • Schriner K.F.
        • Fawcett S.B.
        Development and validation of a community concerns report method.
        J Community Psych. 1988; 16: 306-316
        • Fawcett S.B.
        • Seekins T.
        • Whang P.L.
        • Muiu C.
        • Suarez de Balcazar Y.
        Involving consumers in decision-making.
        Soc Pol. 1982; 13: 36-41
        • Jackson K.
        • Seekins T.
        • Offner R.
        Involving consumers and service providers in shaping rural rehabilitation agenda.
        Am Rehabil. 1992; 18 (48): 23-29
        • Arnold N.
        • Seekins T.
        • Nelson R.
        A comparison of vocational rehabilitation counselors: urban and rural differences.
        Rehabil Couns Bull. 1997; 41: 2-14
        • Jones M.
        • Petty R.
        • Bolles C.
        • Matthews R.M.
        Independent living: a survey of program and service needs.
        Rehabil Couns Bull. 1986; 29: 278-283
        • Youngbauer J.G.
        • Budde J.
        Needs conference for underserved consumers with mental retardation.
        Research and Training Center on Independent Living, Univ. of Kansas, Lawrence1994
        • Snyder J.
        • Temple L.
        • Youngbauer J.G.
        Needs conference for underserved consumers with psychiatric disabilities.
        Research and Training Center on Independent Living, Univ. of Kansas, Lawrence1995
      11. White GW, Chapman V, Jay A, Branstetter A, Mayo L, Isola J. Using the consumer concerns report method to identify key issues for Perúvians with disabilities. Proceedings of the 128th Annual Meeting of the American Public Health Association, Disability Forum; 5107.0: November 15, 2000 - Board 8 Abstract #8663. Boston, MA.

        • White G.W.
        • Nary D.E.
        • Froehlich A.K.
        Consumers as collaborators in research and action.
        J Prev Interv Community. 2001; 21: 15-34
        • Balcazar F.
        • Seekins T.
        • Fawcett S.B.
        • Hopkins B.L.
        Effects of training on advocacy behaviors, and generalization to community engagements and outcomes.
        Am J Community Psychol. 1990; 18: 281-296
        • White G.W.
        • Thomson R.J.
        • Nary D.E.
        An empirical analysis of the effects of a self-administered advocacy letter training program.
        Rehabil Couns Bull. 1997; 41: 74-87
        • Innes W.
        • Enders A.
        • Seekins T.
        • Merrit D.
        • Kirshenbaum A.
        • Arnold N.
        Geographic analysis of independent living center services: urban and rural distribution equity.
        J Dis Pol Stud. 2000; 10: 207-224
        • American Psychological Association
        Publication manual of the American Psychological Association.
        American Psychological Association, Washington (DC)1994