Process Evaluations and Qualitative Research Projects

Evaluation of American Recovery and Reinvestment Act (ARRA) Comparative Effectiveness Research Dissemination

IMPAQ is conducting an evaluation of the effectiveness of the various dissemination and implementation approaches by AHRQ’s Office of Communications and Knowledge Transfer for the various comparative effectiveness research (CER) products prepared as part of the Effective Healthcare Program (EHC). Phase 1 of the task order is a 3-year ARRA project to evaluate the effectiveness of four different dissemination methods:

• A publicity center that disseminates EHC findings nationwide through marketing/media efforts (i.e., press releases, audio podcasts, videos), organizational partnerships, and “Virtual Centers” on leading national consumer and clinician Health-related Websites.
• Regional offices that develop partnerships with health care organizations, businesses, unions, consumer and other organizations to create and develop ongoing dissemination partnerships and to conduct outreach to their members.
• Online continuing education activities for new target audiences, including pharmacists, nurses, nurse practitioners, physician assistants, and other allied health professionals.
•  Academic detailing where trained clinicians conduct in-person educational meetings with other individual clinicians, clinician groups, and health plan formularies across the country.

IMPAQ, and its subcontractor Battelle, will determine the level of increased awareness, understanding, use, and benefits of CER among target audiences by collecting and analyzing quarterly data from the four dissemination contractors and conducting annual mail surveys of clinicians, telephone surveys of consumers, focus groups of targeted clinician and consumer audiences, and one-on-one interviews with other targeted audiences.  We will evaluate the dissemination contractors’ initiatives at midpoint and at the end of the contract to inform AHRQ of results and, in conjunction with AHRQ, recommend adjustments to their initiatives.

Chronic Disease Self Management Program (CDSMP) Process Evaluation and Detailed Outcome Evaluation Design

Administration on Aging (AoA) contracted with IMPAQ International to conduct a thorough process evaluation of the Stanford University's Chronic Disease Self-Management Program (CDSMP) in AoA-funded settings as well as to design a detailed operational plan for the conduct of the most rigorous, yet feasible, outcome evaluation. AoA is particularly interested in a Randomized Controlled Trial (RCT) outcome evaluation.  IMPAQ has previously completed for AoA and Agency for Healthcare Research and Quality (AHRQ) a feasibility study of an RCT evaluation design for CDSMP. CDSMP teaches consumers skills to manage their conditions and build their self-confidence so they can be successful in adopting healthy behaviors, improve communications with their physician, and enhance their quality of life. The program consists of workshops conducted once a week for two and a half hours over six weeks in community-based settings such as senior centers, congregate meal programs, faith-based organizations, libraries, YMCAs, YWCAs, and senior housing programs. People with different chronic health conditions attend together, and the workshops are facilitated by trained and certified leaders, at least one of whom has a chronic illness.

Process Evaluation: Through this contract, IMPAQ will provide AoA detailed information about actual implementation of the CDSMP program by AoA grantees and specifically with the 60+ population. In addition to collecting and analyzing data on staffing, resource allocation, fidelity to model, one of the areas of focus will be assessing whether and how these grantees built sustainable statewide distribution and delivery systems which increase the availability of evidence-based self management programs and provide a distribution channel for other evidence based programs that may be delivered by community based organizations. IMPAQ will explore the reasons why some programs become sustainable and some do not. Methods of data collection will include site visits, detailed telephone calls with sites, focus groups, and a web –based process evaluation survey that will be administered to the universe of AoA CDSMP grantees and host sites. IMPAQ will prepare a OMB PRA clearance package for the web-based process evaluation survey.

Outcome Evaluation: In general, individuals who complete a CDSMP have shown statistically significant improvements in self-efficacy, health behaviors and health status. There is also some evidence of improved health care utilization and reduced health care costs. But, to date, there is little specific evidence about the effects of the CDSMP on seniors, 60+, with chronic diseases. IMPAQ International will develop a detailed, rigorous design for an outcome evaluation of AoA CDSMP grantee programs to rigorously estimate the impact of CDSMP on adults of ages 60+ on a variety of outcome such as health care utilization including hospitalization and use of emergency services, health care costs, quality of life/community integration, and self-efficacy. RCT is the preferred design. IMPAQ will also develop requirements of a data collection system that is capable of supporting an RCT design.

Green Jobs and Healthcare Implementation Study

IMPAQ is leading an implementation evaluation of four grant programs awarded with American Recovery and Reinvestment Act funds by the Department of Labor in early 2010, including:

• Energy Training Partnership Grants
• Pathways out of Poverty Grants
• State Energy Sector Partnership and Training Grants
•  Healthcare and High Growth Training Grants

Together these programs include over 150 projects with total award amounts of nearly $665 million dollars, with wide variation in grantee organizations and populations served.  The programs focus on workforce development and training to prepare workers for careers in various industries related to renewable energy and energy efficiency, healthcare, or other emerging industries.  IMPAQ is gathering information to assess the implementation of these grants in areas including program context, design and goals, recruitment, training, support, partnerships, proposed program outcomes, and program sustainability.  IMPAQ is reviewing site documentation, conducting site visits to select grantees, conducting a survey of grantees, and conducting focus groups with participants.  IMPAQ will produce a report to address key research questions related to the implementation of grant programs in these industry sectors.

National Content Developer

IMPAQ International is supporting the Centers for Medicaid & Medicare Services (CMS) in implementing the Partnership for Patients (PfP) initiative. The initiative will support physicians, nurses and other clinicians working in and out of hospitals to make patient care safer and to support effective transitions of patients from hospitals to other settings. The two main goals of this priority project are: 1) to reduce preventable hospital-acquired conditions (HACs) by 40% over a 3 year period (2010-2013), and 2) to reduce preventable complications during a transition from one care setting to another and reduce hospital readmissions by 20% during the same time period.  IMPAQ’s is responsible for developing an organizational evaluation tool (OET) addressing various “Areas of Focus” with regard to patient safety. The areas of focus will include patient safety culture, measurement of health care processes and outcomes, infection control, procedures, risk/capital, medications, nursing practices, communication, and purchasing practices. The OET will be used to: 1) assess current practices and capabilities in hospitals; 2) provide guidance to Hospital Engagement Contractors (HECs) regarding areas of focus; 3) provide feedback to hospital boards and hospital leadership around the current state of practices and identify organization specific needs; and 4) provide hospitals with baseline data and allow them to track their progress over time.

Community Colleges as the Classroom for America’s Workforce: Best Practices and Lessons Learned from the Field in Working with the Public Workforce System

The purpose of this project is to study a sample of community colleges to identify challenges, promising strategies and best practices regarding retraining America’s workforce for in-demand industries.  Specific tasks include:

• Identifying sample community college sites
• Conducting site visits, which include in-depth interviews and focus groups with stakeholders
• Collecting additional data such as web-based surveys, Workforce Investment Act Standardized Record Data (WIASRD), data from the Federal Research and Evaluation Database (FRED), evaluation data from the High Growth Job Training Initiative and Community-Based Job Training Grants, and data from the IMPAQ’s completed WIA Net-Impact Evaluation
• Conducting qualitative and quantitative data analysis to provide a comprehensive understanding of the interaction between community colleges and local workforce development systems.

The final deliverable for this project is a Final Report summarizing the results of the analysis and providing appropriate recommendations to DOL.

Longitudinal Program Evaluation of HHS Healthcare Associated Infections (HAI) National Action Plan (NAP)

In response to the preventable toll that healthcare-associated infections (HAIs) have had on human life and healthcare expenditures, the Department of Health and Human Services (HHS) released the HHS Draft National Action Plan (NAP) to Prevent Healthcare-Associated Infections in January 2009.  Congress responded to the Draft NAP and funded operative divisions of HHS to begin the implementation of the plan with funds included in the FY 09 appropriations  and funds from The American Recovery and Reinvestment Act of 2009 (ARRA).

During this project, IMPAQ will develop and implement an iterative longitudinal comprehensive program evaluation framework of the Healthcare Associated Infections National Action Plan (HAI NAP).  This evaluation will employ the Context-Input-Process-Product (CIPP) evaluation model to assess overall initiative impacts, outcomes, and the adoption and diffusion of HAI interventions using both qualitative and quantitative assessment approaches.  The results of the ongoing evaluations will be used to provide feedback that will guide improvements to the ongoing and future programs and initiatives of the HAI NAP. Specifically, we will look at the leadership and strategy of the HAI NAP, knowledge development, infrastructure development, HAI prevention practice adoption, and the effects on stakeholders. Thus, this project will involve interviews with agency leaders, key stakeholders, and NAP work group members; document and literature reviews; review of HHS data systems; online surveys; and focus groups.

Market Research, Environmental Scan and Ethnographic Research on the Use of Communication Technologies and Digital Media Related to Nutrition and Physical Activity

The U.S. Department of Agriculture (USDA) Food and Nutrition Service (FNS) is interested in improving understanding of nutrition and choices related to nutrition and physical activity among school-aged children.  IMPAQ, along with our subcontractor Samuels and Associates, will conduct three tasks in this project.  First, a summary of existing literature and market research will be prepared.  Second, three environmental scans will be conducted:  (a) identifying nutrition education programs (both in school settings and in the broader community), (b) examining use of the USDA Dietary Guidelines for Americans and MyPyramid, and (c) assessing the use of social networking, computer/video games and other digital technology to promote sound nutrition and physical activity.  Finally, ethnographic research involving interviews with up to 30 key informants will be conducted, including 6 in-person site visits.

Communication Dissemination Evaluation

Process Evaluation for the National Reemployment Technical Assistance Initiative

Evaluation Synthesis

Applying Lessons Learned in Community Collaboration to Health IT

To augment AHRQ’s existing health IT research, this project focuses on stakeholder collaboration and how models of community planning and development, both in healthcare and other industries, can be used for establishing sustainable Regional Health Information Organizations (RHIO) and health information exchange (HIE) activities. In particular, the project seeks to identify characteristics that distinguish successful and unsuccessful community efforts to collaborate on shared benefits, which can then be replicated to support RHIO and HIE development.  For this project, IMPAQ is:

• Conducting a comprehensive environmental scan of the literature related to community planning and development, and RHIO and HIE initiatives
• Convening an “Innovation Meeting” comprised of RHIO and HIE leaders, and national and international community planning experts who will assess the applicability of community collaboration models to sustainable RHIOs and HIE, including identifying best practices, illustrative models, and case studies
  
• Preparing a final report synthesizing results and findings from the environmental scan and “Innovation Meeting”, as well as presenting recommendations for ongoing research, implementation, and policy work in the RHIO and HIE fields
  
• Developing, executing, and evaluating a communication/dissemination plan that will drive awareness and integration of project findings and recommendations into practice at national, state, and local levels