Home / Project Showcase / Process Evaluations and Qualitative Research ProjectsView Printable Version
Project Showcase

Process Evaluations and Qualitative Research Projects

Process Evaluations and Qualitative Research Projects

Featured Project:  
Effective Health Care Program Governance & National Priority-Setting: An Appreciative Inquiry & Evaluation Study

Client:  Agency for Healthcare Research and Quality,
            U.S. Department of Health and Human Services
 
The purpose of this project is to develop a conceptual framework and formalized governance structure for the Agency for Healthcare Research and Quality’s (AHRQ) Effective Health Care (EHC) Program.  Specific tasks include:
  • The conduct of interviews, case studies, and an Appreciative Inquiry Workshop with EHC program staff and stakeholders
  • The design and administration of a web-based survey to gather quantitative inputs from stakeholders
  • The conduct of a cross-organization comparison of approaches with international organizations engaged in a similar mission to the EHC program.
 
The final deliverable for this project is a set of alternative roadmaps that AHRQ can use to develop a revised, formalized governance structure to lead the EHC program into the future.
 

Evaluation of American Recovery and Reinvestment Act (ARRA) Comparative Effectiveness Research Dissemination

Client: Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services

IMPAQ is conducting an evaluation of the effectiveness of the various dissemination and implementation approaches by AHRQ’s Office of Communications and Knowledge Transfer for the various comparative effectiveness research (CER) products prepared as part of the Effective Healthcare Program (EHC). Phase 1 of the task order is a 3-year ARRA project to evaluate the effectiveness of four different dissemination methods:

  • A publicity center that disseminates EHC findings nationwide through marketing/media efforts (i.e., press releases, audio podcasts, videos), organizational partnerships, and “Virtual Centers” on leading national consumer and clinician Health-related Websites.
  • Regional offices that develop partnerships with health care organizations, businesses, unions, consumer and other organizations to create and develop ongoing dissemination partnerships and to conduct outreach to their members.
  • Online continuing education activities for new target audiences, including pharmacists, nurses, nurse practitioners, physician assistants, and other allied health professionals.
  •  Academic detailing where trained clinicians conduct in-person educational meetings with other individual clinicians, clinician groups, and health plan formularies across the country.

IMPAQ, and its subcontractor Battelle, will determine the level of increased awareness, understanding, use, and benefits of CER among target audiences by collecting and analyzing quarterly data from the four dissemination contractors and conducting annual mail surveys of clinicians, telephone surveys of consumers, focus groups of targeted clinician and consumer audiences, and one-on-one interviews with other targeted audiences.  We will evaluate the dissemination contractors’ initiatives at midpoint and at the end of the contract to inform AHRQ of results and, in conjunction with AHRQ, recommend adjustments to their initiatives.

Green Jobs and Healthcare Implementation Study

Client: Employment and Training Administration, U.S. Department of Labor

IMPAQ is leading an implementation evaluation of four grant programs awarded with American Recovery and Reinvestment Act funds by the Department of Labor in early 2010, including:

  • Energy Training Partnership Grants
  • Pathways out of Poverty Grants
  • State Energy Sector Partnership and Training Grants
  •  Healthcare and High Growth Training Grants

Together these programs include over 150 projects with total award amounts of nearly $665 million dollars, with wide variation in grantee organizations and populations served.  The programs focus on workforce development and training to prepare workers for careers in various industries related to renewable energy and energy efficiency, healthcare, or other emerging industries.  IMPAQ is gathering information to assess the implementation of these grants in areas including program context, design and goals, recruitment, training, support, partnerships, proposed program outcomes, and program sustainability.  IMPAQ is reviewing site documentation, conducting site visits to select grantees, conducting a survey of grantees, and conducting focus groups with participants.  IMPAQ will produce a report to address key research questions related to the implementation of grant programs in these industry sectors.

Community Colleges as the Classroom for America’s Workforce: Best Practices and Lessons Learned from the Field in Working with the Public Workforce System

Client:  Center for Program Planning and Results of the Office of the Assistant Secretary for Administration and Management (OASAM), U.S. Department of Labor

The purpose of this project is to study a sample of community colleges to identify challenges, promising strategies and best practices regarding retraining America’s workforce for in-demand industries.  Specific tasks include:

  • Identifying sample community college sites
  • Conducting site visits, which include in-depth interviews and focus groups with stakeholders
  • Collecting additional data such as web-based surveys, Workforce Investment Act Standardized Record Data (WIASRD), data from the Federal Research and Evaluation Database (FRED), evaluation data from the High Growth Job Training Initiative and Community-Based Job Training Grants, and data from the IMPAQ’s completed WIA Net-Impact Evaluation
  • Conducting qualitative and quantitative data analysis to provide a comprehensive understanding of the interaction between community colleges and local workforce development systems.

The final deliverable for this project is a Final Report summarizing the results of the analysis and providing appropriate recommendations to DOL.

Longitudinal Program Evaluation of HHS Healthcare Associated Infections (HAI) National Action Plan (NAP)

Client: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services

In response to the preventable toll that healthcare-associated infections (HAIs) have had on human life and healthcare expenditures, the Department of Health and Human Services (HHS) released the HHS Draft National Action Plan (NAP) to Prevent Healthcare-Associated Infections in January 2009.  Congress responded to the Draft NAP and funded operative divisions of HHS to begin the implementation of the plan with funds included in the FY 09 appropriations  and funds from The American Recovery and Reinvestment Act of 2009 (ARRA).

During this project, IMPAQ will develop and implement an iterative longitudinal comprehensive program evaluation framework of the Healthcare Associated Infections National Action Plan (HAI NAP).  This evaluation will employ the Context-Input-Process-Product (CIPP) evaluation model to assess overall initiative impacts, outcomes, and the adoption and diffusion of HAI interventions using both qualitative and quantitative assessment approaches.  The results of the ongoing evaluations will be used to provide feedback that will guide improvements to the ongoing and future programs and initiatives of the HAI NAP. Specifically, we will look at the leadership and strategy of the HAI NAP, knowledge development, infrastructure development, HAI prevention practice adoption, and the effects on stakeholder. Thus, this project will involve interviews with agency leaders, key stakeholders, and NAP work group members; document and literature reviews; review of HHS data systems; online surveys; and focus groups.

Market Research, Environmental Scan and Ethnographic Research on the Use of Communication Technologies and Digital Media Related to Nutrition and Physical Activity

Client:  Food and Nutrition Service, U.S. Department of Agriculture

The U.S. Department of Agriculture (USDA) Food and Nutrition Service (FNS) is interested in improving understanding of nutrition and choices related to nutrition and physical activity among school-aged children.  IMPAQ, along with our subcontractor Samuels and Associates, will conduct three tasks in this project.  First, a summary of existing literature and market research will be prepared.  Second, three environmental scans will be conducted:  (a) identifying nutrition education programs (both in school settings and in the broader community), (b) examining use of the USDA Dietary Guidelines for Americans and MyPyramid, and (c) assessing the use of social networking, computer/video games and other digital technology to promote sound nutrition and physical activity.  Finally, ethnographic research involving interviews with up to 30 key informants will be conducted, including 6 in-person site visits.

Communication Dissemination Evaluation

Client: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services
 
The purpose of this project was to conduct a review and comprehensive evaluation of the current dissemination mechanisms employed by the various units of the Agency for Healthcare Research and Quality (AHRQ) to assess their types, methods, and levels of effectiveness.  IMPAQ conducted interviews with AHRQ Center Directors and Portfolio Leads to evaluate the effectiveness of the Office of Communication and Knowledge Transfer’s support of communication dissemination processes.  IMPAQ also investigated public and private sector use of “Web 2.0” communication mechanisms on health topics, and explored opportunities for AHRQ to establish new and innovative approaches to the dissemination of health-related research and best practices information.  IMPAQ also conducted quick-turn around market and messaging research, planning, scheduling and conducting focus groups within one month to test messaging related to Comparative Effectiveness Research.
 

Process Evaluation for the National Reemployment Technical Assistance Initiative

Client: Employment and Training Administration, U.S. Department of Labor 
 
As part of the overall National Reemployment Technical Assistance Initiative, IMPAQ conducted a process evaluation of the reemployment practices currently being used by local workforce investment programs at six selected sites across the U.S. that have been recipients of either a Workforce Innovation in Regional Economic Development (WIRED) or Regional Innovation (RIG) grant from the U.S. Department of Labor.  IMPAQ conducted comprehensive site visits, including the use of structured interviews and focus groups, to evaluate current approaches to program integration, use of labor market information and assessment tools, and the strategies employed by the sites with the goal of identifying best practices which can be shared with other workforce investment agencies throughout the U.S. 
 

Evaluation Synthesis

Client: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services 
 
The purpose of this project is to conduct a synthesis of selected evaluations completed for AHRQ since 2004 to determine the organizational and management elements common across the evaluated programs, initiatives, and products that affected or contributed to different levels of success in achieving their goals.  These factors may include, among others, the Agency’s level of support, clarity of goals and objectives, the design and implementation of the evaluation, appropriateness and level of involvement of AHRQ staff, level of demand for the information and products produced, ability to disseminate and spread uptake of positive findings and products, as well as external factors beyond the Agency’s control.  
 
To conduct this study, IMPAQ staff developed a set of criteria (success factors) which were applied in reviewing 25 evaluation and research reports conducted by AHRQ and its contractors.  IMPAQ also conducted individual interviews with senior AHRQ staff and contractor staff to identify the factors that contributed to the success or lack of success of the evaluation effort.  IMPAQ prepared a synthesis report, which identifies the key success factors common across the evaluations that were presented to AHRQ senior management for use in designing and implementing future research projects.
 

Applying Lessons Learned in Community Collaboration to Health IT

Client: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services

To augment AHRQ’s existing health IT research, this project focuses on stakeholder collaboration and how models of community planning and development, both in healthcare and other industries, can be used for establishing sustainable Regional Health Information Organizations (RHIO) and health information exchange (HIE) activities. In particular, the project seeks to identify characteristics that distinguish successful and unsuccessful community efforts to collaborate on shared benefits, which can then be replicated to support RHIO and HIE development.  For this project, IMPAQ is:

  • Conducting a comprehensive environmental scan of the literature related to community planning and development, and RHIO and HIE initiatives
  • Convening an “Innovation Meeting” comprised of RHIO and HIE leaders, and national and international community planning experts who will assess the applicability of community collaboration models to sustainable RHIOs and HIE, including identifying best practices, illustrative models, and case studies
  • Preparing a final report synthesizing results and findings from the environmental scan and “Innovation Meeting”, as well as presenting recommendations for ongoing research, implementation, and policy work in the RHIO and HIE fields
  • Developing, executing, and evaluating a communication/dissemination plan that will drive awareness and integration of project findings and recommendations into practice at national, state, and local levels
    •  

Child Care Services Program Assessment and Benchmarking Study

Client:  National Institutes of Health, U.S. Department of Health and Human Services
 
IMPAQ conducted an assessment of the National Institutes of Health’s childcare service program to determine the effectiveness of the existing program, as well as document areas of improvement for ensuring its continued competitiveness with comparable organizations.  As part of this project, IMPAQ conducted interviews with NIH senior officials, performed a benchmarking study of ten government and private sector organizations recognized as being “best in class,” and administering a web-based survey to NIH employees to assess their satisfaction with the current programs and services being offered.  The results of the study were used to develop program goals and recommendations for future improvements to NIH’s childcare services program.
 
Read the Final Report: