Home / Project Showcase / Impact Evaluation Studies ProjectsView Printable Version
Project Showcase

Impact Evaluation Studies Projects

Impact Evaluation Studies Projects

 

Featured Project:
Evaluation of the Medicare Acute Care Episode (ACE) Demonstration

Client: Centers for Medicare & Medicaid Services, U.S. Department of Health and Human Services 

In early 2009, the Centers for Medicare & Medicaid Services implemented the Acute Care Episode (ACE) Demonstration to test the use of a global payment for an episode of care as an alternative approach to payment for service delivery under traditional Medicare fee-for-service. The ACE demonstration seeks to align financial incentives within health care groups to provide quality care according to best practices at a savings to Medicare. Selected applicants, specifically physician-hospital organizations (PHOs), routinely provide at least one of the two main procedures included in the demonstration: Hip/knee replacement surgery; and/or Coronary artery bypass graft (CABG) surgery.

IMPAQ International, LLC is assisting CMS in the design and implementation of an independent evaluation of the feasibility and cost effectiveness of the bundled payment methodology and the improvement in quality of care and other benefits to Medicare beneficiaries. Key analytical tasks include defining appropriate comparison populations, collecting primary and secondary data, defining and implementing methods for (quality) outcomes evaluation and budget neutrality, and assessing shared savings as addressed in the legislation.

The Evaluation of the Occupational Safety and Health Administration Site Specific Targeting Program

Client: Occupational Safety & Health Administration, U.S. Department of Labor

For this contract, IMPAQ International is conducting a rigorous, random-assignment evaluation of OSHA’s site specific targeting (SST) program. The purpose of the evaluation is to estimate the impact of both High Rate Letters and SST inspections on injury/illness rates and compliance outcomes for establishments targeted by the SST program. The specific research questions to be answered are: 1) what is the effect of a High Rate letter with no inspection on employer outcomes; 2) what is the incremental effect of an inspection on employer outcomes; 3) what is the total effect of the SST program (High Rate letter and inspection) on employer outcomes; 4) do program impacts vary between employers on OSHA’s primary list and OSHA’s secondary list; 5) do program impacts vary by industry, location, or other employer characteristics; and 6) do program impacts vary by the timeline between interventions?

To conduct the evaluation, IMPAQ gathers three types of data: administrative data from OSHA on injury/illness rates and the outcomes of SST inspections; 2) Dun & Bradstreet data on employer characteristics, and 3) qualitative data from document reviews and interviews with OSHA staff. Using these data, IMPAQ is producing descriptive and multivariate analyses of the characteristics of employers included in the evaluation.  Additionally, IMPAQ is generating estimates of the impacts of both High Rate letters and inspections on outcome variables of interest. The qualitative data is used in a process evaluation that focuses on understanding the processes used to implement the design of the evaluation, the data protocols established to collect inspection data, best practices in the inspection process, and other relevant issues.

The Evaluation of the Wage and Hour Division's Use of Remedies

Client:  Wage and Hour Division, U.S. Department of Labor

For this project, IMPAQ International is conducting a quasi-experimental evaluation of DOL’s use of remedies under the Fair Labor Standards Act (FLSA) to study the differential responses of employers to enforcement actions taken by DOL. The key outcomes of interest are both the future compliance of the investigated firms, but also the compliance rates of similar firms in the same industry and/or geographic area (i.e. the ripple effect).  Historical WHD administrative data, together with employer matched Dun and Bradstreet (D&B) data, provides data for an IMPAQ’s white paper examining the use of remedies across industries and geographic areas. The paper will also examine recidivism rates among firms with previous violations.

The project also includes a process evaluation to assess employer attitudes and experiences with FLSA enforcement and examine the FLSA enforcement approaches used across WHD district offices. The process evaluation enhances the findings of the quasi-experimental study by providing insights on how employers respond to different FLSA enforcement practices.

Evaluation of the Maryland Multi-Payer Patient Centered Medical Home Program

Client: Maryland Health Care Commission

Pursuant to legislation passed in 2010, Maryland established the Maryland Multi-payer Patient Centered Medical Home Program (MMPP). The Maryland Health Care Commission (MHCC) is providing 53 selected practices with education and technical assistance to help them transform into PCMHs. MHCC is interested in learning whether the MMPP can improve health care quality and, thereby, health outcomes, while reducing costs of care. The IMPAQ team is analyzing whether the MMPP improves health care quality and health outcomes while reducing the cost of care and assess the process of transforming practices into primary care medical homes. IMPAQ is using a pre/post, mixed methods evaluation design that consists of three parts:  assessment of access, quality, utilization, disparities and cost outcomes, using administrative data; analysis of implementation and practice transformation, using interviews, site visits, and administrative data; and assessment of satisfaction among patients and providers, using existing survey instruments.

Growing America Through Entrepreneurship (Project GATE)

Client:  Employment and Training Administration, U.S. Department of Labor
 
Project GATE, an experimental design demonstration and evaluation project initiated by the U.S. Department of Labor, was designed to investigate the impacts of providing microenterprise start-up services to interested individuals.  Applicants to Project GATE were randomly assigned to either a program group or a control group, with the participants offered services, such as: assessment of business needs, classroom training, one-on-one technical assistance, and assistance in applying for business financing. 
 
The evaluation of Project GATE was based on:

1) data collected during site-visits,
2) interviews conducted with treatment and control group members conducted at 6 months, 18 months, and 60 months following random assignment, and
3) administrative data obtained from participating states.

IMPAQ implemented both an impact and a process evaluation of the effectiveness of the GATE service model.  In its final impact evaluation report, IMPAQ staff developed policy recommendations regarding the national implementation of the GATE model in the U.S.

Read the Project GATE Final Report: 

Evaluation of the Project GATE II Grants

Client: Employment and Training Administration, U.S. Department of Labor

Following implementation of Project GATE (Growing America Through Entrepreneurship), the Employment and Training Administration, U.S. Department of Labor, awarded a series of grants known as GATE II.  Four states were awarded grants to implement the GATE model of entrepreneurship training:  Virginia, North Carolina, Alabama, and Minnesota.   IMPAQ International is conducting a three-year evaluation of the GATE II project.  This evaluation will consist of an implementation analysis, investigating how states implemented GATE II and identifying best practices, as well as an impact analysis, identifying the impact of GATE II on the self-employment and labor market outcomes of participants. 

The impact evaluation of GATE II is particularly interesting because the states have selected different target groups, different random assignment approaches to assign treatment and control group members, and different service models.  One of the states, Minnesota, is not implementing a random assignment design.  To evaluate program impacts in Minnesota, we plan to use a non-experimental impact evaluation method such as propensity score matching or regression discontinuity.

Evaluation of the Reemployment and Eligibility Assessment (REA) Initiative

Client: Employment and Training Administration, U.S. Department of Labor

This project, conducted for the U.S. Department of Labor, Employment and Training Administration, studied the implementation of the Reemployment and Eligibility Assessment (REA) Initiative and, in four states, its impact on Unemployment Insurance (UI) duration, exhaustion, overpayments, and return to work.  Designed to reduce unemployment spells and increase savings for state UI Trust Funds, REA provided UI claimants with in-person interviews involving a review of 1) UI eligibility, 2) provision of labor market information, 3) development of a work-search plan, and 4) referral to training and/or other reemployment services, as necessary.  This project resulted in a final report to Congress designed to provide decision-making information relevant to future funding of the REA Initiative and state funding allocations.

Evaluation of Aging and Disability Resource Centers (ADRCs)

Client: Administration on Aging, U.S. Department of Health and Human Services

The Administration on Aging (AoA) currently funds Aging and Disability Resource Centers (ADRCs).  AoA, in partnership with the Agency for Healthcare Research and Quality (AHRQ), has determined the need for a rigorous evaluation of ADRCs to determine their effectiveness in enhancing elders’ ability to access Long Term Services and Supports (LTSS). AoA has contracted with IMPAQ International to conduct a comprehensive evaluation of ADRCs, focusing on evaluating the effect of ADRCs on the experience of accessing LTSS and identifying community characteristics that facilitate ADRCs’ development.  Currently, access to many states’ LTSS systems is fragmented, with providers offering programs in isolation. As states begin to transform their delivery of services from fragmented to fully integrated systems, one vehicle for achieving more integrated access to LTSS is the ADRC grant program. IMPAQ will assess ADRCs through a rigorous evaluation design, quantitative process evaluation, and outcome evaluation. The evaluation will involve working closely with program stakeholders at the state and local levels and utilizing existing claims and administrative data sources, as well as documents from ADRC’s technical assistance provider in the evaluation, to ensure the feasibility and validity of the overall evaluation plan.

Evaluation of the Community-Based Alternatives to Psychiatric Residential Treatment Facilities Demonstration Program

Client:  Centers for Medicare & Medicaid Services, U.S. Department of Health & Human Services
 
IMPAQ is currently conducting a longitudinal evaluation of the Community-Based Alternatives to Psychiatric Residential Treatment Facilities (PRTF) demonstration program.  This demonstration provides grants to ten states to provide home and community-based services (HCBS) to youth as alternatives to PRTFs under a 1915(c) waiver.  The overall goal of the evaluation project is to determine if the provision of HCBS to youth under this demonstration results in the maintenance or improvement in a child’s functional status and costs no more than anticipated aggregate PRTF expenditures in the absence of the demonstration.  For this project, IMPAQ has designed a Minimum Data Set (MDS), provided technical assistance to states in establishing data collection mechanisms, and processed and conducted statistical analyses on the data collected.  Once all of the data have been collected and processed, IMPAQ will analyze the results and prepare a report summarizing the impact of the demonstration program on meeting its objectives.
 
Read the CBA Implementation Status Report:

Workforce Investment Act Net Impact Evaluation

Client: Employment and Training Administration, U.S. Department of Labor
 
IMPAQ conducted a nonexperimental net impact evaluation of the Workforce Investment Act (WIA) program to estimate the average impact of the program across individuals, that is, the difference in average earnings or employment following program participation, as compared to the average if these individuals had not participated in the program.  The study focused on the Adult and Dislocated Worker programs which offer three levels of service:
  • Core services, including outreach, job search and placement assistance, and labor market information
  • Intensive services, involving more comprehensive assessments, development of individual employment plans, and counseling and career planning.
  • Training services, occupational and basic skills training that link customers to job opportunities in their communities.
To support this study, IMPAQ collected and processed administrative data from 12 states, covering approximately 160,000 WIA participants and nearly 3 million comparison group members.  IMPAQ employed state-of-the-art nonexperimental methods to compare WIA program participants with matched comparison groups of individuals who either did not receive WIA services or did not receive WIA training services.
 
Read the WIA Net Impact Evaluation Final Report:
 
WIA Net Impact Evaluation Final Report

Evaluation of the Senior Risk Reduction Demonstration Under Medicare

Client:  Centers for Medicare & Medicaid Services, U.S. Department of Health and Human Services
 
The purpose of this project is to evaluate the effectiveness and impact of the Senior Risk Reduction Demonstration (SRRD), which is being sponsored by CMS.  Under the SRRD, demonstration vendors are providing risk reduction services to a random sample of non-institutionalized Medicare fee-for-service beneficiaries, aged 67 – 74, and enrolled in Medicare Parts A and B – both nationwide and from communities which have exemplary information and referral and assistance service programs for seniors. IMPAQ’s role on the project is to design and conduct a longitudinal impact evaluation of the SRRD, including the collection and analysis of Health Risk Appraisal (HRA) survey data, analysis of CMS Medicare claims data, and the conduct of site visits to assess program impacts and budget implications of the program. 

Designing an Evaluation of Three Administration on Aging (AoA) Programs

Client:  Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services

The Administration on Aging (AoA) currently funds three programs aimed at helping elders remain in the community, and out of nursing home facilities, as long as possible:  Aging and Disability Resource Centers (ADRCs), Community Living Programs (CLPs), and Evidence-Based Disease and Disability Prevention Programs (EBDDPs).  Each of these programs targets a slightly different sector of the aging population and utilizes unique approaches to meet the overarching goal of keeping elders in the community.  AoA, in partnership with the Agency for Healthcare Research and Quality (AHRQ), has determined the need for a rigorous evaluation of these interrelated programs to determine their effectiveness in enhancing elders’ abilities to remain in the community, deter placement in nursing facilities, and self manage chronic conditions.

IMPAQ has been tasked with designing a comprehensive evaluation of these three AoA programs, focusing on process and impact.  The process evaluation design addresses both the research interests surrounding organizational/structural aspects and the service components of each program.  The impact evaluation addresses the effect of each program on beneficiary outcomes, health care utilization, and costs.  In addition, the evaluation identifies mechanisms to implement continuous quality improvement strategies within each program. IMPAQ is working closely with program stakeholders at the state and local levels and utilizing existing claims and administrative data sources in the development of this design, to ensure feasibility and validity of the overall evaluation plan. 

Evaluating the Impact of OSHA’s Enforcement Interventions

Client:  Center for Program Planning and Results of the Office of the Assistant Secretary for Administration and Management (OASAM), U.S. Department of Labor

The scope of this project is to design a rigorous evaluation of the impact of OSHA enforcement procedures on the future safety and health compliance of businesses.  The evaluation will provide answers to questions in four general research areas:

  • What is the overall impact of OSHA enforcement procedures on future employer compliance?
  • Which employer characteristics are strong indicators of future compliance?
  • What are the best practices and measures to reduce future occupational injuries and illnesses among employers?
  • How can OSHA best routinely measure and report the effectiveness of its compliance programs?

To design the evaluation, IMPAQ will review critical documentation and information on OSHA’s enforcement procedures.  IMPAQ will then receive relevant OSHA inspection and enforcement data files, develop descriptive statistics, and conduct a review and analysis of the data.  IMPAQ will examine rates and patterns of compliance and recidivism as related to variables such as employer characteristics, OSHA penalties, and time between inspections.  IMPAQ will use these analyses to provide initial information to OSHA, and inform the evaluation design.  Simultaneously, IMPAQ will interview key OSHA staff about enforcement procedures and data files.  The final deliverable will be a rigorous evaluation design that OSHA can implement to determine the impact of enforcement activities on subsequent employer safety and health compliance.

Evaluating the Impact of the Wage & Hour Division’s Enforcement Procedures

Client: Office of the Assistant Secretary for Administration and Management (OASAM), U.S. Department of Labor

Within OASAM, the Center for Program Planning and Results (CPPR) funds and manages evaluations of select DOL agencies and programs in support of the Government Performance and Results Act of 1993 (GPRA).  The goal of this act is to improve the effectiveness of DOL’s agencies and programs.  One such agency, the Wage and Hour Division (WHD), is responsible for enforcing a variety of federal labor laws regarding the minimum wage, family and medical leave, migrant workers, lie detector tests, temporary worker programs, and prevailing wages for government service and construction contracts.  WHD selects employers to be investigated for possible violations of these federal labor laws, and conducts investigations to determine actual violations.  A primary goal of WHD is to reduce the rate of recidivism among employers who have been investigated.

IMPAQ International is conducting a one-year study to design a rigorous random assignment evaluation which would allow estimation of the impact of a variety of factors on compliance and the recidivism rate of employers.  These factors include WHD practices and procedures regarding selection and investigation of employers; employer characteristics; and other factors.  At the conclusion of this project, IMPAQ will deliver an evaluation design report and recommendations for improvements in procedures within WHD.

AMANECE Evaluation (Mexico)

Client:  Carso Health Institute, Mexico City, Mexico
 
IMPAQ recently conducted an evaluation of the implementation of the AMANECE project in Mexico, which applies an innovative prevention, promotion, and care model based on network connectivity and technology innovation aimed at a significant improvement of health care and a reduction of maternal and child mortality.  The AMANECE approach includes the use of technology (e.g., cell phones, computers) to connect pregnant women and health services providers and incorporates the perception of women not as an object of care, but rather as the subject and main player in each of their process stages.  For this project, IMPAQ prepared a quasi-experimental evaluation design, designed and administered a health survey to a group of women of child-bearing age and health services providers (health promoters and clinical staff), collected demographic and health outcome data on program participants, and analyzed the results to assess program impacts.

Impact of Active Labor Market Programs (Romania)

Client:  The World Bank
 
This quasi-experimental impact evaluation examined the effectiveness of active labor programs in Romania implemented under the World Bank Employment and Social Assistance Project.  The primary goal of this project was to provide information on the net impact of selected Active Labor Market Programs (ALMPs) with different types of unemployed clients, and under varying labor market conditions, and to improve their targeting and cost-effectiveness.
 
To evaluate the net impact of ALMP services in Romania, IMPAQ employed a quasi-experimental design.  A survey was administered to over 4,000 respondents, including a sample of over 2,000 ALMP participants and a similar comparison sample of individuals who did not participate in ALMP.  Based on the impact evaluation comparing economic outcomes for a sample of program participants and non-participants, IMPAQ developed labor market policy recommendations, including recommendations for improved targeting.

Impact of the Millennium Challenge Accounts Roads Projects (Burkina Faso and Senegal)  

Client: Millennium Challenge Corporation

IMPAQ is currently evaluating the impact of the Millennium Challenge Accounts Roads Projects in Burkina Faso and Senegal. Those Millennium Challenge Corporation funded interventions are intended to improve primary and rural roads deemed critical to stimulate economic growth. The five-year impact evaluation project will measure how the Millennium Challenge Corporation's investments benefit families and private firms' income, agricultural productivity, and reduced transportation costs; all factors inducing economic growth.