Impact Evaluation Studies Projects
Featured Project: Client: Centers for Medicare & Medicaid Services, U.S. Department of Health and Human Services
Evaluation of the Medicare Acute Care Episode (ACE) Demonstration
In early 2009, the Centers for Medicare & Medicaid Services implemented the Acute Care Episode (ACE) Demonstration to test the use of a global payment for an episode of care as an alternative approach to payment for service delivery under traditional Medicare fee-for-service. The ACE demonstration seeks to align financial incentives within health care groups to provide quality care according to best practices at a savings to Medicare. Selected applicants, specifically physician-hospital organizations (PHOs), routinely provide at least one of the two main procedures included in the demonstration: Hip/knee replacement surgery; and/or Coronary artery bypass graft (CABG) surgery.
IMPAQ International, LLC is assisting CMS in the design and implementation of an independent evaluation of the feasibility and cost effectiveness of the bundled payment methodology and the improvement in quality of care and other benefits to Medicare beneficiaries. Key analytical tasks include defining appropriate comparison populations, collecting primary and secondary data, defining and implementing methods for (quality) outcomes evaluation and budget neutrality, and assessing shared savings as addressed in the legislation.
For this contract, IMPAQ International is conducting a rigorous, random-assignment evaluation of OSHA’s site specific targeting (SST) program. The purpose of the evaluation is to estimate the impact of both High Rate Letters and SST inspections on injury/illness rates and compliance outcomes for establishments targeted by the SST program. The specific research questions to be answered are: 1) what is the effect of a High Rate letter with no inspection on employer outcomes; 2) what is the incremental effect of an inspection on employer outcomes; 3) what is the total effect of the SST program (High Rate letter and inspection) on employer outcomes; 4) do program impacts vary between employers on OSHA’s primary list and OSHA’s secondary list; 5) do program impacts vary by industry, location, or other employer characteristics; and 6) do program impacts vary by the timeline between interventions?
To conduct the evaluation, IMPAQ gathers three types of data: administrative data from OSHA on injury/illness rates and the outcomes of SST inspections; 2) Dun & Bradstreet data on employer characteristics, and 3) qualitative data from document reviews and interviews with OSHA staff. Using these data, IMPAQ is producing descriptive and multivariate analyses of the characteristics of employers included in the evaluation. Additionally, IMPAQ is generating estimates of the impacts of both High Rate letters and inspections on outcome variables of interest. The qualitative data is used in a process evaluation that focuses on understanding the processes used to implement the design of the evaluation, the data protocols established to collect inspection data, best practices in the inspection process, and other relevant issues.
For this project, IMPAQ International is conducting a quasi-experimental evaluation of DOL’s use of remedies under the Fair Labor Standards Act (FLSA) to study the differential responses of employers to enforcement actions taken by DOL. The key outcomes of interest are both the future compliance of the investigated firms, but also the compliance rates of similar firms in the same industry and/or geographic area (i.e. the ripple effect). Historical WHD administrative data, together with employer matched Dun and Bradstreet (D&B) data, provides data for an IMPAQ’s white paper examining the use of remedies across industries and geographic areas. The paper will also examine recidivism rates among firms with previous violations.
The project also includes a process evaluation to assess employer attitudes and experiences with FLSA enforcement and examine the FLSA enforcement approaches used across WHD district offices. The process evaluation enhances the findings of the quasi-experimental study by providing insights on how employers respond to different FLSA enforcement practices.
Pursuant to legislation passed in 2010, Maryland established the Maryland Multi-payer Patient Centered Medical Home Program (MMPP). The Maryland Health Care Commission (MHCC) is providing 53 selected practices with education and technical assistance to help them transform into PCMHs. MHCC is interested in learning whether the MMPP can improve health care quality and, thereby, health outcomes, while reducing costs of care. The IMPAQ team is analyzing whether the MMPP improves health care quality and health outcomes while reducing the cost of care and assess the process of transforming practices into primary care medical homes. IMPAQ is using a pre/post, mixed methods evaluation design that consists of three parts: assessment of access, quality, utilization, disparities and cost outcomes, using administrative data; analysis of implementation and practice transformation, using interviews, site visits, and administrative data; and assessment of satisfaction among patients and providers, using existing survey instruments.
The evaluation of Project GATE was based on:
1) data collected during site-visits,
2) interviews conducted with treatment and control group members conducted at 6 months, 18 months, and 60 months following random assignment, and
3) administrative data obtained from participating states.
IMPAQ implemented both an impact and a process evaluation of the effectiveness of the GATE service model. In its final impact evaluation report, IMPAQ staff developed policy recommendations regarding the national implementation of the GATE model in the U.S.
Read the Project GATE Final Report:
Following implementation of Project GATE (Growing America Through Entrepreneurship), the Employment and Training Administration, U.S. Department of Labor, awarded a series of grants known as GATE II. Four states were awarded grants to implement the GATE model of entrepreneurship training: Virginia, North Carolina, Alabama, and Minnesota. IMPAQ International is conducting a three-year evaluation of the GATE II project. This evaluation will consist of an implementation analysis, investigating how states implemented GATE II and identifying best practices, as well as an impact analysis, identifying the impact of GATE II on the self-employment and labor market outcomes of participants.
The impact evaluation of GATE II is particularly interesting because the states have selected different target groups, different random assignment approaches to assign treatment and control group members, and different service models. One of the states, Minnesota, is not implementing a random assignment design. To evaluate program impacts in Minnesota, we plan to use a non-experimental impact evaluation method such as propensity score matching or regression discontinuity.
Evaluation of the Reemployment and Eligibility Assessment (REA) InitiativeClient: Employment and Training Administration, U.S. Department of Labor
This project, conducted for the U.S. Department of Labor, Employment and Training Administration, studied the implementation of the Reemployment and Eligibility Assessment (REA) Initiative and, in four states, its impact on Unemployment Insurance (UI) duration, exhaustion, overpayments, and return to work. Designed to reduce unemployment spells and increase savings for state UI Trust Funds, REA provided UI claimants with in-person interviews involving a review of 1) UI eligibility, 2) provision of labor market information, 3) development of a work-search plan, and 4) referral to training and/or other reemployment services, as necessary. This project resulted in a final report to Congress designed to provide decision-making information relevant to future funding of the REA Initiative and state funding allocations.
The Administration on Aging (AoA) currently funds Aging and Disability Resource Centers (ADRCs). AoA, in partnership with the Agency for Healthcare Research and Quality (AHRQ), has determined the need for a rigorous evaluation of ADRCs to determine their effectiveness in enhancing elders’ ability to access Long Term Services and Supports (LTSS). AoA has contracted with IMPAQ International to conduct a comprehensive evaluation of ADRCs, focusing on evaluating the effect of ADRCs on the experience of accessing LTSS and identifying community characteristics that facilitate ADRCs’ development. Currently, access to many states’ LTSS systems is fragmented, with providers offering programs in isolation. As states begin to transform their delivery of services from fragmented to fully integrated systems, one vehicle for achieving more integrated access to LTSS is the ADRC grant program. IMPAQ will assess ADRCs through a rigorous evaluation design, quantitative process evaluation, and outcome evaluation. The evaluation will involve working closely with program stakeholders at the state and local levels and utilizing existing claims and administrative data sources, as well as documents from ADRC’s technical assistance provider in the evaluation, to ensure the feasibility and validity of the overall evaluation plan.
Evaluation of the Community-Based Alternatives to Psychiatric Residential Treatment Facilities Demonstration Program
Core services, including outreach, job search and placement assistance, and labor market information
Intensive services, involving more comprehensive assessments, development of individual employment plans, and counseling and career planning.
Training services, occupational and basic skills training that link customers to job opportunities in their communities.
The Administration on Aging (AoA) currently funds three programs aimed at helping elders remain in the community, and out of nursing home facilities, as long as possible: Aging and Disability Resource Centers (ADRCs), Community Living Programs (CLPs), and Evidence-Based Disease and Disability Prevention Programs (EBDDPs). Each of these programs targets a slightly different sector of the aging population and utilizes unique approaches to meet the overarching goal of keeping elders in the community. AoA, in partnership with the Agency for Healthcare Research and Quality (AHRQ), has determined the need for a rigorous evaluation of these interrelated programs to determine their effectiveness in enhancing elders’ abilities to remain in the community, deter placement in nursing facilities, and self manage chronic conditions.
IMPAQ has been tasked with designing a comprehensive evaluation of these three AoA programs, focusing on process and impact. The process evaluation design addresses both the research interests surrounding organizational/structural aspects and the service components of each program. The impact evaluation addresses the effect of each program on beneficiary outcomes, health care utilization, and costs. In addition, the evaluation identifies mechanisms to implement continuous quality improvement strategies within each program. IMPAQ is working closely with program stakeholders at the state and local levels and utilizing existing claims and administrative data sources in the development of this design, to ensure feasibility and validity of the overall evaluation plan.
The scope of this project is to design a rigorous evaluation of the impact of OSHA enforcement procedures on the future safety and health compliance of businesses. The evaluation will provide answers to questions in four general research areas:
- What is the overall impact of OSHA enforcement procedures on future employer compliance?
- Which employer characteristics are strong indicators of future compliance?
- What are the best practices and measures to reduce future occupational injuries and illnesses among employers?
- How can OSHA best routinely measure and report the effectiveness of its compliance programs?
To design the evaluation, IMPAQ will review critical documentation and information on OSHA’s enforcement procedures. IMPAQ will then receive relevant OSHA inspection and enforcement data files, develop descriptive statistics, and conduct a review and analysis of the data. IMPAQ will examine rates and patterns of compliance and recidivism as related to variables such as employer characteristics, OSHA penalties, and time between inspections. IMPAQ will use these analyses to provide initial information to OSHA, and inform the evaluation design. Simultaneously, IMPAQ will interview key OSHA staff about enforcement procedures and data files. The final deliverable will be a rigorous evaluation design that OSHA can implement to determine the impact of enforcement activities on subsequent employer safety and health compliance.
Within OASAM, the Center for Program Planning and Results (CPPR) funds and manages evaluations of select DOL agencies and programs in support of the Government Performance and Results Act of 1993 (GPRA). The goal of this act is to improve the effectiveness of DOL’s agencies and programs. One such agency, the Wage and Hour Division (WHD), is responsible for enforcing a variety of federal labor laws regarding the minimum wage, family and medical leave, migrant workers, lie detector tests, temporary worker programs, and prevailing wages for government service and construction contracts. WHD selects employers to be investigated for possible violations of these federal labor laws, and conducts investigations to determine actual violations. A primary goal of WHD is to reduce the rate of recidivism among employers who have been investigated.
IMPAQ International is conducting a one-year study to design a rigorous random assignment evaluation which would allow estimation of the impact of a variety of factors on compliance and the recidivism rate of employers. These factors include WHD practices and procedures regarding selection and investigation of employers; employer characteristics; and other factors. At the conclusion of this project, IMPAQ will deliver an evaluation design report and recommendations for improvements in procedures within WHD.
This quasi-experimental impact evaluation examined the effectiveness of active labor programs in Romania implemented under the World Bank Employment and Social Assistance Project. The primary goal of this project was to provide information on the net impact of selected Active Labor Market Programs (ALMPs) with different types of unemployed clients, and under varying labor market conditions, and to improve their targeting and cost-effectiveness.
To evaluate the net impact of ALMP services in Romania, IMPAQ employed a quasi-experimental design. A survey was administered to over 4,000 respondents, including a sample of over 2,000 ALMP participants and a similar comparison sample of individuals who did not participate in ALMP. Based on the impact evaluation comparing economic outcomes for a sample of program participants and non-participants, IMPAQ developed labor market policy recommendations, including recommendations for improved targeting.
Impact of the Millennium Challenge Accounts Roads Projects (Burkina Faso and Senegal)Client: Millennium Challenge Corporation
IMPAQ is currently evaluating the impact of the Millennium Challenge Accounts Roads Projects in Burkina Faso and Senegal. Those Millennium Challenge Corporation funded interventions are intended to improve primary and rural roads deemed critical to stimulate economic growth. The five-year impact evaluation project will measure how the Millennium Challenge Corporation's investments benefit families and private firms' income, agricultural productivity, and reduced transportation costs; all factors inducing economic growth.