Impact Evaluation Studies Projects

Evaluation of the Project GATE II Grants

Following implementation of Project GATE (Growing America Through Entrepreneurship), awarded a series of grants known as GATE II.  Four states were awarded grants to implement the GATE model of entrepreneurship training:  Virginia, North Carolina, Alabama, and Minnesota.   IMPAQ International is conducting a three-year evaluation of the GATE II project.  This evaluation will consist of an implementation analysis, investigating how states implemented GATE II and identifying best practices, as well as an impact analysis, identifying the impact of GATE II on the self-employment and labor market outcomes of participants. 

The impact evaluation of GATE II is particularly interesting because the states have selected different target groups, different random assignment approaches to assign treatment and control group members, and different service models.  One of the states, Minnesota, is not implementing a random assignment design.  To evaluate program impacts in Minnesota, we plan to use a non-experimental impact evaluation method such as propensity score matching or regression discontinuity.   

Evaluation of the Reemployment and Eligibility Assessment (REA) Initiative

This project, conducted for the U.S. Department of Labor, Employment and Training Administration, will study the implementation of the Reemployment and Eligibility Assessment (REA) Initiative and, in four states, its impact on Unemployment Insurance (UI) duration, exhaustion, overpayments, and return to work.  Designed to reduce unemployment spells and increase savings for state UI Trust Funds, REA provides UI claimants with in-person interviews involving a review of 1) UI eligibility, 2) provision of labor market information, 3) development of a work-search plan, and 4) referral to training and/or other reemployment services, as necessary.  This project will result in a final report to Congress designed to provide decision-making information relevant to future funding of the REA Initiative and state funding allocations.

Evaluation of the Community-Based Alternatives to Psychiatric Residential Treatment Facilities Demonstration Program

Workforce Investment Act Net Impact Evaluation

• Core services, including outreach, job search and placement assistance, and labor market information
• Intensive services, involving more comprehensive assessments, development of individual employment plans, and counseling and career planning.
• Training services, occupational and basic skills training that link customers to job opportunities in their communities.

Evaluation of the Senior Risk Reduction Demonstration Under Medicare

Evaluation of the Medicare Acute Care Episode (ACE) Demonstration

In early 2009, the Centers for Medicare & Medicaid Services implemented the Acute Care Episode (ACE) Demonstration, which will test the use of a global payment for an episode of care as an alternative approach to payment for service delivery under traditional Medicare fee-for-service. The ACE demonstration seeks to align financial incentives within health care groups to provide quality care according to best practices at a savings to Medicare. Selected applicants, specifically physician-hospital organizations (PHOs), routinely provide at least one of the two main procedures included in the demonstration: Hip/knee replacement surgery; and/or Coronary artery bypass graft (CABG) surgery. The ACE demonstration will run for 3 years involving up to 5 sites representing 11 hospitals in 4 states: Colorado, New Mexico, Texas, and Oklahoma. 

IMPAQ International, LLC will assist CMS in the design and implementation of an independent evaluation of the feasibility and cost effectiveness of the bundled payment methodology and the improvement in quality of care and other benefits to Medicare beneficiaries. Key analytical tasks include defining appropriate comparison populations, collecting primary and secondary data, defining and implementing methods for (quality) outcomes evaluation and budget neutrality, and assessing shared savings as addressed in the legislation.

Designing an Evaluation of Three Administration on Aging (AoA) Programs

The Administration on Aging (AoA) currently funds three programs aimed at helping elders remain in the community, and out of nursing home facilities, as long as possible:  Aging and Disability Resource Centers (ADRCs), Community Living Programs (CLPs), and Evidence-Based Disease and Disability Prevention Programs (EBDDPs).  Each of these programs targets a slightly different sector of the aging population and utilizes unique approaches to meet the overarching goal of keeping elders in the community.  AoA, in partnership with the Agency for Healthcare Research and Quality (AHRQ), has determined the need for a rigorous evaluation of these interrelated programs to determine their effectiveness in enhancing elders’ abilities to remain in the community, deter placement in nursing facilities, and self manage chronic conditions.

IMPAQ has been tasked with designing a comprehensive evaluation of these three AoA programs, focusing on process and impact.  The process evaluation design addresses both the research interests surrounding organizational/structural aspects and the service components of each program.  The impact evaluation addresses the effect of each program on beneficiary outcomes, health care utilization, and costs.  In addition, the evaluation identifies mechanisms to implement continuous quality improvement strategies within each program. IMPAQ is working closely with program stakeholders at the state and local levels and utilizing existing claims and administrative data sources in the development of this design, to ensure feasibility and validity of the overall evaluation plan.  

Evaluating the Impact of OSHA’s Enforcement Interventions

The scope of this project is to design a rigorous evaluation of the impact of OSHA enforcement procedures on the future safety and health compliance of businesses.  The evaluation will provide answers to questions in four general research areas:

• What is the overall impact of OSHA enforcement procedures on future employer compliance?
• Which employer characteristics are strong indicators of future compliance?
• What are the best practices and measures to reduce future occupational injuries and illnesses among employers?
• How can OSHA best routinely measure and report the effectiveness of its compliance programs?

To design the evaluation, IMPAQ will review critical documentation and information on OSHA’s enforcement procedures.  IMPAQ will then receive relevant OSHA inspection and enforcement data files, develop descriptive statistics, and conduct a review and analysis of the data.  IMPAQ will examine rates and patterns of compliance and recidivism as related to variables such as employer characteristics, OSHA penalties, and time between inspections.  IMPAQ will use these analyses to provide initial information to OSHA, and inform the evaluation design.  Simultaneously, IMPAQ will interview key OSHA staff about enforcement procedures and data files.  The final deliverable will be a rigorous evaluation design that OSHA can implement to determine the impact of enforcement activities on subsequent employer safety and health compliance.

Evaluating the Impact of the Wage & Hour Division’s Enforcement Procedures

Within OASAM, the Center for Program Planning and Results (CPPR) funds and manages evaluations of select DOL agencies and programs in support of the Government Performance and Results Act of 1993 (GPRA).  The goal of this act is to improve the effectiveness of DOL’s agencies and programs.  One such agency, the Wage and Hour Division (WHD), is responsible for enforcing a variety of federal labor laws regarding the minimum wage, family and medical leave, migrant workers, lie detector tests, temporary worker programs, and prevailing wages for government service and construction contracts.  WHD selects employers to be investigated for possible violations of these federal labor laws, and conducts investigations to determine actual violations.  A primary goal of WHD is to reduce the rate of recidivism among employers who have been investigated.

IMPAQ International is conducting a one-year study to design a rigorous random assignment evaluation which would allow estimation of the impact of a variety of factors on compliance and the recidivism rate of employers.  These factors include WHD practices and procedures regarding selection and investigation of employers; employer characteristics; and other factors.  At the conclusion of this project, IMPAQ will deliver an evaluation design report and recommendations for improvements in procedures within WHD.

AMANECE Evaluation (Mexico)

Evaluation of the Housing Counseling Program

As a subcontractor to Abt Associates, IMPAQ developed a web-based participant tracking system to collect data on a sample of individuals receiving pre-purchase and foreclosure mitigation counseling services through the U.S. Department of Housing and Urban Development’s (HUD) housing counseling program.  The data collected through the tracking system is being used to analyze the impact and outcomes of the receipt of services by program participants.