Home / Project Showcase / Impact Evaluation Studies ProjectsView Printable Version
Project Showcase

Impact Evaluation Studies Projects

Impact Evaluation Studies Projects

Featured Project: 

Growing America Through Entrepreneurship (Project GATE)

 

Growing America Through Entrepreneurship (Project GATE)

Client:  Employment and Training Administration, U.S. Department of Labor
 
Project GATE, an experimental design demonstration and evaluation project initiated by the U.S. Department of Labor, was designed to investigate the impacts of providing microenterprise start-up services to interested individuals.  Applicants to Project GATE were randomly assigned to either a program group or a control group, with the participants offered services, such as: assessment of business needs, classroom training, one-on-one technical assistance, and assistance in applying for business financing. 
 
The evaluation of Project GATE was based on:

1) data collected during site-visits,
2) interviews conducted with treatment and control group members conducted at 6 months, 18 months, and 60 months following random assignment, and
3) administrative data obtained from participating states.

IMPAQ implemented both an impact and a process evaluation of the effectiveness of the GATE service model.  In its final impact evaluation report, IMPAQ staff developed policy recommendations regarding the national implementation of the GATE model in the U.S.

Read the Project GATE Final Report: 

 

Evaluation of the Project GATE II Grants

Client: Employment and Training Administration, U.S. Department of Labor

Following implementation of Project GATE (Growing America Through Entrepreneurship), awarded a series of grants known as GATE II.  Four states were awarded grants to implement the GATE model of entrepreneurship training:  Virginia, North Carolina, Alabama, and Minnesota.   IMPAQ International is conducting a three-year evaluation of the GATE II project.  This evaluation will consist of an implementation analysis, investigating how states implemented GATE II and identifying best practices, as well as an impact analysis, identifying the impact of GATE II on the self-employment and labor market outcomes of participants. 

The impact evaluation of GATE II is particularly interesting because the states have selected different target groups, different random assignment approaches to assign treatment and control group members, and different service models.  One of the states, Minnesota, is not implementing a random assignment design.  To evaluate program impacts in Minnesota, we plan to use a non-experimental impact evaluation method such as propensity score matching or regression discontinuity.  

 

Evaluation of the Community-Based Alternatives to Psychiatric Residential Treatment Facilities Demonstration Program

Client:  Centers for Medicare & Medicaid Services, U.S. Department of Health & Human Services
 
IMPAQ is currently conducting a longitudinal evaluation of the Community-Based Alternatives to Psychiatric Residential Treatment Facilities (PRTF) demonstration program.  This demonstration provides grants to ten states to provide home and community-based services (HCBS) to youth as alternatives to PRTFs under a 1915(c) waiver.  The overall goal of the evaluation project is to determine if the provision of HCBS to youth under this demonstration results in the maintenance or improvement in a child’s functional status and costs no more than anticipated aggregate PRTF expenditures in the absence of the demonstration.  For this project, IMPAQ has designed a Minimum Data Set (MDS), provided technical assistance to states in establishing data collection mechanisms, and processed and conducted statistical analyses on the data collected.  Once all of the data have been collected and processed, IMPAQ will analyze the results and prepare a report summarizing the impact of the demonstration program on meeting its objectives.
 
Read the CBA Implementation Status Report:
 
 

Workforce Investment Act Net Impact Evaluation

Client: Employment and Training Administration, U.S. Department of Labor
 
IMPAQ conducted a nonexperimental net impact evaluation of the Workforce Investment Act (WIA) program to estimate the average impact of the program across individuals, that is, the difference in average earnings or employment following program participation, as compared to the average if these individuals had not participated in the program.  The study focused on the Adult and Dislocated Worker programs which offer three levels of service:
  • Core services, including outreach, job search and placement assistance, and labor market information
  • Intensive services, involving more comprehensive assessments, development of individual employment plans, and counseling and career planning.
  • Training services, occupational and basic skills training that link customers to job opportunities in their communities.
To support this study, IMPAQ collected and processed administrative data from 12 states, covering approximately 160,000 WIA participants and nearly 3 million comparison group members.  IMPAQ employed state-of-the-art nonexperimental methods to compare WIA program participants with matched comparison groups of individuals who either did not receive WIA services or did not receive WIA training services.
 
Read the WIA Net Impact Evaluation Final Report:
 
 

Evaluation of the Senior Risk Reduction Demonstration Under Medicare

Client:  Centers for Medicare & Medicaid Services, U.S. Department of Health and Human Services
 
The purpose of this project is to evaluate the effectiveness and impact of the Senior Risk Reduction Demonstration (SRRD), which is being sponsored by CMS.  Under the SRRD, demonstration vendors are providing risk reduction services to a random sample of non-institutionalized Medicare fee-for-service beneficiaries, aged 67 – 74, and enrolled in Medicare Parts A and B – both nationwide and from communities which have exemplary information and referral and assistance service programs for seniors. IMPAQ’s role on the project is to design and conduct a longitudinal impact evaluation of the SRRD, including the collection and analysis of Health Risk Appraisal (HRA) survey data, analysis of CMS Medicare claims data, and the conduct of site visits to assess program impacts and budget implications of the program.
 

Evaluation of the Medicare Acute Care Episode (ACE) Demonstration

Client: Centers for Medicare & Medicaid Services, U.S. Department of Health and Human Services 

In early 2009, the Centers for Medicare & Medicaid Services implemented the Acute Care Episode (ACE) Demonstration, which will test the use of a global payment for an episode of care as an alternative approach to payment for service delivery under traditional Medicare fee-for-service. The ACE demonstration seeks to align financial incentives within health care groups to provide quality care according to best practices at a savings to Medicare. Selected applicants, specifically physician-hospital organizations (PHOs), routinely provide at least one of the two main procedures included in the demonstration: Hip/knee replacement surgery; and/or Coronary artery bypass graft (CABG) surgery. The ACE demonstration will run for 3 years involving up to 5 sites representing 11 hospitals in 4 states: Colorado, New Mexico, Texas, and Oklahoma. 

IMPAQ International, LLC will assist CMS in the design and implementation of an independent evaluation of the feasibility and cost effectiveness of the bundled payment methodology and the improvement in quality of care and other benefits to Medicare beneficiaries. Key analytical tasks include defining appropriate comparison populations, collecting primary and secondary data, defining and implementing methods for (quality) outcomes evaluation and budget neutrality, and assessing shared savings as addressed in the legislation.

Designing an Evaluation of Three Administration on Aging (AoA) Programs

Client:  Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services

The Administration on Aging (AoA) currently funds three programs aimed at helping elders remain in the community, and out of nursing home facilities, as long as possible:  Aging and Disability Resource Centers (ADRCs), Community Living Programs (CLPs), and Evidence-Based Disease and Disability Prevention Programs (EBDDPs).  Each of these programs targets a slightly different sector of the aging population and utilizes unique approaches to meet the overarching goal of keeping elders in the community.  AoA, in partnership with the Agency for Healthcare Research and Quality (AHRQ), has determined the need for a rigorous evaluation of these interrelated programs to determine their effectiveness in enhancing elders’ abilities to remain in the community, deter placement in nursing facilities, and self manage chronic conditions.

IMPAQ has been tasked with designing a comprehensive evaluation of these three AoA programs, focusing on process and impact.  The process evaluation design addresses both the research interests surrounding organizational/structural aspects and the service components of each program.  The impact evaluation addresses the effect of each program on beneficiary outcomes, health care utilization, and costs.  In addition, the evaluation identifies mechanisms to implement continuous quality improvement strategies within each program. IMPAQ is working closely with program stakeholders at the state and local levels and utilizing existing claims and administrative data sources in the development of this design, to ensure feasibility and validity of the overall evaluation plan.  

Evaluating the Impact of OSHA’s Enforcement Interventions

Client:  Center for Program Planning and Results of the Office of the Assistant Secretary for Administration and Management (OASAM), U.S. Department of Labor

The scope of this project is to design a rigorous evaluation of the impact of OSHA enforcement procedures on the future safety and health compliance of businesses.  The evaluation will provide answers to questions in four general research areas:

  • What is the overall impact of OSHA enforcement procedures on future employer compliance?
  • Which employer characteristics are strong indicators of future compliance?
  • What are the best practices and measures to reduce future occupational injuries and illnesses among employers?
  • How can OSHA best routinely measure and report the effectiveness of its compliance programs?

To design the evaluation, IMPAQ will review critical documentation and information on OSHA’s enforcement procedures.  IMPAQ will then receive relevant OSHA inspection and enforcement data files, develop descriptive statistics, and conduct a review and analysis of the data.  IMPAQ will examine rates and patterns of compliance and recidivism as related to variables such as employer characteristics, OSHA penalties, and time between inspections.  IMPAQ will use these analyses to provide initial information to OSHA, and inform the evaluation design.  Simultaneously, IMPAQ will interview key OSHA staff about enforcement procedures and data files.  The final deliverable will be a rigorous evaluation design that OSHA can implement to determine the impact of enforcement activities on subsequent employer safety and health compliance.

 

Evaluating the Impact of the Wage & Hour Division’s Enforcement Procedures

Client: Office of the Assistant Secretary for Administration and Management (OASAM), U.S. Department of Labor

Within OASAM, the Center for Program Planning and Results (CPPR) funds and manages evaluations of select DOL agencies and programs in support of the Government Performance and Results Act of 1993 (GPRA).  The goal of this act is to improve the effectiveness of DOL’s agencies and programs.  One such agency, the Wage and Hour Division (WHD), is responsible for enforcing a variety of federal labor laws regarding the minimum wage, family and medical leave, migrant workers, lie detector tests, temporary worker programs, and prevailing wages for government service and construction contracts.  WHD selects employers to be investigated for possible violations of these federal labor laws, and conducts investigations to determine actual violations.  A primary goal of WHD is to reduce the rate of recidivism among employers who have been investigated.

IMPAQ International is conducting a one-year study to design a rigorous random assignment evaluation which would allow estimation of the impact of a variety of factors on compliance and the recidivism rate of employers.  These factors include WHD practices and procedures regarding selection and investigation of employers; employer characteristics; and other factors.  At the conclusion of this project, IMPAQ will deliver an evaluation design report and recommendations for improvements in procedures within WHD.

AMANECE Evaluation (Mexico)

Client:  Carso Health Institute, Mexico City, Mexico
 
IMPAQ recently conducted an evaluation of the implementation of the AMANECE project in Mexico, which applies an innovative prevention, promotion, and care model based on network connectivity and technology innovation aimed at a significant improvement of health care and a reduction of maternal and child mortality.  The AMANECE approach includes the use of technology (e.g., cell phones, computers) to connect pregnant women and health services providers and incorporates the perception of women not as an object of care, but rather as the subject and main player in each of their process stages.  For this project, IMPAQ prepared a quasi-experimental evaluation design, designed and administered a health survey to a group of women of child-bearing age and health services providers (health promoters and clinical staff), collected demographic and health outcome data on program participants, and analyzed the results to assess program impacts.
 

Evaluation of the Housing Counseling Program

Client:  U.S. Department of Housing and Urban Development 

As a subcontractor to Abt Associates, IMPAQ developed a web-based participant tracking system to collect data on a sample of individuals receiving pre-purchase and foreclosure mitigation counseling services through the U.S. Department of Housing and Urban Development’s (HUD) housing counseling program.  The data collected through the tracking system is being used to analyze the impact and outcomes of the receipt of services by program participants.

 

Impact of Active Labor Market Programs (Romania)

Client:  The World Bank
 
This quasi-experimental impact evaluation examined the effectiveness of active labor programs in Romania implemented under the World Bank Employment and Social Assistance Project.  The primary goal of this project was to provide information on the net impact of selected Active Labor Market Programs (ALMPs) with different types of unemployed clients, and under varying labor market conditions, and to improve their targeting and cost-effectiveness.
 
To evaluate the net impact of ALMP services in Romania, IMPAQ employed a quasi-experimental design.  A survey was administered to over 4,000 respondents, including a sample of over 2,000 ALMP participants and a similar comparison sample of individuals who did not participate in ALMP.  Based on the impact evaluation comparing economic outcomes for a sample of program participants and non-participants, IMPAQ developed labor market policy recommendations, including recommendations for improved targeting.