In May 2017, the U.S. SBA published a guide for practitioners on using data to effectively evaluate the impact of business assistance programs titled Building Smarter Data for Evaluating Business Assistance Programs – A Guide for Practitioners.
Jonathan Simonetta, IMPAQ’s Chief Monitoring and Evaluation Officer, collaborated with program practitioners, economists, statisticians, and performance and policy staff from several federal agencies in contributing to the guide.
As well-designed impact evaluations are essential to program managers and policy makers in assessing business assistance programs, the guide was developed to promote understanding of the role of administrative and other key data and best data practices in designing rigorous impact evaluations.
Key Insights from the Guide
Carefully crafted impact evaluations provide evidence of program effectiveness. In a highly competitive environment for scarce resources, these evaluations can demonstrate program value through improved return on investment compared to other programmatic approaches.
When combined with other data, governmental and/or private, data from federal business assistance programs can be used to conduct rigorous impact analyses that compare the performance of businesses receiving assistance from similar non-assisted businesses.
The scope of an analysis can be expanded by the addition of historical data as well as the inclusion of additional research questions when more data are available. The results of this approach may yield lower evaluation costs and ease the survey burden on businesses and taxpayers, while promoting greater quality.
Advanced planning is an essential, yet often overlooked, element of successful impact evaluation. There are several initial steps needed to accomplish a successful design:
- Engage experts in program evaluation designs early. They will help to identify and develop the best methodologies and data to support cost-effective, high-quality evaluations.
- Before beginning a pilot or new program, identify, in advance, administrative data that will be needed to evaluate both the services delivered by the program and their impacts. For existing programs, early assessment of the quality and availability of data will expose data insufficiencies and allow for corrective actions.
- Analyze other data sources to determine if there are sufficient participant identifiers to effectively link data sets. Such links provide more robust data for evaluation. It is essential that necessary data security, confidentiality, and privacy assurances are not jeopardized.
- By engaging appropriate departmental personnel, particularly attorneys and policy officers, during the preparation of data use plans, integrated data for evaluation can be addressed early in the evaluation and design process.
The guide was published as part of the SBA’s Program Evaluation & Evidence Registry (PEER), which features research conducted by SBA programs and others in order to help answer the question, “What works in small business assistance programs?” and to make those answers broadly available so that program managers, policymakers, researchers, and the public can make evidence-based decisions.