Guide to Rapid Impact Evaluation

Guide to Rapid Impact Evaluation

This guide provides an overview of the method for rapid impact evaluation (RIE) and how RIE can best be used in the Government of Canada. It focuses on the results of three pilot projects conducted in three federal departments. The pilot projects used a mixed-method approach, so they should not be taken as definitive models for the use of RIE in other contexts. Departments are welcome to adapt the RIE method to suit their particular needs and requirements.

Acknowledgements
This guide has been prepared by the Centre of Excellence for Evaluation of the Treasury Board of Canada Secretariat. It is based on pilot projects supported by Dr. Andy Rowe, who developed the RIE method in 2004 and 2005 to assess the environmental and economic effects of decisions about managing natural resources.

The Results Division thanks Dr. Rowe for his support and advice, Natural Resources Canada, the Public Health Agency of Canada and Public Safety Canada for their participation in the pilot projects. The Division also thanks the departments and agencies that reviewed and provided comments on draft versions of this document.

Summary: what is a rapid impact evaluation?
A rapid impact evaluation (RIE) provides a structured way to gather expert assessments of a program’s impact. An RIE engages a number of experts to provide a balanced perspective on the impacts of a program and ultimately increase acceptance and adoption of the RIE’s findings. Each expert assesses program outcomes relative to a counterfactual, which is an alternative program design or situation, in order to assess the program’s impact relative to alternatives. Three types of experts are consulted:

program stakeholders who affect the program or are affected by it
external subject matter experts
technical advisors
Time required for an RIE
Depending on the availability of experts, two to six months are needed for an RIE. Some time for learning is also required when an organization first conducts an RIE.