HAL2

We have been responsible for many of the evaluations of federal government science and technology programs, and for many of the evaluations and evaluation frameworks of a broad range of publicly supported, collaborative R&D investment programs, run at arm's length from government. We are very familiar with the Treasury Board Guidelines for program evaluation and accountability and have contributed significantly to federal methodologies for evaluating the economic and social impacts of science and technology programs. Our evaluation studies have also covered a wide variety of industry sectors including telecommunications, information technology, energy, environment, transportation, space, geomatics, and science & technology.

The definition of program evaluation, as practised today under the guidelines set out by the Treasury Board of Canada, includes:

• The function, which is the formal assessment of programs in terms of their rationale, objectives achievement, impacts and effects, alternatives, and, frequently, future directions;
• The procedure, which, according to generally accepted approaches and methodologies, is the performance of an evaluation assessment or study for a particular program, which defines the program to be evaluated (logic model), identifies the issues to be evaluated, develops and assesses the methodologies and options for addressing these issues, analyzes the impacts and effects, identifies and assesses various alternatives to the program; and sets out future directions based on the evaluation results, and
• The process, which includes developing evaluation frameworks for new or modified programs, undertaking evaluation assessments and evaluation studies, and making decisions and taking actions resulting from the findings and recommendations of the evaluation studies.

Taking these three aspects together, program evaluation can be defined as a periodic, independent and objective review and assessment of a program, set of related programs, or a logical program component. The purpose is to determine the program's adequacy or relevance under the current and expected situation, and all its impacts and effects. Cost effectiveness of the existing program and alternative means to achieve the same objectives are major concerns of program evaluation.

The objectives of the program evaluation function in departments or agencies are to provide the deputy head with the information necessary for program improvement, resource allocation and accountability. Such information must be well-founded, timely and appropriate to the needs of the deputy head.

The role of program evaluation within the public sector has clarified and strengthened over the past couple of decades. It is now realized that the impacts of government involvement in the social and economic structures of the country should be the focus for assessment rather than just economy and efficiency of operation. Coupled with this realization has been the development of an increasingly complex and interdependent socio-economic system which makes the identification of the true and incremental impacts and effects of government actions that much more difficult.

The program evaluation procedure borrows the methodological and investigative approaches of interdisciplinary systems analysis appropriate for the particular social and economic systems involved. The program evaluation function and process institutionalize the requirement to undertake evaluation and to put into place the changes that are indicated from its findings.

In recent years, our firm has undertaken numerous evaluation studies, including frameworks, reviews and evaluations, all according to the Treasury Board of Canada (TBC) Guidelines on Program Evaluation.

The Treasury Board Evaluation Policy is based on three fundamental principles:

• That achieving and accurately reporting on results is a primary responsibility of public service managers.
• That rigorous and objective evaluation is an important tool in helping managers to manage for results.
• That departments, with the support of the Treasury Board Secretariat, are responsible to ensure that the rigour and discipline of evaluation are sufficiently deployed within their jurisdictions.

Evaluation has two main purposes:

• To help managers design, or improve the design of, policies, programs and initiatives.
• To provide, where appropriate, periodic assessments of policy or program effectiveness, of impacts both intended and unintended, and of alternative ways of achieving expected results.

Departments should embed the discipline of evaluation into the lifecycle management of policies, programs and initiatives to:

• Develop results-based management and accountability frameworks for new or renewed policies, programs and initiatives.
• Establish ongoing performance monitoring and performance measurement practices.
• Evaluate issues related to the early implementation and administration of the policy, program or initiative, including those that are delivered through partnership arrangements (formative or mid-term evaluation).
• Evaluate issues related to relevance, success, and cost-effectiveness.

The standards cover the following:

Planning the Evaluation: Based on a departmental strategic plan, the evaluation takes account of the objectives and priorities of both the department and the government. The full range of evaluation issues is considered at the planning stage, as represented in a program framework for the evaluation.

Accountability: The evaluation addresses issues that are needed for accountability reporting including those involving key performance expectations.

Measurement and Analysis: The evaluation will produce timely, pertinent and credible findings and conclusions that management and other stakeholders can use with confidence. The evaluation findings are relevant to the evaluation issues addressed and follow from the evidence. Conclusions are consistent with and follow from the findings.

Evaluation Reports: Evaluation reports present the findings and conclusions in a clear and balanced manner and indicate their degree of reliability. Reports are consistent with Cabinet and Treasury Board submission procedures and external reporting requirements.

Results Based Management and Accountability Frameworks

Public sector managers are expected to define anticipated results, continually focus attention on results achievement, measure performance regularly and objectively, learn from this information, and adjust to improve efficiency and effectiveness.

The RMAF is intended to help managers achieve the following: a sound governance structure, a results-based logic model, a sound performance measurement strategy, and adequate reporting.

Different results-measurement activities occur at different points in time as part of the ongoing management of a policy, program or initiative. This continuum, from the initial consideration of performance measurement, through performance monitoring to formative and summative evaluation, is presented in the following table. While shown as a linear process, it must be remembered that performance measurement is iterative, and review and feedback are important parts of the process. Evaluators are key to stages 7 and 8, and often involved in stages 0, 1, and 2.

RMAF Activities

There are three key parties involved in the development and implementation of a Results-base Management and Accountability Framework: managers, evaluation specialists, and in the case of a Treasury Board commitment, analysts of the Treasury Board Secretariat.

The RMAF contains several components:

1. Profile - a concise description of the policy, program or initiative, including a discussion of the background, need for the program, target population, delivery approach, resources, governance structure, and intended results.

2. Logic Model - an illustration of how activities are expected to lead to outputs, immediate outcomes, intermediate outcomes, and eventually, ultimate outcomes.

3. Ongoing Performance Measurement Strategy - a plan for the ongoing measurement of performance, including the identification of performance indicators and a measurement strategy describing how these indicators will be collected, how often, and at what cost.

4. Evaluation strategy - a plan for the evaluation of the policy, program, or initiative, including the identification of formative and summative evaluation issues and questions, the identification of associated data requirements, and a data collection strategy.

5. Reporting strategy - a plan to ensure the systematic reporting on the results of ongoing performance measurement as well as evaluation, to ensure that all reporting requirements are met.

Evaluations typically occur at two points in the lifecycle of a policy, program or initiative: Formative or midterm evaluations (normally within the first two years), and Summative evaluations (normally within five years of start-up).

There are three primary issue areas for evaluation that need to be considered:

• Relevance - does the policy, program or initiative continue to be consistent with departmental and government-side priorities, and does it realistically address an actual need?
• Success - is the policy, program, or initiative effective in meeting its objectives, within budget and without unwanted negative outcomes? Is it making progress toward the achievement of the intended outcomes?
• Cost-effectiveness - are the most appropriate and efficient means being used to achieve objectives, relative to alternative design and delivery approaches?

Potential users of performance information might include: program/policy/initiative management, central agencies, and stakeholders (internal and external). Uses of this information will depend on the type of user and could include management decision-making, accountability and communication/information sharing.

Evaluation Implementation Issues 

We highlight below what we have found to be the critical or most difficult components of evaluation studies that we have experienced in projects for Industry Canada and other departments.

The impacts and effects that we are concerned with in an evaluation are those which are directly due to the program under review. These impacts and effects are called incremental, a term which is usually defined as the difference between what did happen with the program in place, and what would have happened if the program had not been in place. As difficult as it might be to identify and measure the actual and relevant impacts and effects that did happen, it is usually much more difficult to estimate what impacts and effects would have happened without the program in place. This difficulty involves not only estimating how social and economic actors would have reacted in a world different from that which prevailed, but also attempting to specify what the different government programs or responses would have been to this different world.

A concept related to incrementality is that of attribution. It is often possible to determine that certain activities would not have taken place had it not been for the program under review (incremental activities), but that these same activities were the beneficiary of more than one government program (or influencing tax policies, or programs of a different level of government). Such incremental activities may give rise to impacts and effects that are not wholly attributable to the program under review. In these cases, if the cooperating programs are to be credited with some of the incremental impacts, these impacts must be attributed to the various contributing programs in some way.

Evaluation Issues and Scope of the Evaluation

Evaluation studies must be undertaken with limited resources, and perhaps as important in the current rapidly changing environment, within a given timeframe, in order to be relevant to decision makers. At the same time, however, there is almost always strong pressure for the evaluation to be comprehensive, to cover all the issues, and to report on all aspects of a program. Evaluation methodology calls for the issues to be identified and assessed for importance or priority by the deputy head or other client of the evaluation. Once issues are ranked, evaluation options can be prepared which address the important issues, as well as contribute partial information to the less important issues. Problems often arise at this point in the evaluation assessment if clear direction from the deputy head (or other senior managers or steering committee) is not forthcoming. In its place, too often, is a request to provide the broadest possible coverage but within a very tight budget and/or timeframe.

The Problems of Data Collection and Analysis

Most evaluations require that some additional data be collected and analyzed together with existing data from a variety of sources. The problems that are encountered are common to any primary data collection exercise, especially one that is severely time constrained. These problems involve approvals for collecting the data, preparation of satisfactory data collection instruments and procedures, identification of appropriate populations, samples, and contact specifics, securing adequate response rates within time and budget, and coding and analyzing the resultant database together with existing data. The challenge of data analysis is one of extracting unbiased, objective and supportable findings from the collection of databases, which, at best, provide quantitative information on a subset of relevant variables, but not complete information on the exact variables desired.