• Increase font size
  • Default font size
  • Decrease font size

HAL2

Member Area
Performance Measurement

For years organizations have sought ways to effectively measure their performance against plan and to then clearly understand the meaning and cause for any resulting variances. Over time this requirement has been transformed from ad hoc approaches to the development and adoption of formal methodologies for performance measurement.

The Need for Performance Measurement

With the wave of organizational changes launched in the 1980's and characterized by the likes of delayering, right sizing, re-engineering, and focus on core competencies, many organizations adopted performance measurement as a means to assess the consequences of their restructuring.

Organizations of all types are increasingly being asked to not only do more with less, but also to prove that they are doing it. This means that they need to measure their performance to both manage their operations and to demonstrate to others how well they are doing. This is especially true in the public sector. Increased scrutiny of government spending, both from inside and outside government, and a need to ensure public trust has meant that organizations receiving public funds must clearly achieve and demonstrate continuing improvements in efficiency, effectiveness, and impact.

In recent years, many governments in Canada have undertaken reforms to enhance their accountability systems. At the federal level, all departments have been required to submit annual performance reports to Parliament since 1997. At the provincial level, the governments of Saskatchewan, Alberta, British Columbia, and Ontario, among others, have also launched performance measurement initiatives to focus more directly on linking results with programs and plans.

Performance measurement in the public sector is not easy. Mintzberg has argued that "many activities are in the public sector precisely because of measurement problems; if everything was so crystal clear and every benefit so easily attributable, those activities would have been in the private sector long ago." The purpose, objectives and outputs of public sector institutions are fundamentally different than those of the private sector. However, significant progress has been made in developing approaches to public sector performance measurement and most public organizations now have performance measurement systems.

The Purpose of Performance Measurement

Performance measurement is about seeking real-time answers to the question "Are we getting what we expect at an acceptable cost?" Organizations measure to learn what is happening in order to identify problems, make corrections, and demonstrate success. Performance measurement serves to address such matters as the extent to which an organization's initiatives have been successful, whether they can be improved, whether they should continue, and whether there are more cost-effective alternatives. Ultimately, performance measurement is a tool that improves analysis, communication, learning, and understanding.

Different measures are of interest to different stakeholders for different purposes. For example, the public seeks assurances that their money is being well spent. The Government needs to understand how organizations contribute to the goals of the country. Senior management is interested in the strategic direction of the organization and its success in fulfilling its mandate. Operational management will use the measures to monitor and control the operations of the organization on a routine basis. All of these uses are inter-related, but each has its own special requirements.

The importance of performance measurement lies as much in the discipline involved in analyzing the relationship among outcomes, activities and clients, as in the values of the measures themselves. An understanding of the relationships between and among the measures allows a better focus on achieving the mission and objectives of the organization.

Performance Measurement Frameworks

Evaluation frameworks in the public sector reflect the concerns that a public sector organization tends to have. These include dimensions such as social relevance, cost-effectiveness, and public good. The framework sets the stage for both ongoing monitoring and periodic reviews.

The following table outlines the themes typically covered in a public sector evaluation:

ThemesExamples
Rationale and RelevanceLegitimacy of the role for government
Consistency with public policies and priorities
Fit in the innovation system
Fit with other government programs
Management and Efficiency Management approach and style
Resource usage - amount, type, value
Employee satisfaction
CostsRisks
Objectives Achievement and Impact

Results - timeliness, quality, output, client satisfaction
Long-term impacts and effects - economic, social, environmental
Reach - breadth and depth of influence, number and type of clients

FutureService provision alternatives
Strategic choices
Operational choices
Resource requirements

All performance measurement frameworks are similar in that they link strategic plans with work plans; the long-term vision with short-term operations. The evaluation framework also allows current issues to emerge that help to tell the 'performance story' beneath the measures.

An example of these broader concerns is provided by the four evaluation categories used within the federal industry portfolio:

CategoriesObjectivesDirect OutcomesUltimate Impacts
Advancement of KnowledgeTo build and maintain Canadian capability for excellence in scientific discovery. Development of new knowledge in strategic areas. Strong research base in universities and government laboratories.
Innovation with FirmsTo ensure the application and commercialization of S&T to create jobs and economic growth through collaboration and support for firms.Enhanced innovative capability.
New, improved products, processes, and services. 		Increased competitiveness.
Effective positioning of firms and research in the world.
Innovation System SupportTo provide the infrastructure required by the local system of innovation.Supportive legal, regulatory, physical, human resource, and social systems.Effective regional system of innovation.
ManagementTo develop and apply new mechanisms for the governance and management of S&T. Effective leadership and performance management.
Improved service coordination and delivery. 		Maximize efficiency and effectiveness.
More entrepreneurial government organizations.

The Performance Measurement Process

The process of performance measurement can be thought of in terms of the following cycle:

  • Planning: Involves defining the goals and priorities of the organization, and articulating the strategy to achieve those goals. This occurs at government, organizational, and departmental levels. The planning at each level must be aligned with the other levels.
  • Performance Management: Involves the evaluation of progress towards the goals established in the planning stage. Data is collected through a number of lines of enquiry, possibly including information systems, document review, interviews, workshops, and surveys. The data is analyzed to uncover issues and evaluate performance.
  • Reporting: Involves sharing progress and achievements with stakeholders, possibly including the general public, government, strategic management, and operational management.
  • Feedback: Involves using the conclusions drawn from the information to plan strategic and operational strategies to ensure that goals and priorities are achieved during the next cycle.

The experiences of HAL and others in performance measurement for public sector science and technology organizations have resulted in a number of 'lessons learned' that can help guide future performance measurement development. The following serves as an example of these lessons:

Planning

  • Participation - A sense of ownership of the performance framework is needed to encourage utilization. This means that all levels should participate in their creation. In particular, staff need to be involved. They must become the organization's measurers, not those of an outside expert.
  • Direction of influence - The performance framework needs contributions from both the top-down and the bottom-up. Top-down direction by itself may alienate staff. On the other hand, the vision, mission, and strategic directions of the organization must be clearly evident in the measures.
  • Choice of measures - Both quantitative and qualitative measures have a place. Some measures should be standardized at the top level of the organization, but many will need to be specific to the needs of individual units. Measures should be chosen for their relevancy, not their ease of collection. Measures should be results, rather than activity driven, and long-term, rather than short-term.
  • Number of measures - There is no need to measure everything. However, overly simple measures may not be meaningful. Consequently, careful planning is needed in deciding what to measure.

Performance Management

  • Use of measures - Data needs to be organized, analyzed and interpreted. Ensure that measures do not hide the true story. Ensure that the cause and effect behind the measures is understood.
  • Data collection - Decisions need to be made about the sources, frequency, and methods for data collection. There will be trade-offs to be made among cost, timeliness, and performance. Many data collection methods can benefit from the use of electronic information systems.
  • Lines of enquiry - Use multiple lines of enquiry that support and compliment each other, such as information systems, document reviews, interviews, surveys, and workshops.
  • Issues - Do not forget to get at the issues underneath the quantitative performance measures. For example, what have been the unintended impacts of organizational activities?

Reporting

  • Electronic Reporting System - An information infrastructure can help in managing data flows. In particular, Web technology can support data collection and data reporting.
  • Tailored Reports - Reports should be tailored to the needs of the stakeholder in terms of the information content, delivery timing, and delivery format. Stakeholders include the public, government, strategic management, and operational management.
  • Learning - Performance is measured so that the organization can learn and improve. Without learning, there is little purpose in going to the trouble of measuring performance.
  • Advocacy - A performance report can also provide the substance for success stories that may positively influence resource allocation decisions and publicly demonstrate the achievement of good results.

Experience shows that performance measurement perfection is not achieved from the start. Rather, it is a continuous process that requires time and a certain amount of trial and error. Start small, possibly with a pilot project, and adjust over time. Furthermore, ensure that performance measurement is linked to a performance measurement plan and management procedure.