HAL2

Allows decision-makers to explicitly understand and measure the uncertainty involved in key factors that might influence decision outcomes.

Process Overview

The fact that someone needs to make a decision is itself evidence that there is uncertainty associated with the future and that there is some value in the ability to evaluate and plan for that uncertainty. The only certainty in forecasting is that every important assumption about the future will be wrong to some extent. Knowing this, how should decision-makers (and their technical advisors) deal with the risk of being wrong.

One common approach is to estimate 'high' and 'low' point estimates. This approach is usually of little practical value, however, since it offers no guidance as to the relative likelihood of one scenario over another. Moreover, using 'high' and 'low' point estimates typically assumes that all estimates differ from their expected values in the same direction, an outcome that is just as remote as everything turning out exactly as projected.

Probability provides a way around the limitations of discrete point estimates outlined above. Probability measures the confidence that an expected outcome will actually materialize. To understand how probability aids decision making, consider a simple example. Before the advent of powerful computers, weather forecasters would simply assert their mean expectations - "we do not expect rain today". The decision on today's picnic would be easy - full steam ahead. Now, the same forecast incorporates the probability for each causal factor in the determination of rain, and the forecaster announces that, "there is a 25 percent chance of rain by mid-afternoon". A more reasoned decision regarding the picnic is now possible. If the event involves costly logistics for hundreds of people, a rain date might well be announced. In the past, provision for risk was not possible and a good few dollars - not to mention tempers - were lost.

Risk assessment, while not in common use, is by no means new, and has been used to assess long-range decisions by public agencies and private firms alike. By attempting to measure the risks of each of the key factors that might influence a major decision and incorporating those risks into the analysis, the resultant forecast will not offer a single "take it or leave it" answer. Rather, information can be presented that actually reflects the uncertainties involved and how they might influence the future.

What is the Process?

HAL has developed an integrated and fully automated Uncertainty Analysis Process for evaluating the uncertainty inherent in decision making. We believe the Process to be the most advanced management-oriented system of decision evaluation and risk management available today.

The HAL Uncertainty Analysis Process is characterized by five key attributes:

• It deals with uncertainty and risk using advanced statistical techniques;
• It allows sound management intuition to be applied quantitatively in the evaluation process;
• It depicts the business structure of the real world and can be used to model many situations;
• It helps clarify and sharpen understanding of the critical factors affecting the situation; and
• It facilitates systematic evaluation and consensus building on controversial matters.

The process has been used previously by HAL in a wide variety of applications, including assessing the potential for ecomonic benefits resulting from Canadian space mission and other technology development programs.

How the Process Works

Each factor and assumption of importance to the decision process is reviewed and assessed. Those factors and assumptions that either affect the decision criteria strongly, or which affect the decision less strongly but are unknown or very uncertain, are included in the analysis in probabilistic terms. That is, each variable that depends on these factors or assumptions is assigned a range, or probability distribution, reflecting its underlying uncertainty. These estimates are then combined to provide estimates of the probability and uncertainty inherent in the criteria upon which the decision has to be made.

Specifically, the HAL Uncertainty Analysis Process involves four steps:

1. Develop structure and logic models. This step establishes the logic and methodologies involved in the process, and ascertains which variables and assumptions must be considered in the analytic model;
2. Develop initial forecasts and ranges. In this step, estimates and ranges are developed for each assumption identified in Step 1 and recorded. These estimates are based on the consulting team's statistical analysis of historical or actual data and subjective judgement drawn from experience in the field.
3. Expert Review. The purpose of expert review is to identify factors and issues that may have been overlooked in Step 2, and to assist in the assignment of probabilities to certain variables. The experts are best qualified to provide qualitative information necessary to complete the analysis using their experience, training, 'street-wise' judgement and knowledge of relevant facts and issues. Their opinions of the initial estimates and ranges enable the probabilities associated with expected outcomes to be gauged properly.
4. Forecasting. Once the experts have completed their work, the ranges for all variables are transformed by the computer into input probability distributions. And once final distributions are generated for all assumptions and variables, they are combined to yield a probability distribution for each forecast (the figure illustrates this process, for which we use a statistical technique called Monte Carlo Simulation). The figure illustrates the fiinal distribution.

Monte Carlo Simulations: Combining Probability Distributions

Example of Output Results

The Need for the Process

Probability Distributions

Virtually always, subjectivity and expert experience play an important part in the assessment of uncertainty. Embodying advanced mathematical and statistical techniques of applied probability, the HAL Uncertainty Analysis Process permits experts to reflect, quantitatively, both rigour and subjectivity in the evaluation of uncertainty.

Having probability distributions for criteria upon which decisions are made has three main advantages over traditional measures:

1. Probability distributions are graphic. By overlaying the various scenarios it is possible to tell at a glance which factors dominate as well as the risk and return tradeoffs associated with each scenario.

2. A normal curve (or "bell curve") is the traditional method of measuring uncertainty. This method assumes there is an equal chance that the actual result will end up too high as too low. In reality the uncertainty may not be symmetric, and any attempt to fit a normal curve will result in lost information. The Process identifies whether the uncertainty is best depicted by an asymmetrical or ‘skewed’ distribution. Once the true nature of the risk is understood, a more reasoned approach to decision making is possible.

3. The probability distribution also reflects any changes made to the input variables. The impact each input variable has on the overall probability distribution can then be identified.

How does considering probability distributions improve evaluation? Consider the illustrations in the figure below of probability distributions for Programs A and B under two different scenarios. The figure below describes the confidence that a given rate of return for each program will be achieved. In Scenario 1, Program A and Program B have the same expected rate of return. However, because Program A is less risky (distribution is narrow), it is the preferred choice. Decisions based solely on expected value would not be able to distinguish between the two choices.

In Scenario 2 the decision is not quite as easy. In this example Program B has a higher expected return than Program A, yet the risk associated with Program B is also higher. The risk for Program B is higher because there is a higher likelihood that the actual return will be less than the prescribed minimal accepted return. To compare these alternatives, the fact that one distributions is skewed must be taken into account. The eventual choice between these two programs depends on the decision maker’s assessment of the tradeoff between risk and return.

Analysis of Uncertainty for a Program

Other Advantages

Depicting the Real World: The Process can be used to depict the full and subtle complexity of real world environments. For example, the Process can reflect items such as price determination, spreads, profit and risk in business. Used in this way, the Process is not a text-book business model. To be sure, the Process reflects the established mathematics of the business world. But the Process also invites the incorporation of managerial insight into the subtleties of particular decisions. In this way, the Process embodies the accumulated experience of experts in a systematic program of forecasting and the assessment of uncertainty.

Clarifying the Environment: The Process is neither a black box nor an off-the-shelf system. The Process models are built in consultation with experts, a process that provides a much sharper sense of the environment under consideration and a systematic record of accumulated insights.

Consensus Building in Strategic Planning: To purge the planning process of confusion and misunderstanding, the Process deliberately separates the assessment of individual factors that determine forecasts from the complex inter-relationships between them. This permits the systematic appraisal of each factor one at a time, a feature that facilitates reasoned, uncluttered discussion and consensus.

Optimum Use of Expert Resources: As well, by disentangling the discussion of factors from relationships, the Process invites the participation of individuals with valuable expertise in a particular area, but whose knowledge may not extend to the relationships among factors.