Measuring ROI-The Fifth Level of Evaluation / Jack Phillips

Makaleyi indir

Training departments are increasingly being asked to justify every dollar spent. Here’s an approach to calculating return on investment using a time-honored training formula

For too long, the training and development process has escaped the scrutiny of accountability. While expenditures have grown, many training departments have not taken the extra step to show the payoff of their efforts—particularly the more elaborate, comprehensive, and expensive programs. Tools and techniques to measure return on investment (ROI) are now available, and the process has become reliable and acceptable. Measuring the return on training investment should be a requirement in most organizations, at least for some programs.

Before dismissing the idea of measuring your company’s return on training investment, consider the rewards of such an evaluation. An example of how valuable training can be is evident in the literacy training program at Magnavox Electronics Systems Company, West Coast Division, in Torrance, California.

Many literacy programs are undertaken because they are needed, but little attention is paid to the program’s economic returns. Magnavox wanted to know if this type of training returned an economic dividend. After its 18week literacy program was covered verbal and math skills, it was shown to have a significant payoff for the company. The benefits of the program (reduced scrap, rework, and increased productivity) were converted to dollar values. The program yielded a cost-benefit ratio of 8.4:1 and a 741 percent return on training investment! And while these numbers are impressive, the program designers consider them conservative.

In every corner of the training and development field, the pressure to measure the return on investment is increasing. At some time or another, virtually every organization will face this important issue. Many progressive organizations are taking a logical and methodical approach to developing ROI for a limited number of programs, using a sampling basis. Six trends have increased the interest in, and use of, ROI measurement in training and development:

▼ Training and development budgets are continuing to grow, which creates more pressure for accountability.

▼ Training and development are linked to competitive strategies, which make them important areas for measuring program contribution. 

▼ Many programs have failed to deliver what was expected; consequently, program sponsors have requested ROI calculations/ justifications.

▼ The concern for accountability in all functions in an organization is increasing; thus, the training and development function becomes one of many support efforts under scrutiny.

▼ Top executives in a large number of organizations now require ROI information.

▼ To justify their contribution, trainers have increased their interest in the ROI process.

A NEW MODEL FOR EVALUATION LEVELS

Developing ROI for training requires a key modification of a classic model. The four-level framework developed by Donald Kirkpatrick in 1959 does not focus directly on the ROI issue. As shown in figure 1 (page 12), Kirkpatrick defines Level 4 evaluation as the results linked to training. These results could take the form of reduced absenteeism and turnover, quality improvement, productivity, or even cost reduction. But this level of evaluation does not require a specific monetary value (cost savings) to be determined. To obtain a true ROI evaluation, the monetary benefits of the program should be compared to the cost of implementation in order to value the investment. In effect, this process moves evaluation to the next level—Level 5 in our revised Kirkpatrick model. Thus, the fifth level of evaluation is developed by collecting Level 4 data, converting the data to monetary values, and comparing them to the cost of the program to represent the return on training investment.

In practice, many organizations are taking evaluation to this new level for a few selected courses, often using some form of sampling. When the ROI formula is developed, evaluation is conducted at all five levels. For example, in the Magnavox case, Level 1 evaluation— reaction of employees—was measured by post- course surveys. Level 2 learning was measured by TABE (Test of Adult Basic Education) scores before and after training. At Level 3, changes in the behavior of employees were measured by daily efficiency ratings. At Level 4, business results were measured through improvements in productivity and reductions in scrap and rework. Finally, at Level 5, ROI was calculated by converting productivity and quality improvements to monetary values and comparing these to the full program costs to yield an ROI value.

THE ROI PROCESS

Calculating ROI requires a process model, as depicted in figure 2 below. The various elements of evaluation (design instruments, levels, and purposes) form the specific data collection plans. A variety of data collection tools, ranging from questionnaires and surveys to monitoring on-the-job performance, are available to trainers.

Once data is collected, the next step of the ROI analysis begins with deliberate attempts to isolate the effects of training on the data items. At least ten strategies have been used to accomplish this: 

▼ use of controls
▼ trend line analysis (time series)
▼ forecasting methods
▼ participant estimates of training impact
▼ supervisor estimates of training impact
▼ management estimates of training impact
▼ customer input
▼ expert estimates of training impact
▼ subordinate input on training impact
▼ calculations/estimations of the impact of other factors.

The next step is to convert collected data to monetary values. This requires a direct conversion of hard data, such as quantity, quality, cost, or time, which is an easy task for some programs such as technical training. For “soft” data, the task is more difficult, although a variety of techniques are used to place values on the improvements. Among the techniques used are

▼ historical costs
▼ supervisor estimation
▼ management estimation
▼ expert opinion
▼ participant estimation
▼ external studies.

The next step is to calculate the costs for the program. Although there has always been a need to capture training costs, the need is amplified with more attention on accountability and the ROI calculation. The ROI formula is the annual net program benefits divided by program costs, where the net benefits are the monetary value of the benefits minus the costs of the program. The ROI formula is as follows:
 

The process of establishing evaluation targets has two important advantages. First, it provides measurable objectives for the training staff to clearly measure progress for all programs or any segment of the process. Second, adopting targets focuses more attention on the accountability process, communicating a strong message to the training staff about the commitment to measurement and evaluation. 

Makaleyi indir