Measuring training’s return on investment (ROI) / Yekta ÖZÖZER
Investments in training and other HR projects have been increasing each year in terms of investment size, content and number of people exposed, while at the same time many of them are considered to be “non-measurable” investments. Given the pressure of competition, time and feasibility on all types of corporations, today’s CEOs, GMs, VPs in general and training and HR officials and managers in particular have to be much better business people, speaking and making decisions with facts and figures and justifying the feasibility of the investments. However, many HR executives and their departments in the position of requesting large investments, but being unable to deliver any factual feasibility data. Moreover, many of them are criticized for not being factual and lacking business sense.
The following are some examples from important projects in the training and/or HR departments’ 2013-2014 agenda in Turkey and in some other countries we gave ROI consultancy: A 12 month long training program by a university in arrears leadership and management for the top and middle management team of a large corporation, costing US$ 1 million; A 14-month Sales Training and Coaching Program for a retail bank, costing US$ 2+ million; benefits and disadvantages of flexible salary and benefits packages to employees of a large energy group of companies; technical banking programs transitioned from classroom training to e-learning or to blended learning; product and service training for a large insurance company which has over 2,000 agents nationwide; establishment of a corporate academy, etc.
“The ROI Methodology” is the key to transforming these decisions from subjective arguments into reasonable, well measured arguments based on solid metrics.
Some background information on ROI methodology
The Harvard Business Review proclaimed ROI as the tool to measure the payoff of capital investments in the 1920s. In recent years, the application of the concept has been expanded to all types of investments, including human capital investment. In response to the need, The Phillips ROI MethodologyTM, which forms the basis for the certification described in reality-based scenarios, started to be implemented in the 1970s at Lockheed-Martin. There, Dr. Jack Phillips conducted an ROI study of the company’s cooperative education program involving 350 co-op students. Even in those days, there was pressure to show results. Today, many corporations use The Phillips ROI MethodologyTM to measure the ROIs of HR investments. Over 4,000 individuals participated in the certification workshops, where process which is now conducted in over 30 countries, including Turkey, the Middle East and Europe.
The evaluation framework
Figure-1 shows the types of data measured and our progress with collecting and analyzing it in the ROI process.
Level 1: In this level of measurement, the reactions to program’s relevance, appropriateness and trainer’s performance from participants are measured with generic questionnaires and surveys.
Level 2: Learning measurements focus on if participants learned new skills, knowledge, and know how about the training or HR program. In the “Management and Leadership Development” program example, employees must learn subjects like different motivational methods, four different phases of situational leadership, career development and ways to give feedback to their employees.
Level 3: This measures application to the business and includes all the steps like the frequency and use of learned information, knowledge, and skills, actions, tasks, and processes involved. In the “Management and Leadership Development” program example, managers must take actions like guiding, motivating and coaching their employees. While the Level 2 and 3 evaluations are important, they still do not guarantee a positive business impact and/or a positive ROI.
Level 4: Measuring business impact focuses on the actual business results (like output, quality, sales, costs, time, employee engagement or turnover and customer satisfaction). In the “Management and Leadership Development” example, business impact might include improvements in sales, process time or quality, timely delivery, collection rate, inventory level as well as reduction in employee turnovers incidents of violations, fraud, discrepancies and some other losses.
Level 5: ROI is the ultimate level of evaluation, where the HR program’s monetary benefits are compared with the costs. Since our ROI Methodology is deliberately a conservative approach protecting training and HR executives from various critics, we recommend, direct or indirect, all expenses to be fully loaded to the cost figüre
Calculating ROI
ROI (%) = (program benefits - program costs) / program costs
If the fully loaded cost of the “Management and Leadership Development” program above is $202,348, while the proven benefits calculated (through isolating all the external factors) are $583,426. Then the ROI = ($583,426 - $202,348) / $202,348 = 188%. One of the most critical issues of the Methodology is that all the costs including the ones after 12 months after the implementation have been included, while the benefits after 12th months are excluded in the ROI calculation. Even if the benefits may continue after the first year, the impact usually diminishes and isolation mistakes may arise. Therefore, benefits after the 12th month are excluded in short-term projects. However, in some HR projects like Management, Leadership, Coaching and Mentoring traininings or ERP HR module implementations, where the long term benefits are evident, this fact is evidenced at the planning stage before the data is collected and benefits after the 12th month are also taken into account.
Thanks to this protective approach of the methodology for the training and HR executives, whatever the figure they submit to the management (i.e. ROI as 188%), virtually nobody can claim that this investment was indeed less feasible (let’s say 180%). This conservative side of the model increases the confidence of the whole management in the method and hence in the training or HR department.
ROI process model
Evaluation planning: The first step of planning is to develop ultimate objectives for the training. These range from developing objectives for Reaction (Level 1) to developing an objective for the ROI (Level 5). A specific training should have multiple levels of objectives.
With the objectives in hand, the next step is to develop two important planning documents. The Data Collection Plan indicates the types of data collected and its sources, the methods for data collection, the timing of collection and responsible people for collection. The ROI Analysis Plan details how the results will be isolated from other than training influencers, data will be converted to monetary values, the appropriate cost categories, the expected intangible measures and the anticipated target audience for communication.
Collecting data: Data collected before or during the launch of the training program is compared with post-program data.
Isolating the effects of the HR/training program: While this step is often overlooked, it is indeed one of the most critical issues in most evaluations in our works. It is essential because many factors will influence performance data after a training program is implemented. Specific strategies in this step will pinpoint the amount of improvement directly related to the training. The result is increased accuracy and credibility of the ROI calculation. At least one of the total of 10 strategies is used to address this important issue. Here are the three most common strategies:
1. A pilot group of participants in a training program is compared with a comparison (control) group not participating in the program to isolate program impact.
2. A trend line is determined from the pre-program data. This trend is projected to the post-program period as if the training program had not taken place and the difference between the trend line and the actual postprogram data are taken.
3. Participants/stakeholders estimate the amount of improvement related to a training program. Then the potential errors due to other factors are discounted from these estimates and hence the most reliable improvement data is used.
Converting data to monetary values: To calculate the ROI, business impact data need to be converted to monetary values and compared with HR/training program costs. This requires a monetary value to be placed on each unit of data connected with the HR/Training program benefit. 10 strategies are available to convert data to monetary values.
Tabulating the cost of the HR/training program: The denominator of the ROI formula is the cost of the HR/ training program. The typical cost components are: initial analysis and assessment, development/design cost for the HR/training program, participant/stakeholder time for the program, implementation and improvement costs, administration expenses, evaluating and reporting costs, etc. Due to our conservative approach, we fully load all these costs to the total cost figure.
Reporting results: A final operational step of the ROI process is to generate an impact study to document the results achieved by the HR/training program and communicate them to various target audiences, including managers who knows the process. A variety of different reports and formats can be used in this reporting process, ranging from a complete impact study described above to a one-page executive summary.
APPLICATION AND MEASURES
Another major element of the ROI methodology is application and participants involved in ROI Certification are required to conduct at least one ROI project study before they receive a ROI Certification. The types of projects undertaken vary and represent any type of project, program, event, or initiative imaginable. Wide range of different metrics can be used in different projects. For example, productivity/output, quality, process time savings, efficiency, costs, employee satisfaction survey results, customer satisfaction rates are typical measures for Coaching and/or Leadership and Management Development Programs, while time savings, solution time, quality, budgets met, problems not repeated are for a Project Management or a Problem Solving Training.
BENEFITS OF ROI USE AND RESULT
Measuring and presenting ROI with a complete set of data result in earning the respect and support of the senior management. If a program is not effective, and the results are not materializing, the ROI process will prompt changes or modifications.
While the ROI methodology has enjoyed success, it is estimated that in most countries only 10%-15% of HR and Training departments are using this methodology (the rate is lower in Turkey). Yet, researches show that as many as 75%-80% of them plan or wish to use ROI. This huge gap between “actual use” and “plan/ wish” (please refer to Figure-1) shows both a great challenge and a great opportunity. As more organizations understand the success that can be achieved with this methodology, more organizations will embrace it, where the internal support and financing for their projects will also increase.
Download