Why Most Training Doesn’t Change Behavior, Part 2

Wednesday, April 4th, 2007

Why Is It So Hard to Attain Kirkpatrick’s Levels 3 & 4

Kirkpatrick’s Four Levels of Evaluation[1] give trainers and CIOs (chief information officers) a framework for understanding learning initiatives. Each successive level of evaluation provides more in-depth information about the impact and effectiveness of any learning program.

Kirkpatrick’s Level 1 focuses on student Reaction. Training participants are asked at the end of a course to evaluate how much they liked the class, the relevance of the material, the instructor’s performance, etc. A negative reaction at Level 1 almost certainly means a training program will not attain to Level 2.

Kirkpatrick’s Level 2 evaluates Learning. The typical Level 2 analysis is a pre and post-test of student knowledge on the subject matter. The difference between the test students took before exposure to the training and scores after the training becomes an objective measurement of learning. Most corporate and educational training programs function at Kirkpatrick’s Level 2. The test functions to certify information transmission.

Level 3 is Transfer. Evaluating training at this level attempts to answer the question: Did the student acquire skills, knowledge, or attitudes that impact their daily behavior? Can what was put in the head transfer to performance? Most trainers say this is the Holy Grail for which they search, for training that goes beyond cognition to implementation. Measuring Level 3 is very difficult because:

1. One may not exactly what or how often to measure;

2. One doesn’t know when the transfer actually happens; and

3. The transfer usually takes place at some distance from the actually training event.

Companies spend millions on sales training programs that employees will hopefully like (Level 1) and in which they learn techniques, aphorisms and concepts (Level 2). Most sales training produces a momentary spike in productivity, but after 4-5 weeks salespeople return to their old habits and the benefit of the training is likely to disappear entirely.

Kirkpatrick’s Level 4 is a rarified event to be sure. Level 4 builds upon the successful attempt to quantify Level 3 and demonstrates verifiable Results. This is the reason the company implemented the training program in the first place — increased production, improved quality, decreased costs, reduced frequency of accidents, increased sales, higher profits, return on investment, and many more outcomes. Indeed, the marketing language of every training program promises results. However, Level 4 is very rarely quantified and is seldom rigorously measured as a routine follow-up to training programs.

The author was part of an experiment to track and measure Transfer and Results in sales training. Salespeople and sales managers committed to participate in a classroom experience followed by an eight-week reporting program. Of 810 participants, 98% gave the classroom experience a strongly positive Level 1 evaluation. Pre- and post-training assessments showed that learning had indeed occurred (Level 2). Results (Level 3) improved for about three weeks and then participation dropped precipitously. Only six participants completed the entire eight week program. What happened? In most cases, salespeople and managers were unable to sustain the disciplines of the training program because the organization had moved on to the next training program sometimes to justify next year’s training budget and frequently as a result of the turnover of key individuals in the company.

So, one reason Kirkpatrick’s Levels 3 and 4 are so difficult to attain is that few companies are able to make long-term strategic commitments that can survive reorganizations and the departure of key players.

Not only are specific results difficult to link directly to training (because Level 3 is so elusive), but also the people who have paid tens of thousands and millions of dollars for training programs want to avoid the embarrassment of facing up to the fact that the expensive programs they bought don’t really make much difference in the long run. Training vendors are learning to leverage this psychological dimension by increasing the cost of training to increase the cognitive dissonance of those on the hook for results. Because CIO’s paid so much for the program, they will believe the program has to work and believing is seeing.

What’s needed is a training application that allows stakeholders in corporate learning to more clearly validate Levels 3 and 4.

Barnett Learning Group has developed an on-line platform that enables training professionals to certify Kirkpatrick’s Level 3 and Level 4 measurements through a custom LCMS called ProficiencyMaster®.

---

 

[1] Kirkpatrick, D.L. (1994). Evaluating Training Programs: The Four Levels. San Francisco, CA: Berrett-Koehler.