Allison Rossett Guest Post: Evaluation—Words Into Action?

This is one of those topics that never goes away. It reigns supreme in just about every needs study for workplace learning professionals. We say we want to do more and know more. We are eager to check out more tools, and get a better handle on the situation. The topic—evaluation.

We speak fluent Kirkpatrick. When workplace learning and performance (WLP) professionals are asked about the four levels of evaluation, in the USA and beyond, they respond in unison: “Level 1 is reaction, 2 is knowledge; 3 is behavior in the workplace; and Level 4 is results.”

But knowing is not doing, not even close.

An ASTD benchmarking study looked at course evaluations by Kirkpatrick level. It turns out that while almost every course is examined for Level 1 and a third (a third?!?) are checked for Level 2, only 13 percent of courses are examined for Level 3, transfer behavior. Only about 3 percent of courses are held to questions about influence in the field—Level 4.

That data was collected five years ago.

Is it different today?

Technology has changed the shape of workplace learning and performance, shifting learning, information and support into the workplace, and enabling new ways of capturing and communicating data and meaning. ASTD’s own studies of practice, and others, show steady increases in the use of technology for learning and performance. Might this change the current landscape for metrics in learning and performance?

Jim Marshall and I set out to find out. These findings are preliminary. They scratch the surface. Only 110 people responded to our request for participation. We are eager to capture more views from diverse settings. We are eager to find out what you are doing.

Let me tease you with snippets our findings:

• When we asked WHY our respondents gather data, most often they do it to determine participants’ satisfaction with their offerings. Sixty percent reported that they do this habitually. No surprise here.
• We also measure to fulfill compliance obligations, reported as a habit by 48% of respondents.
• Only 25 percent of respondents habitually assess to find out if the learning transfers to performance, and 11% have a habit of seeking data about strategic results. Chew on that.
• No matter the keynotes or magazine covers devoted to integrating talent management with the learning enterprise, only 4% of respondents are investigating this matter.

Our curiosity extended to barriers to metrics.

• Just over half of our respondents said they don’t because nobody asks for this data. Their customers are satisfied with participation numbers.
• Another constraint is the pushback that comes when line managers and executives are asked to play an active part in answering questions about the influence of performance on the field.

These are interesting findings, I think. But we dare not consider them conclusive or actionable, not yet. The sample is too small. You don’t want to spend too long reflecting on findings generated from the practices and opinions of just over 100 colleagues. We need you to add heft to this work.

Our study focuses on why workforce learning people gather data today, how they hope to change and improve those practices, and what gets in their way. Please go to https://www.surveymonkey.com/s/learningmetrics to participate. Your responses are anonymous and very much appreciated. Participation will take only 10 minutes. One other thing—the questions should be interesting to you and will provide you with options for ways to think and talk about our work.

Again, thanks.

Allison Rossett Website