Wednesday, November 1

If you aren't piloting, you aren't doing your job, and you probably won't keep your job

I have seen hundreds of situations in the last 36 months where a training group had a training experience they said they wanted to consider for implementation.

  • Around 25% did not evaluate the program at all.
  • Around 25% evaluated the program by skimming it.
  • Around 20% evaluated the program by asking a few friends to skim it.
  • Around 20% evaluated the program by putting in a small group and asking the participants if they liked it.
  • Finally, around 10% evaluated the program by putting a group through it and measuring the effectiveness.
As a bit of a post script, when re-engaging training professionals a year later, there seem to be a significant correlation between those who didn't pilot well and those that did not have a job. Those most afraid of bothering people with a pilot became those most vulnerable.
Posted by Clark Aldrich at 8:30 AM

1 comment:

Dave Lee said...

I'll trust your data on the unemployeed nature of non-piloters, but I can support the fact that piloting almost always finds problems and areas for improvement.

Throughout my career I've managed easily 100+ pilot sites for everything from companion workbooks, to textbooks, software activities, and full LMS's. What type of breakdowns do pilots catch? Here are 5 (well 5 1/2) of the biggies:

1) False assumptions about content progression. Your SME's already know the answers and have created short cuts for themselves that make them more efficient. The SME's you have review the content now the same shortcuts. When key steps are missing, learners don't get it you have a breakdown that needs to be fixed.

1b) Jargon, jargon, jargon. SME's use it. Learners don't know it. Facilitators often don't know it. (which can undermine the facilitator's credibility.

2) False assumptions about pacing. That activity you thought would take 15 minutes requires 12 minutes of setup and then runs for 30 minutes.

3) Ideas that looked good on paper but just didn't fly in real context. That team activity using hula hoops to teach Venn Diagrams sounded great but what a flop. The older employees spent time showing the younger employees how a hula hoop works.

4) Great activity once the facilitator figured out the poor facilitator guides.

5) Lack of clear link between activities and the learning objectives. Invariably signaled by an insistent raised hand follow by "why would we want to do this anyway?"

Now when would you want these things to happen? In a quiet pilot test that hardly anyone knows about or at company-wide launch with ever line manager on hand wondering why their people are here instead of on the job?

11/02/2006 5:05 AM

Post a Comment

Subscribe to: Post Comments (Atom)