Monday, April 30

Gamification Blog Book Tour, Week Three Stops and Week Two Recap

TODAY"S BLOG STOP:  May 4: UL EduNeering Online Compliance Training Blog

The blog book tour has been a lot of fun, with interesting comments and exciting dialogues and some in-person stops. Here are the stops for week three (with a few changes) and a recap of week two.

April 30: Enid Crystal New York ASTD Chapter Blog
May 1: Jeanette Brooks Word of Mouth blog.
May 2: Cammy Bean Learning Visions
May 3: Koreen Olbrich Learning in Tandem
May 4: I am appearing "live" at the Eduneering Knowledge Summit in Baltimore, MD. Complete with book signing. The blog for the day will be UL EduNeering Online Compliance Training Blog which will have slides and information from the session.

Recap of Week Two

Clark Quinn started the week off at Learnlets with a discussion of the word "gamification" (he would prefer a more meaningful term like "engagification" especially since "gamification" does seem to carry some negative connotations. He provided a balanced and well described critique of the book. You can read Clark's post Kapp’s Gamification for Learning and Instruction.

Next Karl Grieb of the ASTD Philadelphia eLearning SIG provides a description of the content of the book. He describes the break down of the elements of games and points out the section describing ADDIE versus Scrum as a development process. He also highlights the writing about different types of motivation including John Keller's ARCS model. Read the post here.

Then Debbie Richards from Take an e-Learning Break wrote about four themes from the book including "matching game results with game design. She also talked about the Cisco Binary game. I also did webinar for the Houston Chapter of ASTD arranged by Debbie. You can see the slides here.

The week ended on a fun note with Connie Malamed of The eLearning Coach creating a game/post called Are you a gamification wizard? Play the game. See how well you can do. There were also some live appearances. New York City I traveled to New York City and presented at a joint ASTD Special Interest Group (SIG) meeting of the NYU Higher Ed SIG and the eLearning SIG. It was a great time and I met some wonderful folks. Here are a few pictures of our interactive session. I'd like to thank Amy Lui Abel and Enid Crystal for making that meeting happen. Here are some pictures from the event.

You can see the slides for the presentation here.

Lehigh Valley

As part of the tour, I severed on a panel about gamification for the Lehigh Valley Chapter of the Association of Information Technology Professionals. The title was "The Computer Game Industry: Not for Kids Any More." We had a great panel discussion with lots of questions and engaging conversation.

 The panel included myself, Jason Brozena of Caronet which is a hosting company that provides data center services to organizations, including numerous game companies. Having a data center's perspective helped to broaden the attendees understanding of how games are being hosted and other back-end services/technologies. And Larry Wolfe of LiquidInt. Larry will introduce Liquid Mobility Bridge™. This tool was designed to work across various mobile platforms. Larry will also discussed the concept of gamification of business applications. I'd like to thank Mary Rasley, Steve Steven Weitz and Thiep Pham for including me as a panel member.

Thursday, April 26

Measurement Part 4: Implementation Tips and Technology Tools

Designing the program is only the beginning - data are even more valuable in enabling you to consistently improve your learning programs. This week, we’ll discuss the “red meat”: Implementation, data analysis, and program improvement.

Baseline Assessment
Once you have completed your program design, start the implementation phase by taking a baseline assessment. This means using the tools and measurements you have identified to assess your target audience’s current status and performance - and your organization’s status and performance.

Remember that throughout the assessment process, you will likely identify administrative, logistical, and structural problems that can’t be solved through training. While this can be frustrating, it is an opportunity. You may not be able to solve all the organization’s challenges through your work, but as you isolate the factors you can change through training, you can share your findings on needed change within your organization. You can document the administrative, logistical or resource obstacles that are interfering with learning and productivity in your organization - and position yourself and your department as an internal consultant to your organization.

Program Implementation
The theory of change, once articulated, provides the metrics of your success. Your goal, as you implement your learning initiative, will be to collect assessment and performance data, and continually compare this data to your program goals.

Iteration and Improvement
You will know you have succeeded in generating positive change once you can demonstrate the uptick in the metrics you planned to address through your theory of change. Conversely, if you aren’t seeing positive movement in your targeted metrics, you can move into higher level analysis: Evaluation of possible causes for the lack of success, and experimenting with possible means of improvement.

There are two crucial characteristics of the iteration and improvement phase:
1) Lack of success is not failure. If the first incarnation of your learning program is not successful, that means you have useful data about what doesn’t work. Don’t lose heart, but refocus on where the connections between your activities and desired outcomes broke down.
2) It’s never over. Improvement of a learning initiative lasts as long as the program’s goals remain in place. As circumstances change, it will continue to be your responsibility to respond to feedback, improve the program and respect your learners’ needs.

Lessons Learned
In closing, here is my most important lesson learned: When you’re working hard to improve your program, you won’t be overwhelmed by too much data. In contrast, you will be thrilled to have access to great data and may even regret all the questions you didn’t ask. Building processes for collecting and analyzing performance data in your organization will empower you to make informed decisions, allocate training department resources effectively and focus on the changes that will make the biggest improvements in your organization.

Further Reading:

Image credit: logos noesis on Flickr

Monday, April 23

Tour Stops for Week Two of Gamification of Learning and Instruction Tour

TODAY'S STOP:April 27: Connie Malamed The eLearning Coach: Are you a Gamification Wizard This week promises to be an exciting second week of the Gamification Blog Book Tour. Here are the stops for this week.

April 23: Clark Quinn Learnlets

April 24: Karl Grieb ASTD Philadelphia Chapter

April 25: Webinar Presentation for Houston ASTD Chapter "What Research Tells Us about Games, Gamification and Learning" Join the webinar.

April 26: Debbie Richards Take an e-Learning Break And a live appearance by Karl at the NY ASTD Chapters combined SIG Meeting. If you are in NY, you may want to register and attend.

April 27: Connie Malamed The eLearning Coach.

Saturday, April 21

Gamification of Learning and Instruction Blog Book Tour Week One Recap

Week One Recap

 The first week of the blog book tour has ended and it has been a fantastic week with informative blog posts, information  and opinions about gamification and even a bit of controversy.

 The week opened with a posting of the tour stops on the Learning Circuit’s Blog and the Kapp Notes blog and then introduced everyone to the Facebook page for the book and then the discussion really ramped up with Jane Bozarth talking about the how the book takes a common sense look at the subject. Next, New York Time’s bestselling author Kevin Kruse told us how articles appearing within the last year in notable publications such as BusinessWeek, Forbes, Fortune and even the Harvard Business review are talking about how gamification is impacting marketing, service and employee satisfaction (notice training seems to be absent.)

Then on Friday Judy Unrein discussed how the book can benefit instructional designers and Rich Mesch made us all hungry when he reminded us that just like Chocofication (adding chocolate to everything) is not a good idea, neither is gamification of all content a good idea. In some areas it doesn't work or even make sense to add "Gamificaton". We need to be careful how we apply "Gamification." It is not a universal cure-all.

Issues with Gamification and It’s Implications

Another guest on the tour, although not a scheduled stop, has been Kathy Sierra who is a self-confessed gamification curmudgeon and author of a widely popular series of test preparation books to help people pass Java tests to become certified and not only does she prepare people to pass those tests, she has created certification exams that are used to certify programmers. Her test preparation books are some of the best selling on the topic.

 Kathy has brought the perspective that gamification is not good--at all. She doesn’t like the word “gamification” and she had her “heart broken” because so many people that she respects are involved in this tour talking about gamification. She is worried about my insistence on using the word "gamification" (including my urging of others to "take back the word").” She states that “nearly every game scholar and professional game designer (real games, not just Zynga game-like things) is adamantly opposed to the word for many reasons including how misleading it is by including the word 'game'".

She was also “offended” that during the tour I offered a whitepaper for anyone who wanted to leave a comment on every blog entry. She felt that was a crass use of gamification and that I was clearly using an extrinsic reinforcer offering a reward in a feeble attempt to market the book. She felt that in an industry where knowledge is valued that withholding knowledge to shape behavior or action was wrong. Kathy felt that technique was “LEAVE a COMMENT for POSSIBLE WIN scheme” and that it appealed to the basest aspect of Gamification.

In Defense of the Term Gamification

Her concerns are not without merit but I think there is another perspective to consider, especially with her dislike of the term“Gamifiaction”  While “nearly” every game scholar and professional game designer is against gamification and some who initially were proponents of the term have backed off, these people are not controlling the discussion about gamification within businesses and corporations.

The CEOs, Vice Presidents, executives and managers are not tapped into the game developer industry; for the most part they don’t attend the Game Developer’s Conferences, they don't read game developer magazines or blogs.  Instead, they read reports from Gartner that indicate 70 percent of Global 2000 organizations will have at least one gamified application over the next five years. And reports that say the overall market for gamification is predicted to grow to $1.6 billion in the next ten years. (we all know Gartner analysis are fabulously optimistic in their predictions but someone is reading those reports and paying for the analysis).

Additionally, as NY Times bestselling author Kevin Kruse told us, CEOs, VP’s, executives and managers read Forbes, the Harvard Business Review, BusinessWeek and Newsweek which all within the last year have carried articles about gamification. Like it or not the “Gamification” term is out of the bag and it’s not getting back in or disappearing or falling out of favor with executives. The gamification message is targeted toward the major decision makers within organizations and is not being lead by major scholars or figures within the game industry. It's being lead by vendors, marketers and others who can, and are, getting  the word out about gamification and its working.

Learning and development professionals must now react to requests for Gamification, we are not driving the discussion--we are forced to react. And many times we are not even in the discussion at all. In fact, many proponents of gamification feel learning and development professionals have nothing to contribution to the conversation about gamification at all. Actually we have the most to contribute. We understand human motivation and how people process information and how they learn and how to shape behavior so it lasts. We should be in the conversation or it will go in unhealthy directions and have negative consequences for us.

Two Choices

Now we have two choices, one is to “Just Say NO to Gamification.” We can ignore it, we can rally against it, we can talk about how no “serious” game developer believes in gamification but, at the end of the day, business leaders are bombarded by gamification messages in magazines they read religiously by organizations that have the funds to spread the gamification message.

The “anti-gamification” tribe has little funding, organizational structure or reach into CEOs and VPs. So waging a war against the term “gamification” will, ultimately, not be productive. We will just be ignored or bypassed when “serious” gamification issues need to be discussed--you know the kind mentioned in the Harvard Business Review. We may not like it but it’s already happening.

In fact, if we learning and development professionals turn our back on gamification, refuse to take part in this narrowly defined if/then extrinsically motivated movement, then the CEOs, the business managers and executives will go somewhere else. My fear is that a CEO will walk into a training department after having read an article in the Harvard Business Review about Gamificaiton and demand that the training department create a gamification program to train sales reps. Then the training department either doesn’t know anything about gamification (because learning and develop professionals refuse to use the term and its not talked about by the major voices in the field) or they say “no” we don’t do gamification under any circumstances.

The CEO shakes her head and then goes to the marketing department and says, “Can you create a sales training program around gamification?” and they say “Of course!” Now non-learning and development professionals are using gamification, perhaps at its most extrinsic level to create training. And, if it works or even appears to work, the learning and development professionals lose credibility and relevance. We will be out of the loop and away from business discussions.

We can’t sit with the C-Suite people or in meetings uninformed about this thing called “Gamification.” Sure, we can decide to name it something else like “Gamefullness” or “Activityification” but then learning and development professionals will be speaking a different language than executives. We already have this problem by the ton. We can't decide to make it worse by creating a substitute word for “gamification."
 Learning and development professionals should have learned by now that we have to use the language of business to work with the leaders of business to obtain credibility to have influence.

Inventing another term or refusing to discuss that low-brow "gamification" concept doesn't help the profession one bit because we  appear out of touch or not in sync with the business units.

The other choice, the one I have chosen, is to educate ourselves about this thing called Gamification and to expand and broaden the definition—not so that the definition or term is meaningless—but so that when the CEO comes to us and says "can you do gamification of a sales training program", we can say “yes.” And then we intelligently add story elements, challenges and the main tenants of Self-Determination Theory autonomy, competence and relatedness to the training.. The elements of games that actually make a difference, that actually add intrinsic value instead of a crass use of points, rewards or badges. And we can still call it gamification.

We have then met the needs of the CEO and created a meaningful learning experience. Additionally with this approach, we are not left out of the loop. We can shape the gamification discussion about the sales training to be more instructionally sound than if we were not involved or left out. By learning about gamification we become part of the conversation and are not isolated on the sidelines.

 Why Respected Individuals are Talking About Gamification

The reason I enlisted so many respectable people in the learning and development field and the reason I think they agreed to participate is because we can’t hope the term gamification goes away. We need a general discussion within the field of the term, its positive and negative aspects. The term won’t go away no matter how much some of us hate it.

We must talk about Gamification and examine it and see how it can fit into what we are doing. We must be participants in the conversation about gamification and try, in some way, to shape the term. If we don’t, the entire concept and application it will go into a direction that, from an instructional standpoint, is untenable.

We have the ability to influence the application of gamification in the field of learning and instruction right now. If we wait or hesitate we will loose our opportunity. It’s not too late as some would have us believe.

Gamification Resources for L&D Professionals 

In fact, before "The Gamification of Learning and Instruction" was released, the learning and development field had no book about gamification that focused on the learning theories behind gamification or a listing of what gamification is or is not from a learning perspective.  The purpose of the book is not to “glorify” gamification or to get people to be “quotable for having said useful, positive things about gamification.” The purpose is to have a measured, civil discussion about gamification and how it applies or doesn’t apply to learning situations.

The goal is to create competence within the learning and development community to be able to speak intelligently about gamification and decide when it is appropriate and when it is being used to manipulate people and, at times, the distinction can be tricky.

For example, Kathy Sierra rightly pointed out that I was using an extrinsic motivator in terms of proposing a reward (a whitepaper) in exchange for an activity (leaving a comment). I agree that creating that type of arrangement was a crass use of extrinsic motivation. I admit my error and will make the whitepaper available freely to anyone (of course I have to write it first—it will be available end of May).

Helping me to see and correct my use of extrinsic motivation is the point of having open, civil conversations  about things like external motivators. Sometimes we need others to point out how we might unwittingly use external motivators to influence behavior and we might not even be aware of it.

Certification Exams as External Motivators

Even the most intelligent people can unwittingly fall into the "trap" of relying on extrinsic motivation to spur learning results. For example, I wonder how consciously aware Kathy is of her use of  strong external reinforcers in her creation of certification exams and in her creation and marketing of test prep materials that support those exams.

When people study for these extrinsically motivating events known as certification tests (or in gamification parlance “badges”) do these people perceive that that they've been manipulated into preparing to pass these tests? Do they realize their intrinsic motivation to learn a programming language is being systematically and methodically undermined because, as Kathy has suggested, the result of rewarding someone for something they would have done anyway is ultimately de-motivating. In fact many believe the original intrinsically motivated behavior will disappear after the extrinsic reward is removed.

To paraphrase a wise and informed person on the topic of extrinsic motivation: while the programmers Kathy urges and encourage to read her books to earn the badge/certification are all smart, savvy, brain aware people it makes no difference because that knowledge does not protect them from the damaging effects of putting such weight and value on the EXTRINSIC reward of certification. The negative, undermining effects all happen at a level of conscious processing for which Kathy’s poor readers lack the "security clearance" to access let alone override.

This If/then proposition inherent in Kathy's test prep books is the exact same offense Kathy correctly and accurately accused me of committing with my whitepaper. Kathy is extrinsically motivating people to buy her books so they obtain the reward of certification. She isn’t marketing these test prep books as a way to learn programming or as a way to develop a love of programming or as a way to build competence in programming; she is marketing them as a way to obtain an external reward--to pass a test.

So I urge Kathy to do the same thing I have done. I divorced the reward from the action; I am offering the whitepaper freely on the web for no cost. Kathy, please seriously consider offering anyone who wants certification a free copy of your book. Or consider dropping the certification. Or, only write books which tap into intrinsic motivation. I urge you to become the model for which you want others to follow—don’t use external rewards as a motivational tool. Show learning and development professionals how to use only intrinsic motivational techniques so that your readers develop a competence level in programming that is so high, no certification is needed.

Abandon your test prep books and write more books fostering the love of programming through intrinsic devices like autonomy, competence and relatedness. Stand up and no longer present the proposition to your readers that if they buy your book, they will ace the certification exam and receive the extrinsic reward of certification. Be the change you hope to see in others.


This week has certainly been a great first week for the tour and next week is a great line up as well. I have learned a great deal and hope you have too. The interest in the tour is growing and next week we’ll be announcing new tour stops, webinars and other events that have joined the tour.

This upcoming week, we have Clark Quinn who I purposefully asked to be one the stops because he doesn’t like the word “Gamfication” and I wanted everyone to know of his concerns and perspective. And, in light of this discussion it will be interesting to read what he thinks. We have the Philadelphia Chapter of ASTD with Karl Grieb making an appearance as well as Debbie Richards of Texas with her blog Take an e-learning break.

I am doing a live appearance in New York at at Special Interest Group meeting and my friend Connie Malamed of The eLearning Coach will discuss her view of the subject of gamification.

I look forward to their postings and hope that you too decide to post as well. Now that the LEAVE a COMMENT for POSSIBLE WIN scheme  has been removed (and I do apologize if that offended anyone), I urge you to freely post your ideas and thoughts.

Let the community know what you think,  Should we ignore the term "gamification"?  Should we denounce the term as crass if/then behavioral reinforcement? Should we attempt to shape the term and concept or should we find another term?

See you on the tour and if you want to learn more, pick up the book at the ASTD Book store or check out the Pinterest page for Gamification Happenings.

Thursday, April 19

Creating a Measurement Model

In the first two posts in this series, we discussed the value of data and the process for using the theory of change model for learning design. This week, we’ll walk through creating a measurement model for a fictional company, Acme Donuts.

As you work through the learning initiative design process, the key challenge is to balance strategy with tactics: Connecting specific actions that accomplish high level goals. The Theory of Change model, which starts with broad goals and asks you to work backwards to specific actions, forces you to make the connections between each step clear. The narrative step also challenges you to explain the connections: An excellent exercise for testing assumptions and discovering gaps.

I created this sample narrative to outline the many factors relevant to designing a learning program that includes a measurement model from the beginning.

Developing a Program Narrative: Acme Donuts Example

  1. What are your initiative’s goals? To develop functional goals, you need stakeholder buy-in, alignment with organizational goals and specific descriptors. Make sure you’re not developing goals in a vacuum.

    Acme, Inc. is a manufacturer of donuts. Our goal is to increase profits 15% this fiscal year. To do that, we must meet two key preconditions: 1) Increase the recurring revenue generated from long-term sales agreements; 2) The sales team must develop strong communication, sales and negotiation skills to negotiate and close sales agreements.

  2. Whether conscious or unconscious, everyone makes assumptions. A key step in developing a functional initiative is ensuring that you understand what variables you will hold neutral as you target changes in your organization.

    We are making two key assumptions: 1) The product is appropriately priced for its quality. 2) There is sufficient demand in the marketplace to enable us to increase sales.

  3. Work backwards from your goals to the preconditions that must be met. Limit the scope of your discussion to the changes that can be affected through training and development activities. If, for example, Acme’s Donuts just aren’t very good, training sales people more effectively may not do any good at all.

    Both sales team members and managers have described needing opportunities both to study and practice the techniques of sales and communication and to work together as a team to identify best practices and improve performance over time.

  4. Outline specific initiatives you will undertake. Ensure that you outline how those initiatives connect with your preconditions and goals.

    To reach the targeted competencies of communication, sales and negotiation skills, we will embark on a multi-phase initiative: Participation in online learning courses to expose sales team members to the techniques of diagnosing and creating a sales strategy. Group workshops with their peers to enable them to practice their communication and negotiation skills. Connecting each participant to a mentor who will meet regularly with them to enable them to seek feedback and advice. Creating a social network to enable team members to share successes and resources.

  5. Develop specific metrics based on your goals. Ensure that you have a variety of metrics at each level and step of your initiative. Make sure you can assess specific factors so that you can, in turn, make specific improvements to your learning initiatives.

    We will measure the success of our initiative on a monthly basis with these metrics:
    - Sales team’s self-assessment of communication, sales and negotiation skills on a detailed scale
    - Sales manager and VP of Sales assessment of individual and team performance
    - Monthly recurring sales per salesperson and as a group; the rate of change in monthly recurring sales
    - Profit growth

  6. Set expectations. Ensure that you and your colleagues understand and agree on goals and reasonable expectations for results.

    We expect to see some change in the metrics in the first two-three months of the initiative, but we believe that we will see significant improvement within six months as sales team members have an opportunity to implement and improve their new skills in cooperation with their peers. We will use data from assessment and from performance to continually create online courses and group workshops targeting areas of weakness. To the extent possible, we will use internal experts to lead workshops both in person and virtually.
In closing, this example is just the beginning. Check out the Theory of Change website, which provides an excellent case studies from a variety of social and job training programs.

Further Reading
Image credit: dasroofless on Flickr

Monday, April 16

Gamification Blog Book Tour Underway!

See below for the entire list of tour stops!

The blog book tour has started as ASTD and I kick-off a 25 stop blog book tour for the ASTD co-published book The Gamification of Learning and Instruction: Game-based Methods and Strategies for Training and Education. The book is available at the ASTD Book Store.

The tour includes stops at several ASTD chapter web sites including Philadelphia, New York and Houston, Texas.

You are welcome to join the tour, are urged and encouraged to join the tour.

Leave a comment on this posting linking to your blog and it will "officially" become part of the tour. The tour currently includes some well known bloggers and some bloggers you really need to know but we want to expand it with your input, ideas and concepts related to Gamification.

Here are all the tour stop dates.

The blog book tour is a virtual tour so you can just follow along stop by stop. If you don't have the book yet, stop by the ASTD book store a pick up a copy.

The Twitter hashtag for the tour is #gamiLI

There is a Pinterest page for the tour.

And a Facebook page for the tour, stop by and give it a LIKE:)

Here are the tour stops, the day of the stop and a link to the stop are indicated below. You can follow along by going blog to blog and leaving a comment. If you stop at every stop and leave a comment, you will receive a free whitepaper "The First Five Steps to Gamification of Content, Curriculum and Courses."

Also, on April 26th, Join Karl at the Houston ASTD Chapter's Webinar for a live chat and presentation by the author.

Week One:

Oops, already had a change in venue as my scheduling abilities appear to have been less than stellar, please see below for today's stop.

April 16: Learning Circuits Blog.
April 17: Gamification Facebook Page
April 18: Jane Bozarth's Bozarthzone
April 19: Kevin Kruse, Keven Kruse Blog. He is NY Times bestselling author of We: How to Increase Performance and Profits through Full Engagement
April 20: Rich Mesch Performance Punctuated and he will be joined by Judy Unrein OneHundred Forty Words .

Week Two:

April 23: Clark Quinn Learnlets
April 24: Karl Grieb ASTD Philadelphia Chapter
April 25: Webinar Presentation for Houston ASTD Chapter "What Research Tells Us about Games, Gamification and Learning" Join the webinar.
April 26: Debbie Richards Take an e-Learning Break And a live appearance by Karl at the NY ASTD Chapters combined SIG Meeting. If you are in NY, you may want to register and attend.
April 27L Connie Malamed The eLearning Coach.

Week Three:

April 30: Amy Lui Abel New York ASTD Chapter Blog.
May 1: Cammy Bean Learning Visions
May 2: Tom Kuhlmann Word of Mouth
May 3: Koreen Olbrich Learning in Tandem
May 4: "Surprise Blog Appearance"

Week Four:

May 7: Mike Qaissaunee Frequently Asked Q
May 8: Larry Hiner drlarryhiner
May 9: Catherine Lombardozzi Learning Journal
May 10: Brent Schlenker Elearning Development
May 11: Zaid Ali Alsagoff Zaid Learn

Week Five:

May 14: Andrew Hughes Designing Digitally
May 15: John Rice Educational Games Research
May 16: Christy Tucker Experiencing E-Learning
May 17: Justin Brusino ASTD Learning Circuits Blog (we come full circle to discuss the tour and the gamification concept)
May 18: Karl Kapp Kapp Notes. The author provides reflections and lessons learned from the tour.

So join us for this exciting tour and social media event to discuss the pros and cons of Gamification and what it means to learning and development professionals. And follow us on Twitter at #gamiLI.

Gamification of Learning and Instruction

Promote Your Page Too

Friday, April 13

Measuring Learning: The Theory of Change

Last week, we kicked off Measurement Month with a discussion of why measurement matters. This week, we’ll dig into connecting measurement and design to build the metrics in from the beginning.

Throughout this post I’ll use an example from a training program I worked with earlier in my career to illustrate how the measurement models can be put to work in training.

The Alaska Example: Job Training for Teens

In a my last job, I worked with Anchorage Youth Employment in Parks (YEP), a program hiring Anchorage teens to complete park improvement projects while learning job skills in trail building, construction, and habitat restoration. In partnership with a variety of community organizations, this program accomplished two key goals: Complete park improvements and train teens in in-demand job skills.

As program partners secured public job training funds for this program, it was essential to ensure YEP was successfully developing employment-ready teens. After the program’s pilot year, program partners worked with a University of Alaska sociologist to create a data model to measure the YEP program’s effectiveness and continually improve the program.

YEP used the Theory of Change model to guide the development of program metrics and evaluation. The Theory of Change model offers program designers a helpful guide for planning and measuring learning and development. The Theory of Change looks like this (from “How does the Theory of Change Process Work?”):

Step One: Identify Goals
Start your program design with the basics: Goals. Ask yourself and your stakeholders straightforward questions: What skills must participants have when they finished the program? What new responsibilities will they be qualified to take on next? To develop specific and measurable goals, conduct stakeholder interviews and focus groups. As you design your program goals, make sure you have a thorough understanding of your target audience to design initiatives that meet both their needs and the professional growth they wish to accomplish for themselves.

In the YEP example, the goals for the students were to develop competency in several professional outdoor skills: Forest maintenance, trail building and streambank restoration. Participating teens were required to demonstrate technical competence in these professional skills as well as leadership and teamwork abilities.

Step Two: Connect the Preconditions and Identifying the Changes
Each program goal will have a chain of needed preconditions, which in turn require changes from the baseline data. The program design phase is an exploration of each step needed to achieve the preconditions necessary for the end goal.

In the YEP example, program managers worked with experts in each of the subject areas (trail-building, forestry, and watershed restoration) to assess the competencies necessary to reach the desired end goals for the teens. We developed activities (both training programs and work projects) that would enable participating teens to explore different skills and apply them to real work environments. We made sure our summer-long program connected all of the dots between participating teens’ beginning qualifications and our desired end goals.

Step Three: Develop Indicators
The next step is to create a data framework for the “before and after” states achieved through a training program. In the Anchorage example, this meant a case study of the target audience of our training program. We identified their likely background, education, and their desired outcomes from the program, and created scales to measure where an individual would stand within that range at the beginning of the program and the end.

We included measurement of both teens’ soft skills, like leadership, communication, and cooperation, and their specific job skills, in trail-building, forestry, and watershed restoration. We developed detailed assessments for teens to complete at the beginning and end of the program and a shorter, simpler questionnaire that they filled out on a weekly basis. Teens were asked for both self-assessment and for qualitative feedback, to which we later assigned data values. We also asked teens for feedback about the program itself.

In addition to asking the teens to assess themselves and the program, we asked both crew leaders (team leaders) and the program supervisor to complete detailed assessments of the teens’ skills and abilities at the beginning and end of the programs. All of these assessments were used to create our data framework.

Finally, we created long-term goals and measurements for the teens’ success after participation in the program, including employment in our target sectors, college attendance, career-readiness and also advancement in the program in returning years. We designed annual surveys to check in with program alumni and assess their long-term progress.

Step Four: Write the Narrative
The final step in the program design phase of the theory of change model involves creating a narrative: A model describing how your initiative will create change to achieve your program goals. This is more than just a pretty story: This is your opportunity to test your logic in plain English. When you create your narrative, this is a document you will share with your stakeholders to make sure that the steps you outline make sense - and that the data measures you identify connect accurately to the program goals. (In the next post, I’ll share some sample narratives to help guide your program design process.)

Step Five: Implement, Iterate, Improve
This last step is where the really good stuff happens. Once you’ve designed a program, it’s time to implement it, collect data, and use data to make adjustments and improvements.

In the Youth Employment in Parks example, program managers used the Theory of Change model to provide both structure in the program design and metrics to measure its effectiveness. We found the data invaluable in enabling us to continually improve the program, by adding training in specific areas or reducing or removing unneeded components. We ultimately reduced the time spent in content study and achieved greater mastery of the content. This was a positive feedback loop, and achieved improved results continuously as participant feedback enabled us to iterate and improve.

Next week we’ll examine the process of building your measurement model.

Further reading:

Monday, April 9

Why Measurement Matters

For designers, developers and trainers, measurement is a weighted word. Employee performance is difficult to measure objectively - as is the success of a training program. Furthermore, for folks whose strength is creative communication, data analytics is a far cry from our comfort zone.
Many instructional designers have also been turned off of analytics by seeing data used out of context or as the only measure of a complex initiative. As a result, measuring learning initiatives often falls into the pattern of measuring production - "we've provided X resources" or "we've distributed Y manuals" - because these are factors L&D can control.
What really matters for an organization, of course, is not how many manuals L&D creates, but how many behaviors or outcomes L&D changes. With all the demands on our time, it’s easy for L&D departments to be consumed by “putting out fires”: Meeting short term needs, creating resources for squeaky wheels and solving immediate performance problems often come before taking time for strategic decision-making.
So the need is greater than ever for learning and development to embrace data, measurement and analytics to enable us to target initiatives to efficiently meet business objectives, make the best possible use of our limited time and to demonstrate the efficacy of our work to organizational leadership.
Tough Economies Put Pressure on the Numbers
Immersing yourself in data-driven decision making will enable you and your team to make strategic decisions that meet broad organizational goals. Ultimately, the ability to harvest, discuss and process this information correlates to improved financial outcomes for large organizations:

A recent study by KnowledgeAdvisors and Bassi Investments illustrated that a group of companies with high learning and development measurement acumen outperformed the Standard & Poor’s 500 Index in terms of share price appreciation by more than 15 percent.
- "Emerging Issues in Measurement", KnowledgeAdvisors

Many L&D departments dread the measurement and ROI discussion. As a cost center (and not a direct revenue generator), L&D is always challenged to justify its connections to organizational success. If L&D professionals use good measurement programs to demonstrate improved productivity, changed behaviors and improved outcomes, they will be in a stronger position in the next budget cycle.
This month, I’ll outline a framework for measuring changed behavior, and share examples of this model’s success in a social change program I worked with in Anchorage, Alaska. I will also showcase leading organizations’ use of learning analytics to meet organizational goals. We’ll continue next week with a discussion of connecting learning design to a data framework from the beginning.
Further Reading on Measurement:

Image credit: Auntie P on Flickr

Monday, April 2

Welcoming April Guest Blogger: Kelly Meeker

Kelly is the Community Manager for OpenSesame, a marketplace to buy and sell e-learning courses that will work in any LMS. Among other things, Kelly writes and manages the OpenSesame blog and is the face of OpenSesame on Twitter. She’s a great model for a community manager; not only is she knowledgeable about the industry, but she has real and authentic interactions with the community.

Kelly will be blogging this month about measurement, analytics, and data collection.

You can connect with Kelly on Twitter @OpenSesame or catch up with her on the OpenSesame blog.