Creating a Measurement Model

In the first two posts in this series, we discussed the value of data and the process for using the theory of change model for learning design. This week, we’ll walk through creating a measurement model for a fictional company, Acme Donuts.

As you work through the learning initiative design process, the key challenge is to balance strategy with tactics: Connecting specific actions that accomplish high level goals. The Theory of Change model, which starts with broad goals and asks you to work backwards to specific actions, forces you to make the connections between each step clear. The narrative step also challenges you to explain the connections: An excellent exercise for testing assumptions and discovering gaps.

I created this sample narrative to outline the many factors relevant to designing a learning program that includes a measurement model from the beginning.

Developing a Program Narrative: Acme Donuts Example

1. What are your initiative’s goals? To develop functional goals, you need stakeholder buy-in, alignment with organizational goals and specific descriptors. Make sure you’re not developing goals in a vacuum.
   
   Acme, Inc. is a manufacturer of donuts. Our goal is to increase profits 15% this fiscal year. To do that, we must meet two key preconditions: 1) Increase the recurring revenue generated from long-term sales agreements; 2) The sales team must develop strong communication, sales and negotiation skills to negotiate and close sales agreements.
   
   
2. Whether conscious or unconscious, everyone makes assumptions. A key step in developing a functional initiative is ensuring that you understand what variables you will hold neutral as you target changes in your organization.
   
   We are making two key assumptions: 1) The product is appropriately priced for its quality. 2) There is sufficient demand in the marketplace to enable us to increase sales.
   
   
3. Work backwards from your goals to the preconditions that must be met. Limit the scope of your discussion to the changes that can be affected through training and development activities. If, for example, Acme’s Donuts just aren’t very good, training sales people more effectively may not do any good at all.
   
   Both sales team members and managers have described needing opportunities both to study and practice the techniques of sales and communication and to work together as a team to identify best practices and improve performance over time.
   
   
4. Outline specific initiatives you will undertake. Ensure that you outline how those initiatives connect with your preconditions and goals.
   
   To reach the targeted competencies of communication, sales and negotiation skills, we will embark on a multi-phase initiative: Participation in online learning courses to expose sales team members to the techniques of diagnosing and creating a sales strategy. Group workshops with their peers to enable them to practice their communication and negotiation skills. Connecting each participant to a mentor who will meet regularly with them to enable them to seek feedback and advice. Creating a social network to enable team members to share successes and resources.
   
   
5. Develop specific metrics based on your goals. Ensure that you have a variety of metrics at each level and step of your initiative. Make sure you can assess specific factors so that you can, in turn, make specific improvements to your learning initiatives.
   
   We will measure the success of our initiative on a monthly basis with these metrics:
   - Sales team’s self-assessment of communication, sales and negotiation skills on a detailed scale
   - Sales manager and VP of Sales assessment of individual and team performance
   - Monthly recurring sales per salesperson and as a group; the rate of change in monthly recurring sales
   - Profit growth
   
   
6. Set expectations. Ensure that you and your colleagues understand and agree on goals and reasonable expectations for results.
   
   We expect to see some change in the metrics in the first two-three months of the initiative, but we believe that we will see significant improvement within six months as sales team members have an opportunity to implement and improve their new skills in cooperation with their peers. We will use data from assessment and from performance to continually create online courses and group workshops targeting areas of weakness. To the extent possible, we will use internal experts to lead workshops both in person and virtually.
   

In closing, this example is just the beginning. Check out the Theory of Change website, which provides an excellent case studies from a variety of social and job training programs.

Further Reading

• “Getting Serious About Measurement: Metrics for Social Learning”
• “21 Questions to Ask Before Designing a Training Program”, The Training Doctor

Image credit: dasroofless on Flickr