Learning & Development: What to Measure, and Why?
“What should I measure, and why?” are the most frequent questions I’m asked by training Managers, VPs and Directors. What I hear them asking is – what value is there in collecting data beyond the traditional “smile sheets” completed by students after each training session? Were they to invest in the analytical services I offer, how would their training program benefit? Would their sales people be more productive? They are challenging me to prove it.
First – the Why?
Organizations which evaluate the effectiveness of their learning & development program tend to be organizations which have the more effective learning & development programs.
This isn’t to suggest that measurement, assessment and evaluation automatically improve training programs. Rather, it is a statement about “Learning Organizations.” According to Peter Senge (1990: 3), who coined the phrase, learning organizations are: “…organizations where people continually expand their capacity to create the results they truly desire, where new and expansive patterns of thinking are nurtured, where collective aspiration is set free, and where people are continually learning to see the whole together.” Here is the value of measurement, assessment and evaluation: Becoming an organization where people continually expand their capacity to create the results they truly desire. Measurement and analysis provide the feedback necessary for intentional systematic improvement.
As a career analyst with a graduate degree in social science research and 25 years of experience as a business analyst and consultant in finance and organizational development, my natural tendency is to want to measure everything, analyze all the data I can get my hands on. Numbers tell stories, and – based on the stories the numbers tell – I provide my clients with insights into training and performance improvement beyond hunches and gut-feel. But with Learning & Development budgets strained as they are, managers want to know why they should spend more if they don’t have to. I’m not interested in analysis for the sake of analysis, but I am intensely interested in analysis for the purpose of performance improvement, improving the quality and impact of training, to justify the cost of training, and to reduce the cost of training. Analysis of data yields information and insights into performance improvement.
What to measure? Not everything requires or warrants all evaluations. That is, only some – high value – courses/programs would benefit from a full “Level 5″ ROI analysis.
Targeted, focused ROI studies, especially for sales training programs
Training Spending and Staffing Statistics
- Training expenditures per learner
- Training staff to learner ratio
- Percent of spending on L&D staff payroll
- Percent of spending on learning technologies
- Spending allocation by training program area
- Spending allocation by employee type
- Training volume and delivery statistics
- Annual student hours consumed per learner
- Cost per student hour consumed
- Percent of student hours by delivery method
- Number of employees trained, by position title
- Number of courses offered, by content area
- Number of sessions offered, by course, by content area, by delivery method
- Number of registrations compared to number of attendances
- Content development costs
- Time to employee readiness or competence
Analysis of TEACHING
- Quality and effectiveness of:
- Materials
- Instructors/Facilitators
- Instructional design
Analysis of LEARNING RESULTS
- Pre-learning diagnostic assessments
- Post-learning knowledge, concepts
- Post-learning application/behavior change
Analysis of DELIVERY
- Course session location, time
- Method: Classroom vs. Online vs. Self-study vs. Combination
Analysis of TESTING
- Are your exams valid & reliable?
- Are your exams measuring what you think you’re measuring
- Is your exam fair?
- Will the results of your tests predict future success?
- Do the questions properly test their associated learning objectives?
- Will the assessment correctly indicate that a learner has mastered the material?
- Do your questions cover knowledge and comprehension and application?
What to measure, and why? Measurement, analysis and evaluation in and of themselves are not the value – It is how the insights derived from analysis are used for learning planning and improvement.
