Design and Evaluation of Training Programs - Week 9

Introduction: Hello and welcome to week eight of the course Design and Evaluation of Training Programs and Workshops. Today we will do several activities to explore the topics of teacher research as professional development, assessment of training, and strategies for organizing the "shape" of your training course. 

Warm Up: Origin Stories

Last week we discussed how fictional depictions of teachers and teaching can be analyzed as a professional development strategy in order to gain insights about our practice. This week we will do a variation on that theme by creating a work of fiction based on a teaching reality. Click your group link and complete the information required. 

• Group 1: CLICK HERE
• Group 2: CLICK HERE

Now share your origin stories. CLICK HERE to see a digital comic book that I created to share with my team of teachers when we did this activity last December. How else might might teachers create fictional content to explore deeper issues regarding the teaching-learning process?

Task 1: Sharing your PD Journal

Let's take a moment to share one of the entries you made in your PD Journal in Week 8 for the section called "Develop Research Skills". As you share your highlight, let's think about how this tip connects to how we as individual teachers can develop in our practice and how we as trainers can use these techniques to support the growth of the teachers we work with. We can use the RACE framework to facilitate our discussion. 

Click to see full sized image.

Task 2: Teachers as Researchers

In your reading this week from Richards (2017), the author suggests developing resarch skills for professional development.

• What do you think of when you hear the term "research"?
• What forms can research take?
• We tend to associate research with scientists and university students, what are some ways that teaachers become researchers?
• What benefits can teacher research have for individual teachers and their institution?

Doing Teacher Research

Donald Freeman's (1998) book "Doing Teacher Research: From Inquiry to Understanding" offers a practical framework for classroom based research projects for teachers. Click the link below to explore the Teacher Research Cycle.

• Group Worksheet: CLICK HERE

Evaluating Textbooks

Teachers can use a variety of didactic materials in their classes, but nowadays nearly all language courses in the world are based on or supported by the use of commercial textbooks. McGrath (2016) in his book "Materials Evaluation and Design for Language Teaching" stresses the importance of teachers developing a critical or questioning mindset regarding the materials they use.

• Can you think of a time as a teacher, learner, or adminsitrator when you looked at the materials you were using critically?
• In your experience, what factors determine if a textbook is good or appropriate for a given course?
• How might you (and your teachers) go about analyzing and evaluating a textbook?
• What purposes might a project like this serve?

McGrath presents a framework for analyzing, evaluating, and selecting coursebooks which begins by clearly analyzing and defining student learning needs and contextual factors in which the materials will be used. He futher suggests the development of checklists and other data collection instruments to be used to analyze and evaluate potential materials in a systematic way.

Click to see full sized image.

A professional development workshop for teachers could involve them suggesting checklist criteria to create a data collection instrument and then use the instrument to review a textbook and report on their findings. You could also have teachers use other frameworks for textbook analysis such as the ones below.

Click to see full sized image.

Task 3: Reading Response

Last week we ran out of time and we were not able to look in detail at some important theory related to assessment in course design. Since how and when you assess trainees is very much related to how you choose to organize the course, let's review this content before looking at the topic of this week's reading. 

Graves says that assessment serves three overlapping roles in course design:

• Assessing needs: What (and how) do students need to learn with respect to ____?
• Assessing students’ learning: What have students learned with respect to _____?
• Evaluating the course: How effective is/was the course in helping them learn _____?

What experience have you had in developing assessment strategies for these three areas in the courses that you teach? Which of the three areas do you find harder to address and why?

Think of a course you are teaching or have recently taught and list the ways (both formally and informally) in which you have assessed students’ learning. Do you think these ways were effective?

Making Connections

• What is Assessed: What specific Knowledge, Attitudes/Awarenesses, Skills/Abilities might assess in my training course?
• Who Assesses the Learning: Will it be beneficial for me to be the one who assesses all of the learning? Could I also consider allowing the trainees to assess their own learning and the learning of others? Would peer and self-assessment work in my context?
• How is Learning Assessed: What strategies might I incorporate for learning assessment? How can the assessments be organized? Can I find or create a framework that might help me integrate assessment into the design of the course (ex. Pre-During-Post)?
• Evaluating the Course: How can I plan to evaluate the effectiveness of my course? What input do I need from the trainees directly? How can my assessment of trainees’ learning help me evaluate the effectiveness of the course? How can I measure the impact of the course on what the trainees actually do in the classroom?

Practical Ideas

Strategy 1: Use Pre-Post Course Reflections

One simple but powerful learner-centered formative assessment strategy is to design a pre-course and post-course reflection task. In the pre-course reflection have trainees answer questions similar to the following.

• What do you hope to get out of this course?
• What do you already know about the following topics...?
• What specific questions do you have regarding...?

Of course, you will need to personalize the prompts to fit your training context. You may also want to include a short diagnostic quiz in this task. The purpose of the pre-course reflection is to have trainees articulate their goals for the course, specific questions and doubts regarding the content areas, and (in the case of a diagnostic) find how how much they already know.

At the end of the course, have trainees revisit their pre-course reflections and answer prompts similar to the following:

• Look at the goals you had at the beginning of this course. To what degree do you think you have achieved them? Provide specific reasons to support your opinions.
• How did your knowledge of ... change or deepen as a result of this training? Provide specific examples. 
• What were your questions and doubts at the beginning of the course about...? Are you able to answer any of them now? Do you have any new questions or doubts?

You can also have students take the same diagnostic quiz again and reflect on their responses compared to the first one they took.

Strategy 2: Create Digital Checklists for "Complies / Does Not Comply Types" Assessments

One practical way of assessing trainees is asking yourself the question, "Did they do it or not?" This is especially useful when training teachers on procedural tasks where by completing them, they show that they are capable of carrying out the action. These checklists can also be used to establish a global perspective regarding the level of trainee participation. 

• Sample Checklist: CLICK HERE

You might consider adding a third category to your checkboxes: Fully Complies, Partially Complies, Does not Comply. 

Strategy 3: Establish Standard Criteria for Participant Feedback Surveys

By establishing a core set of evaluation criteria for training events, you can measure trainee satisfaction over time and set Key Performance Indicators (KPIs). 

 The Kirkpatrick Model - Donald Kirkpatrick developed this popular four level model for evaluating the effectiveness of training on job performance. It starts with trainees' impressions of the training and evidence of learning and moves on to discover if the training led to any measurable changes in the trainees' on the job behavior and finally, it measures the results gained by the training from an institutional perspective. 

Click to see full size image.

For your training course proposal assignment, you are only required to develop an assessment plan for levels 1 and 2 in the Kirkpatrick model. However, it is important to consider strategies to address levels 3 and 4 in order to determine the true effectiveness of your training efforts. 

• What might be some follow up and support strategies to see to what degree teacher's apply what they learned in the training in their classes? 
• Once you see that teachers are actually applying the training on the job, how do you know if their new behaviors are actually making an impact at an instutional level? 
• What indicators might help you answer that question and how could you gather data regarding those indicators?

Organizing the Course

Last week you read Chapter 7 "Organizing the Course" in Graves (2000) Designing Language Courses and completed a study guide as well as an outline of your syllabus and unit structure.

• Study Guide Answer Key: CLICK HERE

References:

Freeman, D. (1998). Doing Teacher Research: From Inquiry to Understanding. Heinle & Heinle Publishers.

Graves, K. (2000). Designing Language Courses. National Geographic Learning.

Kirkpatrick, D. (2020). The Kirkpatrick Model. Kirkpatrick Partners. https://www.kirkpatrickpartners.com/Our-Philosophy/The-Kirkpatrick-Model

McGrath, I. (2016). Materials Evaluation and Design for Language Teaching (2nd ed.). Edinburgh University Press.

Richards, J. (2017). Jack C Richard's 50 Tips for Teacher Development. Cambridge University Press.