Design and Evaluation of Training Programs - Week 8

Introduction: Hello and welcome to week eight of the course Design and Evaluation of Training Programs and Workshops. Today we will do several activities to explore the topics of reading groups and conferences for professional development, assessment strategies for training course development, and the Kirkpatrick Model for training evaluation. We will also take time to share your Moxie outlines and give peer feedback regarding potential avenues to explore as you begin writing your paper. 

Warm Up: Origin Stories

Last week we discussed how fictional depictions of teachers and teaching can be analyzed as a professional development strategy in order to gain insights about our practice. This week we will do a variation on that theme by creating a work of fiction based on a teaching reality. Click your group link and complete the information required. 

• Group 1: CLICK HERE
• Group 2: CLICK HERE

Now share your origin stories. CLICK HERE to see a digital comic book that I created to share with my team of teachers when we did this activity last December. How else might might teachers create fictional content to explore deeper issues regarding the teaching-learning process?

Task 1: Sharing your PD Journal

Let's take a moment to share one of the entries you made in your PD Journal in Week 7. As you share your highlight, let's think about how this tip connects to how we as individual teachers can develop in our practice and how we as trainers can use these techniques to support the growth of the teachers we work with. We can use the RACE framework to facilitate our discussion. 

Click to see full sized image.

Task 2: Teachers' Associations and Conferences

In your reading this week from Richards (2017), the author suggests joining a teachers' association and attending a professional conference as tips for professional development.

• What teachers' associations are you familiar with?
• What do they do?
• What are some benefits of joining them?

Association Membership and SIGs

• IATEFL Membership Benefits and Costs - CLICK HERE
• IATEFL Special Interest Groups (SIGs) - CLICK HERE

• TESOL Membership Benefits and Costs - CLICK HERE
• TESOL Interest Sections (ISs) - CLICK HERE

• ACTFL Membership Benefits and Costs - CLICK HERE
• ACTFL Special Interest Groups (SIGs) - CLICK HERE

Conferences

• Have you ever attendend a professional conference?
• If so, what was your experience?
• Have you attended a virtual conference or PD event? 
• If so, what was that like? 
• How did it compare with a face to face event?

CLICK HERE to join the CCCN Community of Practice for access to the recordings of National Conference for Teachers of English Costa Rica (NCTE) and the PD Talks: Professional Development Sessions.

Task 3: Ideas for Reading Groups

Richards (2017) discusses the idea of creating a reading group as a professional development initiative. Discuss the questions below.

• In your opinion, how important is reading for language teachers to read from professional literature?
• How often do you read articles or books about teaching and language learning outside the context of a university assignment?
• Do you think you and your teacher colleagues would benefit from regularly reading and processing a text from the professional literature of our field? Why or why not?
• What barriers can you identify to establishing a reading group in your context?

 Practical Ideas

Tip 1: Preparation is Key

Give teachers a task to do! Simply asking teachers to read and discuss is unlikely to lead to an effective exploration of the assigned text. Help teachers process their thoughts about the assigned reading before coming to the discussion session with a preparation task. Faciliate the discussion session with a scaffolded series of tasks or questions.

• Sample Reading Group Preparation Task: CLICK HERE
• Sample Reading Group Faciliatation Sequence: CLICK HERE

Tip 2: Allow for Choice

An alternative to choosing a single article or chapter for all teachers to read, you might provide several articles about a specific topic (teaching reading, classroom management, assessment strategies, etc.) and let teachers choose the one they want to read and process. In the discussion session you can have them explore similar themes that came up in the different articles as well as differing points of view and potentially contradictory ideas.

Tip 3: Make it Last

It takes time and effort to carry out a successful reading group activity and you want that investment to make a difference in the teachers' thinking and practice. For one of the tasks in the discussion session, have teachers collaborate to create a tangible product that serves as a record of their thinking and analysis that can be archived for future reference and/or shared with other faculty members. This could be a blog post, a personal journal, a poster to be put up in the teacher's room, a short write up to be shared as part of a teacher newsletter, or a short video or audio recording are just a couple of ideas that might serve this purpose. 

Tip 4: Mix it Up

We read about reading groups, but what about "Watching Groups"? You could also select a webinar recording that you would like your team of teachers to watch and process. As with the reading groups, it is important to give teachers a task in order to facilitate a structured exploration of the content. You could create your own specific reflection question based on the content of the chosen webinar or use a "generic" processing guide that will work for any topic.

Mark's Generic Processing Guide for Webinars

• What were at least three key ideas of this session? For each one summarize the idea and what it means to you.
• What is something that you could take from this session and apply in your teaching context with no modifications needed?
• What is something you could apply in your teaching context with some modifications? How would you modify it and why?
• Is there anything from the session that would not work in your context or that you would not like to do because it does not fit your beliefs or personality?
• What additional insights, ideas, questions, or points of curiosity were sparked in by this presentation?

 

Task 4: Moxie Peer Input Session

Your final Moxie paper is due on March 23rd. Take advantage of this time to share your ideas with your peers and professor and get some feedback and suggestions of potential aspects to address and explore.

• Umbrella Topic: Teacher Training for Human Augmentation and Disruption
• Guiding Question: What are the characteristics of teacher training that enhances human augmentation and mental disruption?

• Core Idea 1: What is your key idea?
• Significance: Why is this idea interesting to you?
• Branching Ideas: What are some components, questions, and branching ideas?

• Core Idea 2: What is your key idea?
• Significance: Why is this idea interesting to you?
• Branching Ideas: What are some components, questions, and branching ideas?

• Next Steps: What research do you still need to do?

Task 5: Reading Response

Last week you read Chapter 10 "Designing an Assessment Plan" in Graves (2000) Designing Language Courses. 

Graves says that assessment serves three overlapping roles in course design:

• Assessing needs: What (and how) do students need to learn with respect to ____?
• Assessing students’ learning: What have students learned with respect to _____?
• Evaluating the course: How effective is/was the course in helping them learn _____?

What experience have you had in developing assessment strategies for these three areas in the courses that you teach? Which of the three areas do you find harder to address and why?

Think of a course you are teaching or have recently taught and list the ways (both formally and informally) in which you have assessed students’ learning. Do you think these ways were effective?

Making Connections

• What is Assessed: What specific Knowledge, Attitudes/Awarenesses, Skills/Abilities might assess in my training course?
• Who Assesses the Learning: Will it be beneficial for me to be the one who assesses all of the learning? Could I also consider allowing the trainees to assess their own learning and the learning of others? Would peer and self-assessment work in my context?
• How is Learning Assessed: What strategies might I incorporate for learning assessment? How can the assessments be organized? Can I find or create a framework that might help me integrate assessment into the design of the course (ex. Pre-During-Post)?
• Evaluating the Course: How can I plan to evaluate the effectiveness of my course? What input do I need from the trainees directly? How can my assessment of trainees’ learning help me evaluate the effectiveness of the course? How can I measure the impact of the course on what the trainees actually do in the classroom?

 

Practical Ideas

Strategy 1: Use Pre-Post Course Reflections

One simple but powerful learner-centered formative assessment strategy is to design a pre-course and post-course reflection task. In the pre-course reflection have trainees answer questions similar to the following.

• What do you hope to get out of this course?
• What do you already know about the following topics...?
• What specific questions do you have regarding...?

Of course, you will need to personalize the prompts to fit your training context. You may also want to include a short diagnostic quiz in this task. The purpose of the pre-course reflection is to have trainees articulate their goals for the course, specific questions and doubts regarding the content areas, and (in the case of a diagnostic) find how how much they already know.

At the end of the course, have trainees revisit their pre-course reflections and answer prompts similar to the following:

• Look at the goals you had at the beginning of this course. To what degree do you think you have achieved them? Provide specific reasons to support your opinions.
• How did your knowledge of ... change or deepen as a result of this training? Provide specific examples. 
• What were your questions and doubts at the beginning of the course about...? Are you able to answer any of them now? Do you have any new questions or doubts?

You can also have students take the same diagnostic quiz again and reflect on their responses compared to the first one they took.

Strategy 2: Create Digital Checklists for "Complies / Does Not Comply Types" Assessments

One practical way of assessing trainees is asking yourself the question, "Did they do it or not?" This is especially useful when training teachers on procedural tasks where by completing them, they show that they are capable of carrying out the action. These checklists can also be used to establish a global perspective regarding the level of trainee participation. 

• Sample Checklist: CLICK HERE

You might consider adding a third category to your checkboxes: Fully Complies, Partially Complies, Does not Comply. 

Strategy 3: Establish Standard Criteria for Participant Feedback Surveys

By establishing a core set of evaluation criteria for training events, you can measure trainee satisfaction over time and set Key Performance Indicators (KPIs). 

Task 6: The Kirkpatrick Model 

Donald Kirkpatrick developed this popular four level model for evaluating the effectiveness of training on job performance. It starts with trainees' impressions of the training and evidence of learning and moves on to discover if the training led to any measurable changes in the trainees' on the job behavior and finally, it measures the results gained by the training from an institutional perspective. 

Click to see full size image.

For your training course proposal assignment, you are only required to develop an assessment plan for levels 1 and 2 in the Kirkpatrick model. However, it is important to consider strategies to address levels 3 and 4 in order to determine the true effectiveness of your training efforts. 

• What might be some follow up and support strategies to see to what degree teacher's apply what they learned in the training in their classes? 
• Once you see that teachers are actually applying the training on the job, how do you know if their new behaviors are actually making an impact at an instutional level? 
• What indicators might help you answer that question and how could you gather data regarding those indicators?

References:

Graves, K. (2000). Designing Language Courses. National Geographic Learning.

Kirkpatrick, D. (2020). The Kirkpatrick Model. Kirkpatrick Partners. https://www.kirkpatrickpartners.com/Our-Philosophy/The-Kirkpatrick-Model

Richards, J. (2017). Jack C Richard's 50 Tips for Teacher Development. Cambridge University Press.