Teaching Grammar and Writing - Week 9 - Writing Teachers as Assessors of Writing

Introduction: Hello and welcome to Week 9 of the course Teaching Grammar and Writing for the Master's in English Teaching at ULACIT in term IC 2023. In this class we will continue our focus on writing by considering important factors in the complex topic of writing assessment. We'll also look at your Interactive Grammar Review Study Guide activity and consider the orgin, history, and ongoing controversy regarding the PPP approach to lesson design. 

Today's Goals:

• Demonstrate the features of your original Interactive Grammar Review Study Guide activity.
• Explore the complexities of L2 writing assessment by analyzing the construct, task procedures, and scoring system of a standardized writing test.
• Consider the history and controversies surrounding the PPP approach to lesson desgin.

Guiding Questions:

• How can I help students develop explicit knowledge of grammar rules?
• How can I address reliability and validity concerns in the design of a writing assessment?
• How did PPP come to be the dominant planning framework in ELT and what are the conflicting views about it?

Task 1: Activity Type Demo - Interactive Grammar Review Study guide

In this course you will be asked to create sample grammar activities in order to compile a portfolio of grammar activitiy types that describes their basic features, strengths, and challenges. 

• Characteristics: What are the features of an Interactive Review Study Guide activity?
• Example: What activity did you create?
• Strengths: In what ways is the concept of an Interactive Grammar Review Study Guide beneficial in grammar and writing instruction?
• Challenges: What potential limitations or challenges are associated with the inclusion of Interactive Review Study Guides in the grammar class?

• Additional Considerations: CLICK HERE

Task 2: Reading Response - Writing Teachers as Assessors of Writing

Let's take a moment to discuss these questions related to your assigned reading for this week.

• Importance of Assessment: What is the point of assessing learners' writing ability? What are the ethical considerations and real world consequences of writing assessment specifically and other forms of educational assessment more generally? I'm paid to teach. Why do I need to know about core concepts in educational assessment?

• Quote: "Teachers must know enough about assessment practices to be able to look at the assessments being brought into their programs, or being taken externally by their students, and evaluate them. From this perspective, assessment is every teacher’s job (Hamp-Lyons, p. 162).”

• Nature of Assessment: Assessment designers must make choices about the nature of their assessment. One of those choices is whether or not it is feasible to assess an ability directly or if an indirect approach will be sufficient. 

• What your understanding of the meaning, strengths, and challenges, of the following concepts? 

• Direct Assessment
• Indirect Assessment

• Additionally, why did the author say that indirect assessment of writing grew in popularity in the US in the 50's and 60's before a preference for direct assessment returned in recent decades?

• Reliability: What is reliability in educational assessment? Why is it impossible for writing tests to be 100% reliable? What is the highest level of reliability that you can reasonably expect from a well designed writing test? What factors influence reliability when considering performance of both writers and exam raters? How can reliability in direct writing assessment be increased?

• Validity: Validity in assessment is simply the degree to which the test instrument assessess the knowledge and abilities it is supposed to assess. There are several aspects of validity to consider.

• Face: The degree to which the assessment appears to be valid to the test taker or other interested outsider. Wait, why are you asking me multiple choice questions on a test meant to measure by ability to swim?

• Content: The degree to which the content of the assessment reflects the aspects of the knowledge or skills being assessed. For example, a test of writing ability for academic purposes should include content specific to academic writing scenarios in real life. Wait, I'm taking this test as a requirement for a job as an English teacher but it has all these business related scenarios. I'm not trying to work for an international company!

• Criterion: The degree to which the results of the assessment correlate with the person's actual performance ability in real life. For example, a particular score in a test of writing for academic purposes should correlate with specific performance indicators a person is able to carry out in a real life academic context. Wait, this trainee scored 95% on the phone skills apptitude test but he can't handle a basic phone call with real customers! 

• Construct: A construct is the operationalized definition of the knowledge or skill being assessed. A construct cannot be directly observed. Instead, a construct valid assessment requires test takers to perform certain measurable behaviors which have been previously determined to represent aspects of the construct. For example, "good writing ability for an academic context" is a general construct that is not observable. Test designers must develop a list of claims about the nature of "good academic writing" that can be observed by performance of specific behaviors. 

• Validity Generalization: Scores generated by an assessment should make sense to people that look at them. This avoids misinterpretation of the results and incorrect usage of the tests. An assessment has consequences beyond the test itself. Teachers need to be conscious of the concept of assessment "impact". 

• Best Practices: With all of that in mind, here are three best practices when designing an educational assessment for L2 writing or any other ability.

• Consequential Validity: Don't use tests in biased ways that treat individuals or groups unfairly. Assessment should as a minimum do no harm to instruction or to learning.

• Construct Representation: Behaviors test takers are asked to demonstrate should be as similar as possible to the actual abilities the test attempts to assess. 

• Construct Relevance: Tests should only assess skills that are part of the construct. There should be no additional difficulties that are not part of the skill being tested.

• Practical Issues in Validity and Reliability: A valid and reliable assessment of L2 writing abilty also needs to consider five important factors. I have included a list of questions to help you reflect on the complexities of each.

• The Writer: Who are the test takers? What are there cultural, linguistic, and educational backgrounds? What is the purpose of the assessment for them? How familiar are they with what the test requires? How can teachers/testers appropriately inform test takers of the requirements of the test so that they are adequately prepared? 

• The Task: What do test takers have to do during the exam? To what degree does the task elicit observable behaviors that are indicators of the underlying exam construct? How difficult are the tasks? How can test designers ensure that multiple prompts have the same level of difficulty? How "authentic" are the tasks?

• The Text: What is the nature of the text that test takers need to produce? How should the text be organized? What genre and register considerations should be made? How will errors in language, genre, or register be evaluated?

• The Scoring Procedure: What scoring system will be used to evaluate the test? What are the benefits and drawbacks of a holistic scoring scheme? Is it more appropariate to evaluate specific traits rather than overall impression? If so, which traits are important to evaluate? Should all traits be evaluated equally or should some given more weight? What do different scores mean? How clear, specific, and easy to follow are the scoring procedures?

• The Reader(s): Who is going to score the test? How does their background potentially bias the way they interpret test takers' writing samples? What special training do they have? How familiar are they with the scoring procedures? How consistant are they in applying the scoring procedures accurately? What oversight mechanisms can be put in place to ensure higher levels of rater reliability?  

• Analyzing a Real Example: Let's make all of this theory more concrete by analyzing the design of a popular international test of L2 writing ability, the TOEIC Writing test.

• CLICK HERE to access the document. 

• Alternative Assessment: The challenge of reliability in high pressure, standardized testing of writing ability along with growing dominance of the process approach to writing instruction has led to an increase in popularity of portfolios as an alternative to traditional writing assessment.

• Portfolios: A portfolio is a collection of the writer's own workover a period of time. They let students show what they can do by showing what they have done. This includes their strengths as well as improvement opportunities. Portfolio assessment consists of three procedures:

• Collection of writing samples over time.
• Selection of particular writing samples to submit.
• Reflection on what the selected writings demonstrate about the student's learning.

• Final Quotes: These quotes from the conclusion of the chapter help summarize the purpose of the complexities and functions of assessment in L2 writing instruction. What stood out most to you about this chapter?

• “... no assessment is value free. Nor is any assessment free of effects that reach beyond the narrow confines of the assessment event itself.”

• “When a society values test results more than abilities and potentials, and when tests/assessments become the only or primary route to higher education, careers, status, and wealth, inevitably there is a great impact on families’ lives and there is danger that information resulting from these assessments will be liable to educational, administrative, and political abuses.”

• A firm understanding of how assessment works, what it can do, and what it cannot do, is an essential tool for today's teachers.”

Task 3: Power of PPP: Questioning the Dominant Paradigm

Last week we looked at the three so called "interface positions" regarding the relationship between explicit and implicit knowledge. Today we will focus the history and ongoing contravesy surrounding the dominant instructional paradigm in language teaching today, Presentation, Practice, Production (PPP). 

References:

Anderson, J. (2017). A potted history of PPP with the help of ELT Journal. ELT Journal, 71(2), 218-227, https://doi.org/10.1093/elt/ccw055

Hamp-Lyons, L. (2003). Writing teachers as assessors of writing. In B. Kroll (Ed.) Exploring the Dynamics of Second Language Writing. Cambridge University Press.