ASSESSMENT IRREGULARITIES: A DISCUSSION GUIDE FOR CURRENT ISSUES IN TEST ADMINISTRATION

Guy Glidden

Wichita Public Schools

INTRODUCTION

A lot has happened in the past eleven years. In 1985, NATD provided a symposium at the NCME Annual Meeting and published the findings in a paper entitled, "Cheating on Standardized Tests." In 1996, NATD provided a symposium at the NCME Annual Meeting on the topic, "Current Issues in Test Administration" as a vehicle to discuss many of the irregularities (including cheating) which can affect large scale assessment programs. Although it might seem that test directors have been treading water for eleven years, the 1996 symposium provided a reminder of the changing scope of school assessment programs.

• Standardized testing was viewed as the major component of school assessment programs in 1985. By 1996, standardized testing has become only one tool among many aspects of school assessment programs which can include portfolios, interdisciplinary assessments with combined individual and group tasks, fixed response assessments which combine multiple choice and multiple mark question formats, and blended assessments which can include combinations of all of the above. In addition to the positive aspects of many new forms of multiple assessments, the risks of irregular assessment events and inconsistent assessment results also increases.
• Cheating, perceived as the main assessment irregularity of concern in 1985, is viewed eleven years later as only one of a number of factors which can adversely affect assessment results.

The focus of this document is to:

• clarify the concept of assessment irregularity.
• provide outlines of alternative ways to address and control assessment irregularities.
• serve as a discussion guide of the factors which influence assessment irregularities.

ASSESSMENT IRREGULARITIES ARE DEFINED AS:

• Events which can cause "assessed" performances which are not related to actual performances.
• Process variations (intentional or not) which create a bias of outcomes.

The traditional view of assessment irregularities (the focus of the 1985 NATD Symposium, "Cheating on Standardized Tests") placed blame on individuals who are consciously looking for a short cut which will influence the results of assessments. Thus, the "murder mystery" theory is a construct for understanding the issue from this perspective.

As in murder mysteries, irregularities results form a combination of:

MOTIVE

OPPORTUNITY

MEANS

Current and past experiences, however, point to a variety of systematic factors other than cheating which can and do influence assessment results. Several examples illustrate this point:

• A readiness assessment (used during the time period when "Lassie" was a television staple) provided an item which presented pictures of different breeds of dogs and asked the children to correctly identify the picture of the Collie. A number of kindergarten teachers, when viewing the looks of concern on the faces of their children, helpfully said, "you know, Lassie dog."
• On the project section of a recent state-wide assessment, no guidelines were provided for the presentation of the document. The level of school support for these projects ranged widely. Some schools provided institutional support including video taping resources, while others chose not to provide support to students.
• A large scale writing assessment provides information to teachers that the "writing experience should be as much like the normal classroom writing experience as possible." This has been interpreted by many teachers as meaning that students should be able to use word processors (with spell-check) if the school has access to this resource.

In these examples there was no evidence of intent to cheat by the participants as systematic forms of "irregularities" outweigh the impact of cheating.

WHAT ENCOURAGED THESE IRREGULARITIES?

• LACK OF AGREEMENT/UNDERSTANDING OF ASSESSMENT GOALS. It is important to support the process if the process is viewed as arbitrary?
• SUB-OPTIMIZATION OF PROCESS. Which is more important; the measured process or the success of schools and/or students within the process?
• OPPORTUNITIES WHEN MEANS/MOTIVATION EXISTS

In the philosophy of continuous quality improvement, identification of root causes is the key to process improvement. From this perspective, what can and should be done to control assessment irregularities?

Control factors for Assessment Irregularities

• DEVELOP UNDERSTANDING/AGREEMENT OF ASSESSMENT GOALS
• PROVIDE SECURITY FOR ASSESSMENT DOCUMENTS
• CLARIFY AND MONITOR CONSISTENT PROCEDURES AND PROTOCOLS
• PROVIDE SANCTIONS FOR IRREGULARITIES

Two "real life" assessment plans are provided for discussion.

High Stakes State Assessment

• State has a history of high stakes, both for students and school systems.
• A program of rewards and sanctions for schools has been developed.
• A Code of Ethics has been developed as a control factor for irregularities.

School District Assessment Plan for Instructional Support

• Although the purpose of district assessment (instructional support) may be "high stakes" for educational improvement, the short-term consequences are "low stakes."
• Strategies (communication of expectations and consequences, rewards and recognition of quality) are consistent with the purpose of the assessment program.
• Process uses a continuous improvement model.

This "murder mystery" guide is provided as one way to compare the issues and possible assessment irregularity control factors for the assessment plans which are provided.

• - - - - - - - - - - - MOTIVE - - - - - - - - - - - - -
• - - - - - - - - - OPPORTUNITY - - - - - - - - - -
• - - - - - - - - - - - - - MEANS - - - - - - - - - - - -