 
|
|||||||||||||||||||||
              
|
Computer to grade school test essays
Sunday, December 16, 2001 By Eleanor Chute, Post-Gazette Education Writer
This month, pupils around Pennsylvania are taking a test that will be graded by an expert who never needs a cup of coffee, can work for 12 hours straight without a bathroom break and can grade a written essay in a matter of seconds.
PG writers take IntelliMetric software for a test drive
The essay tests, being taken by more than 35,000 pupils in grades five, eight and 11, are being graded by a computer that uses artificial intelligence.
Developers of these automated grading systems say they are at least as accurate as humans, not to mention faster and cheaper. The software programs also eliminate any bias human graders might have, they say.
But critics say computer grading puts more emphasis on style and form than substance, and that grading writing is one area where human subjectivity is valuable, if not crucial.
The online exams being given through Friday are just part of a pilot project using IntelliMetric, a system developed by Vantage Learning of Yardley, Bucks County. Schools can still sign up to take the tests.
State officials haven't decided whether to switch officially from human grading to computerized grading for the official Pennsylvania System of School Assessment writing tests, which are given in grades six, nine and 11.
Pupils in Chartiers Valley in Allegheny County and Aliquippa in Beaver County were among those taking the machine-graded pilot tests.
Patrick Myers, Chartiers Valley school psychologist, said taking the test gave the district useful practice. "We feel that eventually they are going to go to the computer."
If the state does switch to computer grading of the mandated writing tests, the cost of grading essays will be cut in half, said Beth Gaydos, spokeswoman for the state Department of Education.
The state now pays Data Recognition Corp. of Minnesota about $2 for each essay graded by humans. With about 135,000 pupils per grade level answering two essay questions each, that comes to about $1.6 million, meaning that the state could save about $800,000 using the computerized method.
The computers can't grade every writing assignment, only the ones they've been "trained" to analyze.
The computers are given the assigned topic, called a writing "prompt," and then are fed a few hundred essays that have been previously graded by humans.
Different approaches can stump the computer, resulting in the human grader getting the essay anyway.
In testing computerized grading, Pennsylvania is joining a fast-growing national trend.
Artificial intelligence systems already are being used for some graduate school admissions tests, college freshman writing placement tests and online classroom writing exercises.
Besides Vantage, at least two other companies are offering automated essay grading. ETS Technologies Inc., which uses a program called "e-rater", is a for-profit subsidiary of Educational Testing Service in Princeton, N.J.; which oversees the SAT. Knowledge Analysis Technologies of Boulder, Colo., sells a program called "The Intelligent Essay Assessor."
Each system places varying emphases on grammar, content and other elements of writing.
Pennsylvania is spending $810,000 for Vantage to use IntelliMetric to grade the writing tests being taken this month. While the test results are available in seconds, they won't be released until spring because each test will also be graded by humans for comparison purposes.
This is the third IntelliMetric pilot in Pennsylvania, including one that involved 23,000 11th-grade writing tests in the 1999-2000 school year.
While the idea of a computerized grader might disturb some people, using artificial intelligence really "is not like science fiction," said Robert Frederking, a senior systems specialist at Carnegie Mellon University.
He said using computers to grade essay exams "seems like a reasonable thing," but probably would work best with routine assignments, because "with automated things, you don't want to make a life-and-death decision based on the output of this."
It's not quite life or death, but beginning with the Class of 2003, the state will issue diploma seals based on whether a pupil is "proficient" or "advanced" on state tests in writing, reading and math.
Frederking said artificial intelligence systems do make mistakes, and when they do, "they're usually wildly weird, wrong things."
Because of that, computer grading programs usually flag unusually written essays for humans to check.
For the high-stakes Graduate Management Admissions Test, which is used by students applying to graduate business schools, ETS Technologies uses one human and one machine, with another human to resolve differences.
Larry Bosley, vice president for territory for Vantage Learning, said IntelliMetric was reliable enough that a human isn't needed to score most essays. He said it already had been used to grade millions of tests.
Monty Neill, executive director of the National Center for Fair and Open Testing in Cambridge, Mass., known as FairTest, said it was "probably true" that machines could grade as accurately as humans, "given what they ask them to do."
He said the real problem was that "what is asked on the writing samples is pretty trivial."
Dennis Baron, a professor of English and linguistics at the University of Illinois, is skeptical about the machines. Baron, who is working on a book on the interaction between literacy and technology, said that computerized grading programs lacked flexibility.
"What the machine can do is compare what it sees with a set of instructions it's been given," he said. "It can't feel what is being read the way human readers can and make adjustments.
"The argument in favor of the machine is it makes them objective. The argument in favor of the humans is that no reading is objective. All reading is done in context."
All three scoring companies have Web sites that have more details on their technology. The addresses are www.intellimetric.com, www.etstechnologies.com and www.knowledge-technologies.com.
|
||||||||||||||||||||
