LONDON -- Depending on whom you ask, a proposed new international testing system will either be the next big thing in higher education or a pointless, expensive rankings exercise that will be used to criticize faculty at hard-pressed colleges and universities.
At a meeting in its headquarters in Paris last month, the Organization for Economic Cooperation and Development released a study on whether it would be possible to test what students around the world actually learn in colleges and universities. In November, the organization will decide whether to press ahead with the new system, Assessment of Higher Education Learning Outcomes, or Ahelo.
For Andreas Schleicher, the O.E.C.D.'s chief education adviser, the new system is the obvious follow-up to the Program for International Student Assessment, or PISA, an influential exam that tests 15-year-olds around the world in mathematics, science and reading every three years.
The most recent results, from the 2009 test, pleased Shanghai, South Korea and Finland, which were at the top of the tables, and caused hand-wringing in countries like the United States, which ranked considerably further down.
In part, Dr. Schleicher sees Ahelo as a response to the rising influence of university rankings, which are widely reported in the media and which tend to emphasize research over teaching. "We need some language to talk about teaching quality and learning outcomes that isn't tied to research," he said. "If we don't have a way of measuring teaching then we are going to have to rely on reputation -- which only tells you about the past."
Patti M. Peterson, an official at the American Council on Education, which represents college and university presidents, said "trying to take very different systems of higher education and measure across them" is a doomed effort. Dr. Peterson, who oversees the education council's international work, questioned whether Ahelo's aim -- "to support improvement in learning outcomes" -- would be achieved. She worried that the test could become "a Trojan horse for rankings."
"If it's really about improvement, why not let the institutions be anonymous?" Dr. Peterson said.
Her doubts were echoed by John Aubrey Douglass, a professor at the University of California, Berkeley, who is a prominent critic of the Collegiate Learning Assessment, a U.S. test that served as a model for Ahelo.
"If the purpose is for institutions to use the data from this test for self-improvement, you're not going to get there," he said by telephone from Berkeley. "But once it's in the market these things are hard to unseat."
He compared Ahelo to the SAT college entrance exam in the United States. "The SAT has long been shown to correlate poorly with academic success," he said. "But it's impossible to get rid of because so many colleges use it and there is no incentive to develop a better replacement."
"Ahelo has, surprisingly, become highly controversial in recent months," said Philip Altbach, director of the Center for International Higher Education at Boston College. "Many continue to wonder whether it is possible to obtain reliable data from countries with quite different approaches to the curriculum, different arrangements for access to higher education, and other variables."
According to Dr. Schleicher, initially, "we had great doubts that it would be possible to develop a test that works across languages and cultures." But he added, "Now we believe it is possible."
There has also been considerable criticism of Ahelo's cost. So far, the O.E.C.D. has spent about $13 million on the project, even though there is no confirmation it will go ahead.
"Why is it necessary to do a very, very, very expensive multilingual, multinational test?" Dr. Peterson of the American Council on Education said.
Dr. Schleicher said Ahelo "is not going to be a cheap test."
"If you want to measure creativity or collaboration, that's very hard to do in a multiple-choice test," he said. "But we're pretty confident that if countries are willing to spend the money we can give them something."
He used the standardized test for 15-year-olds as an example.
"If countries in the O.E.C.D. area would improve the performance of their education systems by just 25 PISA points -- which is what Poland achieved over the last six years -- the long-term economic gain accruing to students over their lifetime would be about $115 trillion," he said.
The Ahelo proposal covers three areas: economics, engineering and generic skills. The first two parts were chosen for being more amenable to cross-cultural comparison than other fields, while the generic portion is meant to test the kind of skills thought to be attractive to employers.
One sample generic skills exercise asks students to imagine they work for the mayor's office in a town where a three-eyed catfish has been caught in a lake, causing panic among residents and calls for a nearby factory to be closed. Pulling together media reports, environmental data and the testimony of scientists and other interested parties, the students have to prepare a briefing assessing the likelihood that the mutation was caused by chemical contamination, parasites or inbreeding.
Michael J. Feuer, dean of the graduate school of education at George Washington University, remains skeptical about the relevance of such tasks.
"There's a fantasy about educational testing -- that we can come up with a dipstick and know exactly what the oil level is," he said by telephone from Washington. "In relying on test scores to make judgments about institutions, you end up undermining the morale of people who are being judged with only part of their story included."
Many of Ahelo's critics are worried that a test intended to allow universities to compare themselves with similar institutions in other countries would be seized upon by lawmakers and politicians.
"There's a fear factor about how ministers use data and piggyback on the latest educational trends," Dr. Douglass of the University of California said. "One result is a movement toward uniformity in how things are taught. We want to be informed, but we don't all want to be the same."
Dr. Schleicher shares that concern. "A tool you design for improvement gets corrupted when you use it for accountability," he said.
Still, he seemed unfazed by the controversy. "I've been through the same thing with PISA," he said. "In the beginning everybody said, 'Not me!' Now they are all part of it."
"Ahelo is going to reveal the truth about quality in higher education," he said. "Not everybody is going to like the results."world
This article originally appeared in The New York Times.