WASHINGTON -- The U.S. Supreme Court cast doubt yesterday on one of the last vestiges of official racial segregation in America -- California's practice of separating prison inmates by race when they are first incarcerated.

By a 5-3 vote, with Chief Justice William H. Rehnquist not participating, the court instructed the San Francisco-based 9th U.S. Circuit Court of Appeals to use "strict scrutiny" in reviewing California's practice of housing first-time inmates or those transferred from another prison with cell mates of the same race for as long as 60 days.

"Strict scrutiny" means a court must look at a racial classification to determine whether it serves a compelling government interest and whether the policy is narrowly tailored to achieve that objective. If the classification does not pass those two tests, it will be struck down as a violation of the 14th Amendment's guarantee of "equal protection of the laws."

"Because the California Department of Corrections' policy is an express racial classification, it is 'immediately suspect,' " wrote Justice Sandra Day O'Connor in her majority opinion. O'Connor was joined in that opinion by Justices Anthony Kennedy, David H. Souter, Ruth Bader Ginsburg and Stephen Breyer.

Justice Clarence Thomas, joined by Justice Antonin Scalia, dissented on the grounds that the court should defer to the judgment of prison officials because "the Constitution has always demanded less within the prison walls."

Justice John Paul Stevens dissented for a different reason. He would have struck down California's segregation policy without returning the case to lower courts, a course that the court had been urged by the Bush administration to follow.

At oral arguments in November, a lawyer for California argued that the temporary segregation of prisoners was necessary to give prison officials time to determine if they belonged to racially defined gangs. "California is Ground Zero for race-based prison and street gangs," Senior Assistant State Attorney General Frances T. Grunder told the court.

But Bert H. Deixler, the lawyer for convicted murderer Garrison S. Johnson, argued that Johnson's rights under the 14th Amendment were violated when he was repeatedly assigned to share a cell with another African-American inmate.

Yesterday's decision does not guarantee that Johnson will prevail when the case returns to the lower courts. O'Connor went out of her way in his majority opinion to stress that "we do not decide that the CDC's policy violates the Equal Protection Clause."

But Deixler said he considered the decision the "functional equivalent" of a victory because "there's no way on the record of this case that California could withstand strict scrutiny." Calling the justices' decision to remand the case to lower courts a "dignified way out," Deixler said he hoped California would settle the lawsuit by abolishing the blanket segregation policy.

"I'm hopeful that they will decide that this policy has outlived its usefulness and that they should join the rest of the United States," Deixler added.

If California does try to convince lower courts that its segregation policy is narrowly tailored to the goal of minimizing racial violence, the state will be hampered by the fact that virtually every other state and the federal prison system manage to deal with violence without routinely segregating prisoners. In his dissent, Thomas said that Oklahoma and Texas also initially segregate prisoners by race, but O'Connor disputed that claim.

(Pennsylvania does not have a general policy of assigning inmates to cells on the basis of race, said Sheila Moore, a spokeswoman for the state Corrections Department. But she added that there might be exceptions when pairing inmates of different races "might lead to harm to either inmate.")

Yesterday's result confirmed the prediction of legal commentators that the outcome of the case would depend on whether a majority of the court viewed it primarily as a case about prisoners' rights or as a case about the stigma of racial classifications.

In her majority opinion, O'Connor took the first approach, citing Brown v. Board of Education, the 1954 decision in which the court outlawed segregated public schools.

But Thomas' dissent emphasized the points that prisoners enjoy fewer rights than other peoples and that the courts traditionally defer to the judgment of prison officials. Thomas also said that California had a special problem with racially based prison gangs, noting that "there are at least five major gangs in this country--the Aryan Brotherhood, the Black Guerrilla family, the Mexican Mafia, La Nuestra Familia and the Texas syndicate--all of which originated in California's prisons."

Beyond the issue of segregation in prisons, yesterday's opinions shed light on a continuing debate among the justices about whether some racial distinctions are more offensive to the Constitution than others.

In her majority opinion, O'Connor cited not only Brown v. Board of Education, in which blacks were the victims of discrimination, but two cases in which the court took a "color-blind" approach to discrimination. One was Powers v. Ohio, a 1991 case in which the court held that a white defendant could challenge the prosecution's exclusion of blacks from the jury. The other was O'Connor's own majority opinion in a 1993 decision finding fault with a North Carolina congressional reapportionment plan drawn to maximize the influence of black voters.

That inspired Ginsburg, joined by Souter and Breyer, to file a concurring opinion emphasizing that she saw a difference between racial classifications like the segregation of prisoners in California and other classifications, such as those used in university affirmative-action programs, that are designed to compensate for past discrimination.