NEW YORK -- The tourists from California peered through the slats of a metal fence surrounding the World Trade Center site, looking down into the nearly empty 16 acres for a sign of what happened there on Sept. 11, 2001.

Four years after terrorists hijacked jetliners that destroyed the twin towers, Steve and Marta Pilling thought they would find a memorial, something more than the names of the 2,749 victims on panels attached to the fence.

"This reminds me more of a construction site," not the ground zero etched in Americans' consciousness, said Steve Pilling of Murietta, Calif.

The fact that the downtown Manhattan site is both -- a lucrative piece of real estate with grand plans for skyscrapers and museums, and the place where the nation's worst terror attack must be remembered -- has driven a rebuilding process fraught with delicate negotiations and often competing passions of politicians, developers, architects and family members.

"It's the most emotionally charged building project in the world," said Robert Yaro, head of the Regional Plan Association advocacy group in New York.

Common ground at Ground Zero has been hard to find: Ambitious, thoughtful plans for everything from a 1,776-foot tower to a performing arts complex are on paper, but construction on most buildings has yet to begin.

Tomorrow, a day after a ceremony marking the fourth anniversary of the attacks, work is starting on one major project: a $2.2 billion transit hub that replaces a temporary station that opened in 2003.

Leaders of the process say a remarkable amount has been accomplished, and that rebuilding a site like this is unprecedented.

Others say the plans are unfocused and prioritize rebuilding office space with a tallest-in-the-world skyscraper over a memorial and more pressing community needs.

"There's no demand whatsoever for commercial space" in the area, said Fred Siegel, a history professor at the Cooper Union for the Advancement of Science and Art, who said rebuilders have blown an opportunity "to rethink lower Manhattan in toto."

"The memorial itself has been an afterthought," said Bill Doyle, whose son, Joseph, died at the trade center. "It's astounding to me that the only thing they have up there after four years are a couple of posters."

The Freedom Tower has suffered more setbacks and missed deadlines than other plans for the space, which include four more office towers, a memorial surrounded by a grove of oak trees, a performing arts center, and separate museums devoted to Sept. 11 and to freedom.

City police this May forced rebuilding officials to order a third design of the building after police expressed concerns that it was not secure enough to withstand a terrorist attack. After breaking ground on July 4, 2004, with a 20-ton inscribed granite cornerstone at the site, developers now say that cornerstone will have to be moved several feet to be part of the redesigned Freedom Tower.

Larry Silverstein, the private developer who leased the trade center and is supervising building of the Freedom Tower, called the setback "a most unfortunate set of circumstances, a lack of coordination between various governmental entities." But he said that the new design by architect David Childs, is "elegant, exquisite," and called the slender tower topped by a center spire an improvement over the last design, a twisting, glass and steel structure meant to evoke the Statue of Liberty.

A second groundbreaking for the Freedom Tower -- and a groundbreaking on the memorial, "Reflecting Absence" -- will occur within six months, Gov. George E. Pataki and Mayor Michael Bloomberg have announced.

The governor and then-mayor Rudolph Giuliani started the redevelopment process shortly after the attacks by creating the Lower Manhattan Development Corp. But it is Pataki who has taken more of a leadership role, calling downtown Manhattan's resurgence a top priority. He has also been assigned blame more often for the problems.