WASHINGTON -- Crammed into conference rooms with pizza for dinner, some programmers building the Obama administration's showcase health insurance website were growing increasingly stressed. Some worked past 10 p.m., energy drinks in hand. Others rewrote computer code over and over to meet what they considered last-minute requests for changes from the government or other contractors.
As questions mount over the website's failure, insider interviews and an Associated Press review of technical specifications found a mind-numbingly complex system put together by harried programmers, who pushed out a final product that congressional investigators said was tested by the government and not private developers with more expertise.
Meanwhile, the White House said President Barack Obama's longtime adviser Jeffrey Zients will provide management advice to help fix the system. White House press secretary Jay Carney said Mr. Zients will be on a short-term assignment at the Health and Human Services Department before he is due to take over Jan. 1 as director of Mr. Obama's National Economic Council.
Mr. Carney cited Mr. Zients' expertise as a longtime management consultant and his "proven track record" since coming to the White House in 2009, both as interim budget director and as chief performance officer, when he headed an effort to streamline government and cut costs. "We're engaged in an all-out effort to improve the online experience," Mr. Carney said.
This is not the first time that Mr. Obama has turned to Mr. Zients for help solving a major problem. In the 2009, after far more drivers than anticipated signed up for the Cash for Clunkers program that promised rebates to people who traded in their old cars for more fuel-efficient vehicles, Mr. Obama assigned Mr. Zients, his deputy budget director at the time, to help eliminate the backlog. When the same thing happened with sign-ups for an updated version of the GI Bill, one designed to help the 9/11 generation of veterans get a college education, Mr. Obama again turned to Mr. Zients.
"He's not going to be looking under the hood and tell you 'I can fix the coding, I can fix it,' " Kenneth Baer, a senior adviser to Mr. Zients at the budget office, said of Mr. Zients' newest assignment. "His skill is going to be how to identify challenges, prioritize what solutions need to be done next, assessing what talent is already available and then how to motivate them to do that job as quickly and as ably as possible."
Aneesh Chopra, who was Mr. Obama's chief technology officer, said Mr. Zients is extremely skilled in figuring things out from a management perspective. "If I was confident this issue would be resolved before his participation, I am doubly so now," said Mr. Chopra, who also worked with Mr. Zients at the Advisory Board Co., one of two business advisory firms where Mr. Zients held top posts. "Jeff's track record is really a relentless focus on execution."
HHS Secretary Kathleen Sebelius said in a post on HealthCare.gov that her agency is also bringing in more experts and specialists from government and industry, including top Silicon Valley companies. "This new infusion of talent will bring a powerful array of subject matter expertise and skills, including extensive experience scaling major IT systems," she said. "This effort is being marshaled as part of a cross-functional team that is working aggressively to diagnose parts of HealthCare.gov that are experiencing problems, learn from successful states, prioritize issues and fix them."
Project developers for the health care website, who were interviewed on condition of anonymity because they feared that they would otherwise be fired, said they raised doubts among themselves whether the website could be ready in time. They complained openly to each other about what they considered tight and unrealistic deadlines. Website builders saw red flags for months.
An AP review of internal architectural diagrams revealed the system's complexity. Insurance applicants have a host of personal information verified, including income and immigration status. The system connects to other federal computer networks, including ones at the Social Security Administration, IRS, Veterans Administration, Office of Personnel Management and the Peace Corps.
Mr. Obama on Monday acknowledged technical problems that he described as "kinks in the system." But in remarks at a Rose Garden event, he offered no explanation for the failure, except to note that high website traffic caused some slowdowns. He said it had been visited nearly 20 million times -- fewer monthly visits so far than many commercial websites, such as PayPal, AOL, Wikipedia or Pinterest.
"The problem has been that the website that's supposed to make it easy to apply for and purchase the insurance is not working the way it should for everybody," Mr. Obama said. "There's no sugarcoating it. The website has been too slow. People have been getting stuck during the application process. And I think it's fair to say that nobody is more frustrated by that than I am."
The online system was envisioned as a simple way for people without health insurance to comparison-shop among competing plans offered in their state, pick their preferred level of coverage and cost and sign up. For many, it has not worked out that way so far.
Just weeks before the Oct. 1 launch of HealthCare.gov, one programmer said, colleagues huddled in conference rooms trying to patch "bugs," or deficiencies in computer code. Unresolved problems led to visitors getting cryptic error messages or enduring long waits trying to sign up.
Congressional investigators have concluded that the government's Centers for Medicare and Medicaid Services, not private software developers, tested the exchange's computer systems during the final weeks. That task, known as integration testing, is usually handled by software companies because it ferrets out problems before the public sees the final product.
The government spent at least $394 million in contracts to build the federal health care exchange and the data hub. Those contracts included major awards to Virginia-based CGI Federal Inc., Maryland-based Quality Software Services Inc. and Booz Allen Hamilton Inc. CGI Federal said in a statement Monday that it was working with the government and other contractors "around the clock" to improve the system, which it called "complex, ambitious and unprecedented."
The schematics from late 2012 show how officials designated a "data services hub" -- a traffic cop for managing information -- in lieu of a design that would have let state exchanges connect directly to federal servers when verifying an applicant's information. On Sunday, the Health and Human Services Department said the data hub was working but not meeting public expectations.
Administration officials so far have refused to say how many people actually have managed to enroll in insurance during the three weeks since the new marketplaces became available. Without enrollment numbers, it is impossible to know whether the program is on track to reach projections from the Congressional Budget Office that 7 million people would gain coverage during the first year the exchanges were available.
Instead, officials have selectively cited figures that put the insurance exchanges in a positive light. They say more than 19 million people have logged on to the federal website and nearly 500,000 have filled out applications for insurance through both the federal and state-run sites.
The House Energy and Commerce Committee was expected to conduct an oversight hearing Thursday, probably without Ms. Sebelius testifying. She could testify on Capitol Hill on the subject as early as next week.
Uninsured Americans have until about mid-February to sign up for coverage if they are to meet the law's requirement that they be insured by the end of March.