1. Requirements were delayed so much that development didn't start until March of this year. That spells doom for a system with this kind of complexity, regardless of who the implementor is. You need several months of functional, load, and integration testing so that effectively means you would only have had 4 to 5 months to code healthcare.gov. And that's assuming there weren't any big requirements changes.

2. The people responsible for integration (Center for Medicare/Medicaid Services) had no large IT project integration experience. These are people that thought 1 week of full integration testing would be enough.

3. Healthcare.gov had to integrate with legacy systems from the IRS, Medicare/Medicaid, Social Security, in addition to the various state exchanges. Any number of those systems could have serious flaws that would make it extremely difficult to interface with. On any project a poorly implemented legacy system can dramatically affect the effort needed to be successful. Again, even the best companies would've had a sizable challenge dealing with that.

4. One of the biggest challenges in government IT is the customer. The government decision makers often don't know enough about software engineering to make sensible decisions on requirements, timelines, testing, you name it. In this case there was the added political pressure of, "This cannot fail," even though it should've been clear at least a year ago that there's no way they were going to make the deadline. But you get people who think that you can just deploy and fix it as you go along. Or you get people that think you can just add developers to make up for lost time.

5. And what does being a "startup" have to do with it anyway? Either a software company can do the work, or they can't. Whether they are a startup or an established entity really has nothing to do with it.

Large parts of the technology world operate according to the latter model, and they do so for a variety of very valid reasons. Obviously, the government and government contractors do not.

jroseattle's comment presents a speculative model for how to apply a distributed, fault-tolerant model to this kind of technology project.

But I would contend the problems weren't unmanageable. Problematic, yes, but manageable. I will try to address your points:

1. Requirements delay - sure, specifics were delayed. But, there was a basic premise for what this system would entail, and core work could progress without incurring much risk or waste. (I'm unsure when project work by CGI was started versus development work.)

2. IT integration experience - this is dependency management 101. If this is a recognized risk, then CGI should have raised the red flag. This looks like very poor project management.

3. Dependencies - yes, there is a possibility that external dependencies can affect your system. You design and architect for these scenarios, including failure.

4. Technical know-how - I would not expect anyone in government to know one iota about software development. CGI's job, in this scenario, is to elicit those requirements in a way that's useful.

5. Good point, but this project would benefit from some competition. Why not let more teams try this, rather than wrapping it all up with an entity who's only proven competency is simply receiving more contracts?

1. Having a basic idea of what the system will do is helpful, but in a system that has a lot of very specific business logic that has to be defined and HAS to be right in order to comply with the law, you're not going to get very far without good requirements. Furthermore, depending on how the contract was structured, CGI Federal may not have been allowed to start development work until it was authorized by the customer.

2. CGI may very well have raised multiple red flags, and CMS and HHS could have promptly ignored them. That is one of the maddening things about government IT. You can tell the decision makers that they are heading for disaster and they can simply ignore you.

3. Let's say you plan for every one of your dependencies to fail. What useful product could you deliver in that case? If you don't get certain information you cannot give the user a health insurance product according to the law. This is system where you cannot reasonable ignore a component if you can't get it to work in time. Let's say the interface with the IRS is going to take longer than you thought. You can't just cut that out or substitute dummy data. That would amount to failure to deliver a functioning product.

4. Again, you can elicit as hard as you want, if they don't give you the requirements because the law is still up in the air, or because the president might change his mind, or because someone wants to meet with all the stakeholders in the other agencies first, there is absolutely nothing you can do. I've been on projects where the stakeholder with the final say couldn't be bothered to give their input until the software was done. I've also been on projects where the decision maker prevented us from getting the input of the employees that would actually be using the application. Sometimes the people you're dealing with are not rational actors.

5. Fixing the contract award process is certainly part of the solution. Normally many teams could submit a bid for the work, but due to the insane time scale it was skipped for healthcare.gov. I think having a proof of concept competition after some sort of initial proposal selection gate could be useful. It could sort of be like how the military has contractors build prototypes for weapons systems competitions.

> The integrator in this case was the government (CMS is a division of HHS), there's no getting around them as a single point of failure.

This makes it easy to identify where the roadblocks are, and to present that information to interested parties at the appropriate points in time.

> 1. Having a basic idea of what the system will do is helpful, but in a system that has a lot of very specific business logic that has to be defined and HAS to be right in order to comply with the law, you're not going to get very far without good requirements.

Without requirements, how would one expect very specific business logic? And what is so incredibly precise that it derails any possible planning for it? I'm not familiar with that type of problem.

> 2. CGI may very well have raised multiple red flags, and CMS and HHS could have promptly ignored them. That is one of the maddening things about government IT. You can tell the decision makers that they are heading for disaster and they can simply ignore you.

This is very true. Of course, with basic project management 101, these things are duly noted so that when you're hauled in front of Congress, you've documented the situation. Based on CGI's responses the other day, they surely didn't do that -- otherwise, they would have raised it. Unless government IT work of this nature is classified?

> 3. Let's say you plan for every one of your dependencies to fail. What useful product could you deliver in that case?

One that allows you to fix those problems independently, rather than tying the success of the entire project to the success of all integration. In a project where the requirements are constantly shifting, I'd say the "useful product" definition is dynamic and iterative. A strict all-or-nothing definition makes little sense in that environment.

> 4. Again, you can elicit as hard as you want, if they don't give you the requirements because the law is still up in the air, or because the president might change his mind, or because someone wants to meet with all the stakeholders in the other agencies first, there is absolutely nothing you can do. I've been on projects where the stakeholder with the final say couldn't be bothered to give their input until the software was done. I've also been on projects where the decision maker prevented us from getting the input of the employees that would actually be using the application. Sometimes the people you're dealing with are not rational actors.

I get it; I've worked with federal agencies as well. It all goes back to project management and documenting interactions. As for logistics, there is always someone else to go to when a stakeholder is a bottleneck. Here's something that I found that worked in my experience with some fed agencies: identify those who are considered a project dependency, and that project progress relies on specific people. When facts are presented simply, and project management details are documented...individuals get motivated to NOT see their name in lights. As always, your mileage may vary.

> 5. Fixing the contract award process is certainly part of the solution. Normally many teams could submit a bid for the work, but due to the insane time scale it was skipped for healthcare.gov. I think having a proof of concept competition after some sort of initial proposal selection gate could be useful. It could sort of be like how the military has contractors build prototypes for weapons systems competitions.

Absolutely. Basically, it's an iterative phased approach, and let multiple teams go for it.

Of course, with basic project management 101, these things are duly noted so that when you're hauled in front of Congress, you've documented the situation. Based on CGI's responses the other day, they surely didn't do that -- otherwise, they would have raised it. Unless government IT work of this nature is classified?"

Depends on the stakes. At the moment, they're kind of low, I doubt sufficient to piss of your monopsony customer. I mean, when CGI's VP was handed this straight line (http://www.washingtonpost.com/politics/house-panel-grills-co...):

"Later, Rep. Leonard Lance (R-N.J.) asked the contractors whether they could conceive of “a more incompetent administrator” than CMS."

And replied:

"“I have no opinion on that,” Campbell replied."

Yeah, I'll bet she has no opinion on the clowns that put her front and center before the nation's Grand Inquisitor (the US Congress).

In my opinion, multiple facets of government IT have to be fixed:

A. Government leads need training on how to successfully manage IT projects.

In practice this is a huge undertaking because there are a lot of organizational challenges that have to be tackled before a lot of IT projects have a chance of success. Good managers in government will also have to tackle dysfunctional bureaucracy in order to be successful

B. Government employees need to be held accountable for their failures.

C. There needs to be more full time IT professionals in government in positions of leadership

D. Hard deadlines should not be determined politically before it has even been determined if that the deadlines is achievable

E. The contract award process needs to be reformed.

[1] http://jacobinmag.com/2013/10/delusions-of-the-tech-bro-inte...

[1] Quoted from the linked article

That's not to say hand-wavy armchair criticism of what is obviously a quagmire isn't annoying...

What we know is:

The project didn't get seriously started until February/March (e.g. the election created a 3+ month freeze on HHS publicly visible work).

The NYT reported that in the last 10 months, 7 major requirements changes were made.

We've been told the "no window shopping" one was made in August or September.

We've heard from multiple sources that changes were ordered through the week before launch.

Given all the above, how much do you see incompetence, and how much "just not done yet" pre-alpha stuff? I'm mostly a back end developer and am not up to date to judge this, I'm really interested if the above makes any refinements to your judgement.

That's not precise. There was no legal requirement that HHS freeze publicly visible work around election time. The administration, for campaign reasons, didn't want ongoing work to trigger anything that might get in the way of the re-elect. Remind me again which party is obviously oh-so-serious about governing.

But in all seriousness, this is the first major entitlement program passed without bipartisan, widespread political support (more from me here on that: https://news.ycombinator.com/item?id=6622456). The Administration was insane not to factor in that there would be all sorts of "glitches" in the development process because of that. Someone deadly serious and competent, like the shivved Tom Daschle (https://news.ycombinator.com/item?id=6622284), should have been working on this the day after it was signed, if not before. Then again, very few people seem to have that much power in this Administration ... see again the possible object lesson of Daschle.

Also keep in mind that his tax troubles came to light after Geithner's and others', so there was less slack left to give.

Still, suppose this was the one decision that sinks Obamacare....

Well, that's 20/20 ... erm, forcasting???

And, yeah, Sebelius was a known disaster when picked (I live right across the border from Kansas); I mean, she let the state run out of money one month (http://en.wikipedia.org/wiki/Kathleen_Sebelius#Tax_revenue_c...), rather a surprise as I remember. I strongly suspect she follows in the usual Obama Administration pattern Cabinet members being figureheads and cheerleaders instead of true executives, which we'll likely be about to guess about when she testifies this week.

Reid ... I'll just go with a Wall Street Journal writer's description of his coming across like a "slightly overeager undertaker" ^_^.

As I understand it, there is a lot of background integration with various agencies, systems of differing reliability, non-standard interfaces, etc. There are plenty of strategies that can deal with that environment (SOA, asynchronous queueing, etc.)

It's quite apparent this first version was not built with much flexibility in mind. Not flexibility regarding requirement changes, but flexibility for dealing with the infrastructure already in place.

If he (or any politician) wants to really support the idea of open-source they would require that everything that can be open-sourced will be open-sourced. The only exception I see to open-sourcing would be limited to almost strictly to military or defense related software and hardware projects. For every single other system, the source code should be out there in the public. Start with every single non-critical piece of software that is not part of infrastructure like transportation and public utilities and once security practices and norms improve and stabilize among the non-critical stuff, slowly and carefully open source all the rest except for military and defense projects.

At the price tax payers are paying for software, we should own the code outright many many times over.

I haven't looked into this, but can you point to any concrete examples, as opposed to the usual political platitudes?

In the world that most of us live in, this is true. But for companies like CGI their business is not really building architecturally sound systems, it is keeping the doors open to an endless stream of government contract dollars. When the government rewards failure by granting the same vendor another contract to fix the problems, the predictable happens.

Companies can also be disbarred from government contracting, but obviously that never? happens to the big, politically entrenched ones. But I wouldn't want to work for CGI Federal today....

MVP doesn't work well with deep integration. You can break this down to a form that takes one input, and returns one result on a following page. From a UI perspective, this seems like one Agile story. But that one round-trip can spawn so many integration steps. I just got finished with a health care IT project like that. One round-trip step involved integration with a single-sign-on service (which needed to be reconfigured), a rickety SOAP service provider (which had limits in how many test boxes they could set up and was controlled by a different bureaucracy and needed approval processes to turn on each required API method), a separate box returning chunks of patient data wrapped in html (don't ask, this was again out of our division's control), and our own backend system through REST so our resultant data would not be stored on the same server as our webserver (cluster). If some of these backend servers were told "okay, you now have twenty implementations to deal with instead of one", it would have drastically reduced the probabilities of completion.

I've worked with some of the federal agencies that have been listed here, and it's true -- some of these things are nightmares. But, there are architectural patterns and development approaches to deal with those.

Integration makes things challenging, but not impossible.

If you really want companies to build prototypes, they have to have some skin in the game. Otherwise you are all but ensuring fraud.

Maybe you could stipulate that each company has to meet a minimum bar in order to get that $15 million. Maybe advance some of the money up front, then pay the rest if they meet the bar. If they fail to meet the bar, they have to give back some of the money.

In any system you implement you need to give a lot of consideration to how someone might abuse the system because someone absolutely is going to at some point.

Administering this type of procurement arrangement, while certainly important to all the points you have raised, has very little to do with the success of building and delivering healthcare.gov.

The solution is obviously to pick good companies, not lemons. Except, that was the whole problem in the first place.

If we're so clueless that we can disambiguate from companies in this fictional scenario, then we're hosed anyway.

On these big gov projects you would not believe how terrifyingly thin the margins are for integrators due to politicians being very sensitive about being seen to be responsible with the public purse strings, particularly given that they don't understand the technology. In addition to the thin margins, you therefore also see extensive offshoring and very low blended rates.

The projects are still incredibly expensive in the end and barely perform because the cheap labour incurs massive technical debt - and the thin margins means that the integrators try to insulate their risk with many layers of project management, and huge amounts of rigid enterprise architecture and planning up front.

Given this mentality of cutting cost to the bone via forward planning, even proofs of concept are hard to get through, let alone investing in 15 prototypes. I think the way to do this would be as part of a gov tech investment scheme rather than attached to a specific project or program.

I have seen some infographic on how large the code base is and at the beginning of the parent article, I thought the guy is going to argue that the code base is huge because they had to circumvent/workaround the back end problems.

That's a safe assumption.

What's not is that the people calling the shots listen to them. We've seen mountains of evidence so far that they didn't.

E.g. CMS should have started integration testing of back end components and therefore issues long before the 1-2 weeks before launch, as testified to at the House hearings by representatives for CGI Federal, and more critically QSSI, which is responsible for a lot of those back end connections. Here's the latter guy (http://www.washingtonpost.com/politics/house-panel-grills-co...):

"Andrew Slavitt, an executive vice president of Optum, said the testing did not occur “until the last few days.” He said that “ideally, integrated testing would have occurred well before that date.”

Pressed on how long in advance of the launch such testing should be done, Slavitt replied, “Months would be nice.”"

These contractors are experienced, they've seen it all before, including customers from hell like CMS.

Fortunately, as noted elsewhere, CMS is out, QSSI is now the integrator and the fix-it czar is saying the right sorts of things.

"Phase I awards are typically $70,000 to $150,000 in size and the period of performance is generally from six to twelve months"

http://www.acq.osd.mil/osbp/sbir/solicitations/sbir20133/pre...