A Q&A with the FBI's data czar
I told the director, I said, if they [SAIC] pull it off–with a big "if"–if they pull it off, he's going to have one of the best case-management systems that the organization has seen. And he goes–he asked me, he says, "Why are you saying 'if'?"
— Zalmai "Zal" Azmi, chief information officer, FBI
Feb. 14, 2005
Last February, the FBI's chief information officer sat down with Chitra Ragavan, chief legal affairs correspondent for U.S.News & World Report, to discuss the bureau's efforts to transform its antiquated computer systems. The bureau's technology woes have been blamed in part for its failure to detect or prevent the Sept. 11, 2001, terrorist attacks.
Azmi described in detail how the FBI's ambitious $170 million case-management software development project, known as Virtual Case File (VCF), ended in a costly debacle. Mueller has told Congress that taxpayers will take a $104 million bite from the VCF collapse [FBI disputes U.S.News cost estimates (6/8/05)]. Among other things, Azmi said that the contractor–Science Applications International Corporation–failed to design a system that gave the FBI the security measures it needed to compartmentalize case information. Azmi blamed SAIC for planning what he thought was an overly ambitious, all-at-once transition – called Flash Cut Over – from the old "legacy" Automated Case Support system (ACS) to the new VCF system, rather than phasing in the transition of tens of millions of documents. He also blamed SAIC for "fast-tracking" the project to such an extent that it created "silos" of information that could not interconnect. SAIC has said it was the FBI's desire to fast-track the project that contributed to VCF's collapse. Azmi also described how the FBI found 400 bugs in the VCF software and discussed the problems with SAIC–a contention that runs contrary to a recently released Congressional report that said the FBI never shared the bugs with SAIC. Azmi said that FBI director Robert Mueller knew for sure in May 2004 that the VCF system was a failure. However, Mueller did not publicly acknowledge to Congress until this February–nine months later–that he was pulling the plug on VCF. By then the bureau had proffered hundreds of optimistic briefings to members of Congress and their staffs. Mueller also urged his 56 field-office bosses to train their agents in VCF despite knowing that the system was on life support. When they tried to take the tutorial, the program kept crashing.
Here are excerpts from the U.S.News & World Report interview with Azmi.
Q: When you came to the FBI, what was the status of Virtual Case File?
A: I think it was end of November [2003] or beginning of December, I met with SAIC to actually get a demo of their VCF Application . . . And they had said it was going to be ready for deployment, full deployment by Dec. 17, 2003. My impression was that if you really pull it off, I'll be very surprised.
Q: You told them that?
A: Yeah. We had that discussion. Cause . . . the program was very large, I mean the data migration . . . You had specific questions, on how do you migrate the data from one system to the new system. On how you handle security. And . . . really nobody could answer that question. I said, you're building your software in eight silos, how are you going to integrate the eight silos, and there was not a clear answer for that, who was in charge of consolidation. And I think that that's one thing they're [SAIC] saying . . . because they wanted to fast-track everything, they brought eight teams of software developers to develop VCF, as a result they had a code overlap , what we call it because everybody was developing their own solution.
Q: Okay. So you come back from this meeting and you're surprised, right? They're saying you're going to pull it off and you have grave questions?
A: Right.
Q: So, you came back and what did you tell the director?
A: I told the director, I said, if they pull it off with a big "IF" if they pull it off, he's going to have one of the best case management systems that the organization has seen. And he goes–he asked me, he says, "Why are you saying "IF"? I said, because I'm not sure that the demo I saw has the system capabilities and that was true, I did not see the actual data behind the system. And then he asked me the question, he goes, "What do you think of this Flash Cut Over, why can't we–this is a big risk, how can we get rid of it?"
And the idea behind that was we got to bring a legacy system down [Automated Case Support] and we're going to bring VCF system on line. And the question that we had in the discussion was, well, so you're bringing down that old legacy system, you're bringing VCF up and they said, yes . . . So, I said, well if it crashes what happens to the data. They said, "Well, we're going to have to wait them, bring it back on line." That was the major concern the director had and we both shared that concern that, you know, here we're moving from a legacy system. So, okay, it gets old but it's working, it's stable.
So, he said, why don't you take a look at it and then from that point we met with the National Research Council and that was one of their recommendations that we should reconsider our decision. But by then we already knew that, what SAIC delivered and on December 17, was not the product we were looking for.
Q: So, at that meeting with the director did you say to him, "I don't think this is going to work?"
A: I told them, if they pull it off–
Q: But you never said to him –
A: That it's not going to work? No. Because this program is too huge to make that kind of judgment call at one setting. I mean, I don't know if the reasons why I recommended to the director that we would have to have an independent evaluation was because of the complexity of the program. I mean, I can sit down and look at, in terms of architecture, I can look at it in terms of requirements or I can sit down and look at it in terms of security, but to do that from end-to-end you really need an independent team to do it.
Q: But instinctively, did you know it was a dud? Just from your years of having done this?
A: Not really. My concern was in, like I said, first of all we didn't have the legacy data. I mean, the interface was there . . . the work flow, everything worked.
Q: What was missing?
A: . . . The backend data, the rules, the security implementation. I mean the facade was good, you know, you could see the program that was navigating from one screen to the other and people were being assigned tasks, but not with the actual data in the background.
Q: The system worked as long as there was no data in it?
A: The system worked as long as there was no checks for security or actual data.
Q: If you don't have data, what point is the system?
A: But they were going to the data conversation, so that's why I said, I couldn't make a judgment call at that point.
Q: What happened if they put data in it? It crashed?
A: . . . We had a subset of data, very little data. I don't know how many records. But for that small amount of records it worked just fine. There was the other concern; what happens if we put 30,000 users on it? And all of the data out of ACS [Automated Case Support]–would the system be able to handle it?
Q: Did it crash under its own weight because of the security requirements?
A: I don't think security requirements was the one that crashed the system. They took a different model from what we specified.
Q: What do you mean?
A: We asked for a "role-based" security development. . . . That means we get work based on your roles, not on you as an individual. For example if all supervisors would be doing X, Y, Z. So for supervisors we will build a security model. So, as long as you're a supervisor, the system would know how to handle your access. What SAIC implemented was based on the individual, like "Who is Mike?" and that is how the security has been implemented.
Q: And so what?
A: Very difficult to maintain because I have to know all 30,000 people in the bureau what they are doing, what their roles are.
Q: Why didn't they do what you wanted?
A: Difficult to implement, very difficult to implement.
Q: So, basically, what you wanted from them was very difficult for them to do. What they created for you was impossible for you to maintain, right?
A: Yeah. It's cumbersome.
Q: So, there's just no way to make your needs compatible with their abilities on the security problem?
A: The other part of it is the program itself is a proprietary software that was developed by SAIC. I mean they didn't make use of the commercial-off-the-shelf products that were out there. They coded everything, [730,000 lines of code] and that means that I have to know everything–before I can maintain that software, we should know everything about the software that SAIC does, and that is almost impossible.
Q: In a simple way, can you explain what VCF would have done for FBI agents?
A: It would give them automated workflow. That means that I don't do paper work anymore. I do my 302 [investigative case reports], I send it to my boss, he looks at it, he . . . would use one of this public key infrastructure keys, and electronic signature, once it was signed it would up load into the system which would be a record management system and a document management system so we motion control for our documents, we will have a record . . . so that's the capability it will give you. Record management, at any given time you will know where the document is, who has it, who's working on it. Accountability for the agents, for the supervisors, for the record manager, all of those people would know who has the data, where it's going.
Q: Now, tell me the contrasts.
A: The contrasts–we don't have the capability in ACS right now. Our record management is paper-based, our signature authority is paper based, so it's time consuming.
Q: When did the director know for sure that this stuff was not going to work?
A: I would say, probably in May was when I had, I had to brief him and . . .
Q: May 2004?
A: Yes.
Q: You briefed him–what did you say?
A: I told him, I said, surface software is not going to work. And he said, why? And I said, well there's a trend that is developing since my arrival, and [in] December they gave us the software, in January we went back to them and said, well, there are some deficiencies in the software. They said, well, there are 17 deficiencies, we decomposed the 17 deficiencies, they turned into 59 deficiencies. Then we had a two-week sit-down with SAIC and those 59 turned into 400 deficiencies. So there's a trend developing and unless we do an in-depth analysis of the software and invest another year and another $56 million in this program, it is ludicrous, and it just doesn't make any sense. You know, I have a base-line software that I was told that 90 percent was ready, yet they were asking for another $56 million to develop the other 10 percent. Now where's the logic in this one. So . . . and up to that point, while I was meeting SAIC on . . . almost on a weekly basis and doing a proper support with them and meeting with new chief technology officer and new president for the business side of the house, I was collecting information, you know, because this is a tough decision for a new CIO to make. You know, we have to spend, I don't know, about $67/$80 million at that point, 3 years of investment, we had a number of program managers in here working on this, the whole country was waiting for it, and here you are saying, "You know what, sir, I'm sorry, but we have to throw it away." And the question was like, based on what? You know, I had our evaluation, I had the trend, I had all that information, but it wasn't independent, and that was one of the biggest things that I knew would come back and haunt us. You know, it was FBI's evaluation. So, that's why I suggest in . . . again, at the end of May and . . . in June I told him, I said, "Sir, here's what I suggest. We will take the recommendations that the National Research Council has made, we will take a segment of the program that is really mature and develop it forward in a prototype in a pilot mode and then we'll hire an independent company to do an end-to-end evaluation of VCF, to come back and tell us, is it worth [it] for [the] FBI to invest another $56 million and another year plus to move forward or not?"
Q: Why not tank it right then? You're the CIO; you knew it wasn't going to work.
A: Like I said, I didn't have all of the information. I had probably 10 percent of the information and that 10 percent was alarming enough to launch a full investigation of the software.
Q: By then, Director Mueller had a pretty good idea this wasn't going to work then?
A: I'll be honest with you, I think he was so hopeful that the software's going to work.
Q: Why?
A: Because nobody could believe that after three years and all of this money, we didn't have it.
Q: You knew it wasn't going to work, right?
A: Pretty much, I was confident that it didn't work.
Q: He didn't believe it?
A: It was hard for everyone, for everyone, I mean, for all of management, for all of our users, I mean, we couldn't fathom it three years later. You know, $70/80 million later, we don't have anything to show.
Q: The thing that was more interesting was that the director had taken a personal interest in this and had personally managed it, right?
A: Well . . . I will tell you that in 2004 I have met with the director twice a day, every morning and every evening on information technology. So, I know how engaged he was and how much he wanted to know, especially about VCF. When they [SAIC] came back and we had this meeting, and they ask for $56 million and another year, and at the end they said, they are not going guarantee the delivery of the document management system, the record management system . . . it was like, well if you're not going to guarantee delivery of records management system, how are you going to get all of my data from legacy system into the new system. If you can't guarantee this, this piece is not going to happen.
That was a short meeting. And we left and I looked at the director, I said, "We'll take a two-track approach. I'll take the most mature part of the VCF, we'll move forward with it, we'll deliver something by December of 2004." Because we wanted to do a live pilot, real data, in one of our field offices and the other track would be that we're going to do an evaluation of the software, all 730,000 lines of code, we will see how healthy that software is, we'll look at . . . implementation, performance; we'll look at architecture, all of that stuff and I said, at the same time we'll ask for an independent evaluation of commercial-off-the-shelf-products. Are there products that can easily be integrated into VCF, shorten the pipeline. If we decide we're going to go with VCF, can we, you know, infuse some new technologies, commercial-off-the-shelf? So, when I presented that in January, sorry, in June, he approved it and we moved forward.
Q: So, when did the director finally accept that this was not going well? When would you say?
A: I would say the Aerospace Report [an independent evaluation] probably sealed it for everyone . . .
Q: Which came when?
A: January 21st this year.
Q: So, up to then, he was pretty much hoping that it was . . . ?
A: Everybody was, OMB was, DOJ was, everybody was hoping, except for people who were doing software engineering. Cause everyone was like, "Who's going to maintain 730,000 lines of code?"
Q: So where do you go from here?
A: Oh, we have a number of strategies. . . . It's very close held, mainly because it's procurement sensitive because we have outlined the timeline across the technology . . . everyday, I get asked that question, "What are you doing?" And we're sort of tight-lipped about it. Make sure that the people have a fair advantage when the contract goes out. But the bottom line is . . . I don't want people to wait for the mother lode, because by the time we get the mother lode there may be already a change in the technology anyways. I'd rather give them the capabilities that they need right now and it's more critical and important to them . . . I mean, there are certain things that we have to do and we have to right, for example, this whole data transition. I mean, how we going to get our data out of the legacy system?
Q: How much is it?
A: It's about 30 million documents I think, sitting in ACS. Nobody has been successful to do that kind of transition.
Q: The director's pretty savvy about these things . . . one of the things about him is that he's really very . . . pretty sophisticated about computers. So, did it raise red flags with him when SAIC offered this deal where they would make this complete transition from one system to the other?
A: I would say, that piece of it requires a different skill set and that's purely software engineering. I mean, the director is very sharp, he knows his data bases, he knows some of the technology but when it comes to legacy systems, you know, he knew that we had a data base, which is pretty old, but the pieces that probably were not explained to him, is that our security mechanism are coded in natural language, in database, okay. Now, remember what I mentioned about the supervisors, how they have these rights. Think about if you have 100 supervisors, they have been coded for security a hundred times in the system. Then if we have 200 different types of cases, every one of them has been coded separately for security. Now to map this kind of world, into something that is more manageable, that's not an easy task. I don't think . . . I'll be honest with you, I don't even think SAIC had thought about that.
Q: The security needs of . . .
A: Of the data transition. When you are transitioning the data from legacy system into the new relational databases, the security must match. If the security doesn't match, then you have all of the data sitting in a database that provides no security. So, all of the sudden 30,000 users have access to all of the data. That is not how ACS works. Everybody has his own credential. Everybody has his own purpose.
Q: The system that SAIC developed had none of that?
A: I don't know. The way they have delivered it right now, with the source code that I have, no. The source code that we've evaluated, no.
Q: Now, when was Congress first told that most likely this is going to fail?
A: We did one briefing I think in May [2004]. The director did a briefing in May that told them that we would deliver certain capabilities by December . . . and we did good on that one [the New Orleans pilot project]. And I think the next time we told them the VCF was not going to make it before the beginning of this year because by then I had the draft reports I was looking at.
Q: Should they have been told earlier?
A: Ah, no, because these are the things that you always want to have a final proof of . . . in your hand. You just can't go out there and accuse organizations that they couldn't deliver if you haven't done your homework. So, you want to be very careful. Because it almost sounds like FBI is accusing SAIC. And that was not the case. FBI was evaluating SAIC's software. And once we had the proof at hand, that's when we told the director and the director actually made courtesy calls.
Q: Now when was the SAIC contract given out?
A: The initial contract, I think, was in 2001.
Q: Shouldn't somebody have known earlier that it wasn't going to work?
A: I think most of the people should have known. And most of the people should have told him [Mueller]. I mean, SAIC should have known, our supporting contractors should have known, I mean . . . software just doesn't go bad overnight. . . . You have to, to keep in mind that when you look at a program of the complexity of VCF and you've seen the demos that it works, and you have spent all of this time on it, all of this money, and energy and it's very difficult to believe that after three years of software . . . and a company of SAIC's magnitude . . . So a company like this should be able to deliver. And without having that final document that says, okay, we looked at all of these things, we took every one of the sections of the source coding documentation, we mapped it against best practices, we mapped it against standards for software development and we found out that there's no assurance that the software's going to work.