Major Automated Information Systems: Selected Defense Programs Need to Implement Key Acquisition Practices
Highlights
What GAO Found
Of the 15 selected Department of Defense (DOD) major automated information system (MAIS) programs, 13 had cost information available (2 did not, due to revisions to requirements and changes in scope). Of these 13 programs, 11 experienced changes in their cost estimates, including 7 that experienced increases ranging from 4 to 2,233 percent and 4 that experienced decreases ranging from 4 to 86 percent. Two programs remained unchanged in their cost goals. Additionally, of 14 programs that had schedule information available (1 did not due to revisions to requirements), 13 experienced schedule changes—including 12 that had slippages ranging from a few months to 6 years, and 1 that accelerated its schedule. One program remained on schedule. Further, of 11 programs that had system performance data available, 3 programs met their system performance targets, while 8 did not fully meet their targets.
The three programs selected for analysis of risk management demonstrated mixed progress in effectively defining and managing risks. Specifically, the Defense Health Agency's (DHA) Theater Medical Information Program – Joint Increment 2 had implemented key risk management practices. While the Navy's Global Combat Support System – Marine Corps program did not, among other things, update its risk tracking log during a 5-month period in 2013, the program recently updated its risks and mitigation plans, which should help the program to more effectively manage risks going forward. The Defense Logistics Agency's (DLA) Defense Agencies Initiative program had taken steps to implement selected risk management practices, including establishing a risk management board. However, the program was still in the early stages of identifying risks and had not yet identified a comprehensive set of program risks, nor consistently evaluated and categorized its risks. Until this program maintains a complete risk log and mitigation plans, and accurately evaluates and categorizes its risks, it will lack assurance that it is appropriately mitigating all identified risks.
The three programs also demonstrated mixed progress in implementing key requirements management and project monitoring and control best practices. Specifically, the Navy program had implemented all key requirements management best practices and the DLA program had recently taken steps to do so. However, while the DHA program had implemented many requirements management best practices, it had not maintained complete traceability between its requirements and work products. Additionally, the DHA program had not updated its capabilities baseline to reflect program scope changes. Until DHA implements these requirements management best practices, stakeholders will lack assurance that the system will have all intended functionality to meet users' needs. Regarding project monitoring and control, each of the programs lacked key practices. For instance, the DLA program had not tracked significant deviations in performance. Additionally, the Navy program did not always take timely corrective actions to address issues. Further, the DHA program did not use earned value management to track contractor performance in meeting planned cost targets, even though these data were being collected and certain contracts met DOD's threshold for its use; as such, the program had not effectively determined progress against the plan. Until the three programs implement these project monitoring best practices, they will be limited in their ability to manage the programs.
Why GAO Did This Study
The National Defense Authorization Act for Fiscal Year 2012 mandated that GAO select and assess DOD MAIS programs annually through March 2018. This report discusses the results of GAO's second annual assessment. Based on the act's requirements, GAO's objectives were to (1) describe the extent to which selected MAIS programs have changed their planned cost and schedule estimates and met performance targets; (2) assess selected MAIS programs' actions to manage risks; and (3) assess the extent to which selected MAIS programs used key information technology acquisition best practices.
To do so, GAO selected 15 of the 42 DOD MAIS programs based on several factors, including representation from multiple DOD components, and summarized the results of analyses of cost, schedule, and performance across the programs. Further, GAO selected 3 of the 15 programs (1 each from DHA, DLA, and Navy) and assessed them against best practices for risk management, requirements management, and project monitoring and control.
Recommendations
GAO recommends that DOD direct the programs to address respective weaknesses in their risk management, requirements management, and project monitoring and control practices. DOD concurred with six of GAO's recommendations and partially concurred with the remaining two. GAO maintains that it is important that the DHA program trace all capabilities to their associated requirements and update its capabilities baseline to reflect program scope changes.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Defense | To better ensure that the Defense Agencies Initiative (DAI) implements effective risk management and information technology (IT) acquisition best practices, the Secretary of Defense should direct the Director of the Defense Logistics Agency to direct the DAI program office to establish a comprehensive risk log that includes all up-to-date risks with evaluations and categorizations that comply with DLA's defined parameters; and associated mitigation plans. |
The Defense Logistics Agency established a risk log for the Defense Agencies Initiative program that includes risk evaluations and categorizations, and associated mitigation plans. The program also periodically reviews the status of each risk. By taking these actions, the program should have better assurance that it has identified potential problems before they occur and should be better positioned to implement risk-handling activities, as needed, to mitigate adverse impacts if a problem does occur.
|
| Department of Defense | To better ensure that DAI implements effective risk management and IT acquisition best practices, the Secretary of Defense should direct the Director of the Defense Logistics Agency to direct the DAI program office to identify and document significant cost and schedule deviations from the program's plan. |
The DAI program is using earned value management to track cost and schedule performance and to identify deviations from the program's plan. While DAI has not had any significant cost or schedule deviations from its baseline since our 2014 report, officials demonstrated their process for documenting such a deviation, if one occurred. By tracking cost and schedule information and having a plan for documenting significant deviations, DAI stakeholders should have a more complete understanding of the cost and schedule performance of the program, which should help improve their ability to fully monitor and control it.
|
| Department of Defense | To help ensure that the Global Combat Support System-Marine Corps (GCSS-MC) implements effective project monitoring and control best practices, the Secretary of Defense should direct the Secretary of the Navy to direct the GCSS-MC program office to include corrective actions and time frames in future analyses of critical issues and monitor actions taken against those time frames. |
Marine Corps officials stated that the GCSS-MC program did not experience any critical issues after our report was issued in March 2014. The program achieved Full Deployment in December 2015.
|
| Department of Defense | To improve the Theater Medical Information Program - Joint Increment 2 (TMIP-J) program's implementation of IT acquisition best practices, the Secretary of Defense should direct the Director of the Defense Health Agency to direct the TMIP-J program office to update the program's capabilities baseline to reflect program scope changes. |
The TMIP-J Increment 2 program did not update its capabilities baseline subsequent to our March 2014 report. The program achieved Full Deployment in May 2016.
|
| Department of Defense | To improve the TMIP-J program's implementation of IT acquisition best practices, the Secretary of Defense should direct the Director of the Defense Health Agency to trace all capabilities to their associated requirements in the requirements traceability matrix. |
While the department provided an updated requirements traceability matrix for TMIP-J Increment 2, program officials did not trace all of TMIP-J's capabilities to their associated requirements in this matrix. The program achieved Full Deployment in May 2016 and the department does not plan to update the traceability matrix again.
|
| Department of Defense | To improve the TMIP-J program's implementation of IT acquisition best practices, the Secretary of Defense should direct the Director of the Defense Health Agency to implement earned value management in accordance with DOD's policy and train management staff with project oversight responsibilities on the proper use of earned value management. |
The department trained nearly all of its TMIP-J management staff with project oversight responsibilities on the proper use of earned value management. The department also provided an estimated date by which the remaining staff member who was not yet fully trained would complete it. Training these management staff on the appropriate use of earned value management helps to ensure that these officials are able to effectively manage the program. In addition, TMIP-J officials reported that they hired a contractor to assist them in monitoring and evaluating the program's earned value data. By properly training staff and monitoring and evaluating earned value data, TMIP-J officials should have received valuable insights into the program's performance.
|
| Department of Defense | To improve the TMIP-J program's implementation of IT acquisition best practices, the Secretary of Defense should direct the Director of the Defense Health Agency to develop an authoritative listing of all interfaces currently included in the scope of the program. |
The Department developed and provided a listing of all interfaces included in the scope of the TMIP-J Increment 2 program. By developing this listing, program officials should have had a more complete understanding of the status and scope of the program, which should have helped improve their ability to fully monitor and control it.
|
| Department of Defense | To improve the TMIP-J program's implementation of IT acquisition best practices, the Secretary of Defense should direct the Director of the Defense Health Agency to report to stakeholders the number of current and planned system users and sites and provide updates as needed. |
The TMIP-J program reported to stakeholders the number of current and planned sites that would use TMIP-J Increment 2, Release 2. Reporting this information to stakeholders informed them of important program scope information and helps them to appropriately plan for system maintenance.
|