Changes to Requests for Proposals (RFP), proposals and contracts are among the 25 recommendations from a National Research Council (NRC) in assessing how the Defense Department could increase the reliability requirements of defense systems.
Every DoD system must achieve a specified reliability requirement before being approved for acquisition, but operational urgencies have led to systems fielded without demonstrating adequate reliability, according to the March report issued by the National Academies Press, Reliability Growth: Enhancing Defense System Reliability.
“Between 2006 and 2011, one-half of the 52 major defense systems reported on by the DoD Office of the Director, Operational test and Evaluation (DOT&E) to Congress failed to met their prescribed reliability thresholds, yet all of the systems proceeded to full-rate production status,” the report said.
Looking at developmental and operational testing, the report found that “system reliabilities, both actual and estimated, reflect the particulars of testing circumstances, and these circumstances may not match intended operational usage profiles.”
The panel’s recommendations, including those addressing contractors, fall under four aspects of the acquisition process: system requirements, RFPs, and proposals; design for reliability; reliability testing and evaluation; and reliability growth models.
The NRC recommends that the Undersecretary of Defense for Acquisition, Technology and Logistics USD (ATL) ensure that reliability is a key performance parameter and a “mandatory contractual requirement in defense acquisition programs.”
Another recommendation is for all proposals to specify the design for reliability techniques the contractor would use during system design for hardware and software. And, the proposal budget should have a line item for the cost of design-for-reliability techniques, the associated application of reliability engineering methods and schedule adherence.
As well, contracts should mandate an initial plan for system reliability and qualifications–this would include failure definitions and scoring criteria that would be used for contract verification–as well as a description of the reliability organization and reporting structure. Once a contract is awarded, the plan should be regularly updated, and DoD should have access to the plans, updates, data and analysis.
Considering software, all contractor proposals should specify a management plan for software development and require the contractor provide DoD with full access to the software architecture, the metrics being tracked and a log of the management of system development that includes failure reports, when they happened and when they were resolved.
DoD also should mandate that contractors archive and deliver to DoD–including the operational test agencies–all data from reliability testing and other relevant analyses. Failure data should be part of this..
Subcontractors also are addressed. The panel recommends a line item in all acquisition budgets for “oversight of subcontractors’ compliance with reliability requirements” and that oversight plans are part of all proposals.
The recommendations include USD (ATL) mandate that before prototypes are delivered for developmental testing, the contractor provide test data in support of a statistically valid estimate of system reliability consistent with the operational reliability requirement. “The necessity for this should be included in all requests for proposals.”
Both DOT&E and the USD AT&L asked the NRC to conduct the study through its Committee on National Statistics. A Panel on Reliability Growth Methods for Defense Systems was created to conduct the study.