As the Pentagon acquisition community seeks ways to become more efficient with its limited funding, the Defense Department’s top acquisition official said there is some tension between developmental testers and operational testers as they both try to shift their work earlier.
Undersecretary of Defense for Acquisition, Technology and Logistics Frank Kendall said he agreed with the idea of moving testing earlier in program development.
“The point of testing is to gain data, to gain knowledge, to gain information, that you can use to help you make better decisions. The earlier you get that information, the more valuable it is–the better decisions you can make,” he said at the International Test and Evaluation Association’s annual conference in Arlington, Va.
However, he noted that developmental testing (DT) and operational testing have two very different roles in the process and require different environments, and therefore he doesn’t want to see operational testing–which comes at the end of the process–overlap with or cut short developmental testing.
“I think of developmental testers as part of the program manager’s team more than anything else,” he said, whereas operational testers are meant to be independent evaluators outside the development team.
“I do not want operational testing substituting for DT” because they serve such different purposes, Kendall added. In developmental testing, “you want to control the environment, you want to control the situation in which you’re collecting data so you can separate out the correlation so you know what you’ve got. The relatively free-play environment of OT doesn’t get you the same kind of results DT does, from an engineering perspective.”
That said, he understands why Director of Operational Test and Evaluation J. Michael Gilmore wants to see his testing done earlier too.
“The thing that I have worked [on] with Mike Gilmore is that he feels he is discovering things in OT that should have been discovered earlier and corrected earlier. I think there’s something to that,” Kendall said. But, he added, “I have seen some attempts, and to a degree I’ve pushed back on, to extend OT earlier.” Kendall said he opposes those efforts because the point of operational testing is to be a final check before going into production.
The gray area, though, is operational assessments, which are not official operational test and evaluation events but serve as an early check on a system’s performance in realistic environments with actual warfighters. Kendall said program managers and others in the acquisition community would have to use their best judgment about devoting resources to operational assessments, based on what they expect to gain from these events.
Ultimately, Kendall said, “I think there’s more work to be done there, there is some tension, but I think it just requires cooperation” among himself, Gilmore, and Deputy Assistant Secretary of Defense for Developmental Test and Evaluation David Brown.
Brown, who spoke at the same conference, said his idea of shifting left means identifying what information a program manager needs earlier in a program and constructing a test plan to meet those needs, rather than saving all the testing for post-development. Interoperability, cybersecurity, mission context and more are all requirements that cannot be fixed at the end if a test fails; rather, they need to be addressed up front and measured with developmental testing to ensure a program doesn’t progress too far without knowing it will be secure, or interoperable, or mission-relevant.
“You can’t test these features in at the end: you have to build them in with well-informed systems engineering, and the well-informed part comes from test and evaluation,” Brown said.