The Pentagon’s top weapons tester said the Knifefish unmanned undersea vehicle (UUV) testing environment was too similar to how developers trained the system’s classification algorithm.
While testing was adequate under the scope of test objectives, “the testing occurred in areas and environments similar to those in which the system developers trained and tuned Knifefish’s PMA classification algorithm. Therefore, DOT&E is unable to assess how well the system will perform in operational environments that are new to the system,” Director of Operational Test and Evaluation Robert Behler’s office said in an annual report on Pentagon weapon systems.
This is the first time DOT&E included Knifefish in its annual report.
The unmanned autonomous Knifefish system includes two UUVs to conduct mine countermeasures (MCM) operations in shallow water areas. It is deployed off a Littoral Combat Ship (LCS) or vessel of opportunity. The system uses machine-learning algorithms to process sensor data via a post-mission analysis (PMA) station when it surfaces for communications.
DOT&E underscored the test targets included a limited number of mine variants also used in system development.
“Testing did not provide data on the system’s capability to detect other mine variants or its capability to distinguish mines from non-mine, mine-like bottom objects, and clutter.”
The report said these test limitations make DOT&E unable to fully assess Knifefish’s ability to “detect, classify, and identify buried mines as a function of burial depths.”
The report said while Knifefish exceeded the operational availability threshold it did not meet the Navy’s reliability threshold “due to operational mission failures.”
It explained in two sorties hardware failures terminated the sorties. “During PMA, hardware and software faults delayed completion of sensor data analysis,” the report said.
DOT&E’s assessment did not include an evaluation of Knifefish’s cybersecurity since the system, software and interfaces are still in development.
The report recommended the Navy complete system upgrades and conduct further testing to characterize Knifefish performance “in operational environments in which the system capability has not previously been assessed.”
Other specific recommendations are only available in the classified version of this report.
DOT&E said based on the operational assessment the Navy plans to incrementally upgrade and test Knifefish capability to meet operational needs before IOT&E and fleet introduction.
The Navy finished the operational assessment of Knifefish’s performance to detect, classify, and identify moored mines, unburied mines, and mines buried in the ocean bottom by the entrance of Boston Harbor in May 2019. That was in accordance with a DOT&E-approved test plan.
The assessment covered 12 missions in two designated UUV operating areas using moored and bottom mine targets accredited as foreign mine surrogates by the Navy Operational Test and Evaluation Force. Knifefish operators completed six missions in each UUV operating area to test the system’s ability to detect, classify, and identify mines in two operational environments.