The final results from the most recent biometric technology evaluation hosted by homeland security researchers are still being crunched, but the general trends are that vendors have advanced their products to identify people whose faces are partially masked, regardless of demographics, but more work can be done to improve the camera systems that automatically acquire facial images, which are then automatically checked against a database using a matching algorithm, according to the lead testing official.
“I would say on average industry got better, the median system from last year to this year looks like it got better,” Arun Vemury, director of the Biometric and Identity Technology Center at Department of Homeland Security Science and Technology Directorate, told Defense Daily in a virtual interview last Friday. “So, we are seeing general improvements from industry at this type of problem, so I think that’s a good sign.”
The 2021 Biometric Technology Rally was held over three weeks last fall to examine the ability of face, and face and iris capture systems, integrated with biometric matching algorithms to identify people even though they were wearing masks to protect against the spread of COVID-19. The 2020 rally examined these types of technologies using people with and without masks but at that time, industry had had only about two months to tweak their technologies to confront the challenges of masked individuals.
The latest evaluation gave vendors more time to improve their systems to recognize people wearing masks, Vemury said.
The other key aspect of the 2021 rally is that the final analysis will break out the performance by demographics, Vemury said. The reason for this is because in the earlier evaluation “it appeared that the errors were not uniformly distributed,” he said. “It didn’t happen as much for all people of all gender and race backgrounds. It happened more with some groups than for others.”
This data will be shared with vendors so that they can “hopefully make the performance more fair in general,” he said.
Vemury pointed out that facial recognition performance comes down to how systems, which include the camera and its software, and the matching algorithms, are configured. He noted that the facial vendor recognition tests performed by the National Institute of Standards and Technology (NIST) evaluate the matching algorithms, not the image capture devices.
What the rallies have shown is that sometimes the camera systems don’t recognize that there’s a face in front of them, so they don’t take a picture.
“It’s not just a thing about facial recognition, it’s about particular configurations,” Vemury said. “That being said, one thing why we do this test versus relying upon other algorithm evaluations like the NIST tests, is those tests don’t take into account the influence of the camera and the collection process and we still see the bulk of errors happening with the camera. Where people are interacting with the camera and either the camera doesn’t take a photo at all or the camera takes an image that is harder for the matching algorithms to match.”
Vemury said he is trying to “bring more attention” to why cameras fail to take a picture.
“We think this is something that industry and others can help make improvements to honestly close some of these errors because this is the primary source of error in a lot of biometric systems, these capture errors, not the matching errors,” he said.
In the 2020 rally, also held at S&T’s Maryland Test Facility, the median percentage of when cameras didn’t take a photo was 14 percent when people were masked. That number was 6 percent for unmasked individuals, which Vemury still said, “It’s not trivial.”
In 2021, Vemury said the median numbers have improved.
“We saw both an improvement with that camera error issue and I think we’re seeing an improvement with the matching algorithms as well,” he said.
The best facial matching algorithms correctly identified 100 percent of the test subjects without a mask in 2020, and 96 percent who were wearing masks.
The rally in 2020 involved 582 volunteers and in 2021 just over 600, Vemury said. The volunteers represented multiple nations, races, genders and age groups. He also said that about a half-dozen face and face-iris capture systems were used and 10 matching algorithms in the most recent rally.
“Generally speaking, because of the population size we’re working with, we’re focusing mostly on demographic characteristics like skin tone, the race that people associate with as well as gender,” he said. “What we consistently seem to be finding is that skin tone is a bigger factor than race. Race actually has quite a bit of variability but skin tone is a better explainer of some of the differences in performance.”
The 2021 test results will be finalized in March or April, he said.