This fall the Department of Homeland Security will conduct its annual Biometric Technology Rally with a continued emphasis on how well the technology works when travelers are wearing face masks and for the first-time program officials plan to include test results based on the demographics of the volunteers when an analysis of the evaluation is published.
The 2020 Biometric Tech Rally showed that there are camera systems and facial matching algorithms that work well despite volunteers wearing masks. However, the DHS Science and Technology Directorate originally planned to conduct an evaluation where people would be processed one at a time and where small groups of people “interacted with the technologies,” Arun Vemury, director of S&T’s Biometric and Identity Technology Center, told HSR in a recent interview.
But, he said, with the COVID-19 pandemic, S&T “had to pivot within a couple of months” to change its plan and test the technologies against people wearing face masks, which people began wearing to protect themselves and others from the threat of the virus. Vemury added that the change in the rally’s operating concept also forced the participating vendors to shift quickly as well.
“They had to figure out how to redesign their tools to work with people who were wearing and not wearing face masks,” he said. “So, last year was quick and dirty. We had to figure out what was working, what was not working. The companies had to adapt and innovate quickly.”
Despite the limited time, the result was that some of the companies performed very well and the best facial matching algorithms correctly matched individuals wearing masks 96 percent of the time (HSR, Jan. 19, 2021).
Now, given that a year will have passed between last year’s rally and the 2021 evaluation, S&T is looking to see how much vendors have improved their technologies against the challenges of people wearing face masks.
“We saw surprisingly good results from a few companies but we had a lot of good companies who probably didn’t do as well as they would have liked,” Vemury said. “And we’re looking to see quite honestly to see how much better they’re going to get given the months that have passed from the previous rally to the new one right now…We’re looking for improvements where hopefully fewer systems don’t take photos of people wearing faces masks and that the matching algorithms continue to get better.”
Demographic Data
Another feature of the upcoming rally will be publishing results of identification rates based on the demographics of the volunteers that will participate in the rally at the test facility in Maryland. There are concerns that some biometric technologies, particularly the face matching algorithms, do better or worse depending on the race, ethnicity, age and gender of the populations the systems are applied to.
In the past, S&T has collected relevant demographic data from its rallies to go back and analyze but hasn’t published it, Vemury said.
“Now, given a lot of the discussions, we want to make sure we can provide that visibility to others about how these things work, because some of these technologies work very, very well, some may not work as well, and we want to help make sure that that information is available to people who have to make decisions about which systems to buy or procure or at a minimum inform planning,” he said. “The whole intent here with the rally is obviously to help provide information to federal decisionmakers but to help provide feedback industry so that they can make their technologies better.”
For companies that do well regardless of demographics, this will give them a chance to “take credit,” Vemury said.
In the 2020 rally, S&T had a database of 500 people and ran 582 people through the system. For the evaluations, the technologies had to tell program officials someone was in the database and who they were, allowing for testing of false positives and false negatives. The 2021 rally will be “of similar size and scale,” Vemury said.
For the operating concept, the focus will be on high throughput and the goal is to have people processed within four seconds, which is triggered as soon as a person walks into a six by eight square foot space, has their photo taken and a match result issued, and then leaves the space.
“We’re talking about the pace at which people walk,” Vemury said. “So, it’s almost like walking without taking a break or pause.” He added that some of the technologies are “really good” about processing matches rapidly.
Face acquisition and matching are required for the fall rally. Vendors can also bring iris and fingerprint capture and matching technologies but need to be able to hit the time goal, Vemury said. Iris capture and fingerprint take “a little bit longer” than the time it takes to collect a good facial image for matching purposes, usually putting them “outside the time budget” allotted for the study, he said.
“In this case, this scenario test is really focused on high-throughput, really easy to use, very easy to understand process and so far, face seems to be the best performing technology at this time,” he said.
Vemury hopes to begin the study in late September. Depending on whether the pandemic continues to subside or worse, data collection could take a week or three, he said.