FBI, Pentagon helped research facial recognition for street cameras, drones

The FBI and the Defense Department were actively involved in research and development of facial recognition software that they hoped could be used to identify people from video footage captured by street cameras and flying drones, according to thousands of pages of internal documents that provide new details about the government’s ambitions to build out a powerful tool for advanced surveillance.

The documents, revealed in response to an ongoing Freedom of Information Act lawsuit the American Civil Liberties Union filed against the FBI, show how closely FBI and Defense officials worked with academic researchers to refine artificial-intelligence techniques that could help in the identification or tracking of Americans without their awareness or consent.

Many of the records relate to the Janus program, a project funded by the Intelligence Advanced Research Projects Agency, or IARPA, the high-level research arm of the U.S. intelligence community modeled after the Pentagon’s Defense Advanced Research Projects Agency, known as DARPA.

Program leaders worked with FBI scientists and some of the nation’s leading computer-vision experts to design and test software that would quickly and accurately process the “truly unconstrained face imagery” recorded by surveillance cameras in public places, including subway stations and street corners, according to the documents, which the ACLU shared with The Washington Post.

In a 2019 presentation, an IARPA program manager said the goal had been to “dramatically improve” the power and performance of facial recognition systems, with “scaling to support millions of subjects” and the ability to quickly identify faces from partially obstructed angles. One version of the system was trained for “Face ID … at target distances” of more than a half-mile.

To refine the system’s capabilities, researchers staged a data-gathering test in 2017, paying dozens of volunteers to simulate real-world scenarios at a Defense Department training facility made to resemble a hospital, a subway station, an outdoor marketplace and a school, the documents show. The test yielded thousands of surveillance videos and images, some of which were captured by a drone.

The improved facial recognition system was ultimately folded into a search tool, called Horus, and made available to the Pentagon’s Combating Terrorism Technical Support Office, which helps provide military technologies to civilian police forces, the documents show.

The Horus tool has since been offered for use to at least six federal agencies, and their feedback is “continuing to be used to refine the tool,” Department of Homeland Security officials said last year.

The internal emails, presentations and other records offer an unmatched look at the way the nation’s top law enforcement agency and military have aggressively pursued a technology that could be used to undermine Americans’ privacy and already has a counterpart in mass surveillance systems in London, Moscow and across China.

The documents also show that federal officials were more closely involved in the technology’s development than was previously known, even as three states and more than a dozen cities passed laws banning or restricting the technology’s use by local police.

No federal laws regulate how facial recognition systems can be used. Sen. Edward J. Markey (D-Mass.) said Tuesday he intends this year to push again for a bill, first introduced in 2020, that would restrict how federal agencies can tap facial recognition and other biometric search techniques.

“Americans’ ability to navigate our communities without constant tracking and surveillance is being chipped away at an alarming pace,” Markey said in a statement to The Post. “We cannot stand by as the tentacles of the surveillance state dig deeper into our private lives, treating every one of us like suspects in an unbridled investigation that undermines our rights and freedom.”

The tool’s use in domestic mass surveillance would be a “nightmare scenario,” said Nathan Wessler, a deputy director at the ACLU. “It could give the government the ability to pervasively track as many people as they want for as long as they want. There’s no good outcome for that in a democratic society.”

The FBI said in a statement it is “committed to responsible use of facial recognition technology ensuring it appropriately respects individuals’ privacy and civil liberties.” A Defense Department official acknowledged a request for comment but did not respond to a list of questions by the time of publication. An IARPA spokeswoman said the agency is focused on developing the technology rather than how it is applied.

The documents are as recent as 2019, when the ACLU requested and then sued for the records’ release, and they offer no detail on how the research is currently used or deployed. In the years since, facial recognition technology has become widely used by federal investigators and local police.

A Government Accountability Office audit in 2021 found that 20 federal agencies, including the U.S. Postal Service and the Fish and Wildlife Service, had used facial recognition in some capacity, though most of the agencies did “not have awareness” of which tools employees were using and had “therefore not fully assessed the potential risks.”

The documents offer an intimate look at the kinds of daily technical decisions researchers have made in recent years to capitalize not just on breakthroughs in artificial intelligence and computer imaging but also on the fast-growing trove of data related to Americans’ personal lives.

In some emails, FBI scientists talk with academic researchers and technical specialists about the facial recognition tool in close detail, including discussing how the system processed information about a photo of a face using attributes such as “face rectangle x start coordinate,” “pitch of the head” and “probability of being male.”

And in a presentation given at a forensic-science luncheon in Baltimore in 2019, an FBI senior scientist said that some of the “biggest enablers of better face recognition” included “cellphones with cameras” and “social media.”

Named for the two-faced Roman god of beginnings and gateways, Janus launched in 2014 with the goal of “radically expanding the scenarios in which automated face recognition can establish identity,” the documents show.

At that time, federal investigators seeking to use facial recognition were limited largely to databases of “constrained” photos from passports or driver’s licenses to help identify suspects, victims and witnesses of people recorded near the scene of a crime, using slow and imprecise algorithms that tended to “severely underutilize and under-exploit all available face information in a video,” as one research filing said.

Research teams were tasked with developing new algorithms that could help investigators tap into a new generation of surveillance footage, allowing for instant identification and the ability to track the same person’s face across multiple videos and camera angles. The goal was to “change video from an impediment to an advantage,” one document states.

Erik Learned-Miller, a University of Massachusetts at Amherst professor who was part of one of the research teams, said federal officials involved in the program were careful to make “a distinction between things they were willing to do for American citizens and capabilities they wanted to develop for use in the rest of the world.”

The research, he said, was aimed at improving the performance of a technology that was already seeing increasing use by law enforcement at the local, state and federal levels. And the mission was hard to refuse: An FBI official at one point spoke to the researchers about how the system would be used to identify the perpetrators in videos of child sexual abuse.

But the research’s lofty goals did at times leave him wondering how the work might be put to broader long-term use. “The question always in the back of my mind was: What does the intelligence community really want to do with this stuff?” he said.

The Janus project officially ended in 2020, though its work was then folded into the web-based interface Horus — named for another deity, the falcon-headed Egyptian god of the sky. IARPA said in public filings that the Janus program had helped advance “virtually every aspect of fundamental face recognition research” and led to algorithms that were “twice as accurate as the most widely used government-off-the-shelf systems.”

The Janus research marked only a fraction of the FBI’s sizable technical interest in facial recognition. The agency’s Interstate Photo System uses a facial recognition search tool, available to state and local police, that can scan through tens of millions of jail booking photos as well as images of people’s scars and tattoos.

The photo system is part of a broader FBI biometric database, called Next Generation Identification, that contains the fingerprints, palm prints, face photos and eye patterns collected from millions of people applying for citizenship, getting booked into jail or requesting job background checks.

Among the documents revealed by the ACLU, one Interstate Photo System how-to guide tells investigators that they must use it for “investigative purposes only” but that “it is the responsibility of the user agency to develop appropriate usage policies.”

The documents also include forms that local police officers can use to submit a photo to the FBI’s Facial Analysis, Comparison and Evaluation (FACE) Services Unit, which then runs it through a facial recognition search and returns possible matches. Officers can use the form to request the photos also be run through a biometric database of foreign citizens and combatants run by the Defense Department and the passport and visa photos managed by the State Department, the documents show.

In 2019, government auditors said the FBI had access to more than 640 million face photos and that the FACE unit had run more than 390,000 facial recognition searches over the previous eight years.

Beyond government-sponsored research, federal agencies have also paid for access to private facial recognition systems. The FBI signed a $120,000 contract earlier this year with Clearview AI, maker of a facial recognition tool that uses face photos taken without consent from across social media and the public internet. FBI officials said in the contract they were paying for “a search engine of publicly available images … to be used in ways that ultimately reduce crime.”

The Defense Department last year also awarded a nearly $730,000 contract to the video firm RealNetworks for facial recognition software that could be used on autonomous drones for “identification and intelligence-gathering” purposes, a contract shows.

Civil liberties advocates have warned that facial recognition research could hasten the technology’s rollout for real-time public surveillance in the United States. Facial recognition systems have also been shown in research to perform worse when assessing the faces of people of color, and they have been blamed for several recent cases in which Black men were wrongfully arrested for crimes they did not commit.

Proposals to use facial recognition for mass-surveillance efforts in the United States, as they are used in China, have faced public backlash. FBI Director Christopher A. Wray said in January that he was concerned about the Chinese government’s development of similar technologies, saying, “AI is a classic example of a technology where I have the same reaction every time. I think, ‘Wow, we can do that?’ And then I think, ‘Oh, God, they can do that.’”

Clare Garvie, an attorney with the National Association of Criminal Defense Lawyers who has studied facial recognition, said the technology has become “a ubiquitous, routine forensic investigative technique” in the years since the Janus program began.

But the lack of transparency into how widely the technology is used, along with ongoing questions of how reliable it is when used to identify criminal suspects, raises the risk of dangerous misuse.

“It’s one thing for a company or research entity to say the technology discretely performs in [some] capacity in a lab. It is quite another to say it’s ready to be used from drones or on low-quality surveillance images for the purpose of making arrests,” Garvie said. “We’re essentially beta-testing technology on real people with real-world consequences.”