Illustration by Pop Chart Lab, popchartlab.com.
On the afternoon of February 15, 2011, Jaime Zapata, a 32-year-old special agent with US Immigration and Customs Enforcement, was shot to death by members of a drug cartel as he drove along a four-lane highway in Mexico. Zapata’s partner, 38-year-old Victor Avila, who survived the attack, later said that as many as 15 gunmen opened fire with automatic rifles, even though Zapata had identified himself as a diplomat and the armored SUV in which the pair was riding bore license plates identifying it as an official vehicle.
Back in Washington, senior administration officials wanted revenge for what they saw as the deliberate killing of a federal agent and the attempted murder of another. Michele Leonhart, head of the Drug Enforcement Administration and a veteran of the drug wars, asked, “What can we do to make an immediate impact against the cartels, to send a message?” recalls Derek Maltz, the special agent in charge of the DEA’s Special Operations Division. “We decided to do a law-enforcement strike,” Maltz says.
To conduct what became known as Operation Fallen Hero, investigators turned to a little-known Silicon Valley software company called Palantir Technologies. Palantir’s expertise is in finding connections among people, places, and events in large repositories of electronic data. Federal agents had amassed a trove of reporting on the drug cartels, their members, their funding mechanisms and smuggling routes. They had dossiers, reports from informants, surveillance images, intercepted electronic communications, footage from drone aircraft. But investigators lacked a way to assemble and share all that intelligence with one another and to quickly find leads buried in mountains of information.
Investigators with Zapata’s agency bought Palantir’s software, plugged it into their databases, and used it to track down members of the cartel. The results were astonishing.
Palantir helped identify connections among key individuals and organizations. Officials later reported that this kind of painstaking detective work—reading reports, piecing together clues, drawing links among people—would have taken months without technological assistance. With the help of Palantir, large amounts of data from disparate sources were analyzed within days.
Law-enforcement officers across the United States, Mexico, and South America confiscated 467 kilograms of cocaine, 64 pounds of methamphetamine, and 282 weapons and arrested 676 people—including the cartel member suspected of killing Zapata.
Officials were so impressed with Palantir’s software that seven months later they bought licenses for 1,150 investigators and analysts across the country. The total price, including training, was $7.5 million a year. The government chose not to seek a bid from some of Palantir’s competitors because, officials said, analysts had already tried three products and each “failed to provide the necessary comprehensive solution on missions where our agents risk life and limb.”
As far as Washington was concerned, only Palantir would do.
Such an endorsement would be remarkable if it were unique. But over the past three years, Palantir, whose Washington office in Tysons Corner is just six miles from the CIA’s headquarters, has become a darling of the US law-enforcement and national-security establishment. Other agencies now use Palantir for some variation on the challenge that bedeviled analysts in Operation Fallen Hero—how to organize and catalog intimidating amounts of data and then find meaningful insights that humans alone usually can’t.
Palantir has sold its software to the CIA, the military’s Special Command, and the Marine Corps, which use it to help track down terrorists. The FBI, the Defense Intelligence Agency, the National Counterterrorism Center, and the Department of Homeland Security are customers. The director of the National Security Agency has said Palantir’s software could help the agency “see” into cyberspace to defend against hackers and spies attempting to breach government computer networks. The board that’s set up to oversee federal stimulus spending uses the software to spot fraud. The Los Angeles Police Department uses Palantir. So does the New York Police Department, whose intelligence-and-counterterrorism unit rivals the sophistication of the FBI and the CIA.
Palantir’s success in Washington—the likes of which is unheard of for a Silicon Valley start-up—suggests a question: If the US intelligence community had been using the company’s software before 9/11, could it have stopped the attacks?
The answer is a qualified yes. Put aside all the obstacles to preemption that a computer program couldn’t have removed: There were legal restrictions on how the FBI and CIA worked cases together, regulations that limited investigators’ access to information on American citizens, and a set of bureaucratic incentives that favored territoriality over cooperation. But if lawmakers and policymakers overcame those hurdles and if they were no longer an impediment to intragovernmental harmony, then yes, a system designed to locate and then share important clues could give analysts a strong chance of spotting the nascent signals of the 9/11 plot. This may have then led investigators in the direction of some of its architects, whom they could arrest or otherwise take out of commission. If you connect the dots this way, Palantir might have helped save the day.
It’s understandable why such a barrier-busting technology wasn’t being used on September 10, 2001. But the lack of information-sharing that led to the attacks was almost immediately obvious. So why didn’t the government get to work right away on a technological solution to the problem?
There are a few answers to that, starting with this: The government did try to solve the problem, but it failed.
Next: “They provide a product that actually does what they say it can do, and at the price and on the schedule they promise”
In January 2002, the Defense Advanced Research Projects Agency, the military brain trust that invented the Internet and stealth technology, launched a research initiative to develop counterterrorism technologies. Dubbed Total Information Awareness (TIA), its goals were twofold: First, use advanced data-searching-and-integrating software to help the government get a better handle on the massive databases it had already set up but hadn’t used to their full potential. Second, use the same technology to search private data stores that might also hold valuable clues, such as airline reservation systems, credit-card records, and phone and e-mail logs.
The animating hypothesis behind TIA was that technology could help analysts identify potential terrorists amid a sea of mostly innocent people. This was an idea for which the technology didn’t exist. But such a powerful system would also pose a threat to civil liberties, so TIA would encrypt information about American citizens and place legal controls on analysts using the system.
The man who ended up in charge of the effort, retired admiral John Poindexter, had been thinking about the problem since the early 1980s, when he was on the staff of the White House National Security Council. He had been working for nearly two decades under the presumption that computers could assist counterterrorism analysts.
TIA was shuttered in 2003 amid a national debate over its goals and its director’s past—Poindexter was an architect of the Iran-Contra affair and prosecuted for his involvement. But his resignation didn’t diminish the government’s appetite for his ideas. So it should come as no surprise that Palantir’s founders sought Poindexter’s counsel.
One day not long after Palantir was launched in 2004, Poindexter received a call from his friend Richard Perle, ex-chair of the Defense Policy Board, asking if he’d come to Perle’s house for a meeting with two entrepreneurs he knew from Palo Alto—Alex Karp, Palantir’s CEO, and his fellow cofounder Peter Thiel, a billionaire venture capitalist who’d helped start PayPal and was an early investor in Facebook.
The parallels between moving money and fighting al-Qaeda may not be immediately obvious.
Karp, a self-described progressive, knew Thiel, a prominent libertarian, from their days at Stanford Law School. After 9/11, Karp had reconnected with Thiel, who had the idea that Silicon Valley should do something to improve national security and secure civil liberties. They and three other Palantir cofounders wanted to use PayPal’s fraud-detection technology as the model for a new counterterrorism software. Karp wasn’t a natural choice for CEO—although he had spent most of his career helping other people start companies, he hadn’t run them. But the partners revered him for his intellect and business savvy.
The parallels between moving money and fighting al-Qaeda may not be immediately obvious. PayPal, founded in 1998, became famous for letting people electronically pay for goods and services they bought online. But that’s not why it became a successful business. First, engineers had to figure out a way to keep criminals from stealing PayPal customers’ money.
Credit-card companies and banks had written algorithms to detect patterns of fraud and theft, but the algorithms became useless as soon as criminals figured out what triggered an alarm and then adjusted their tactics. PayPal wanted to pit fraud experts against the scammers—an electronic version of cops and robbers. Engineers designed a software program that let human fraud experts—many were former law-enforcement officers—quickly sift through transaction data and start asking questions. Did a transaction occur between two parties in the US? That was probably safe. Two parties in Russia? A bit more suspicious. Did someone in the United States transfer money to someone in Russia? A red flag.
By giving the analysts a way to look into the transaction network and map out connections among the suspected criminals, they could draw conclusions based on their years of expertise. That approach made PayPal the world’s most trusted system for online payments. In 2002, the company was acquired by eBay for $1.5 billion.
“The bread and butter of PayPal was to look at a transaction and to know if it came from a bad IP address,” says Bob McGrew, Palantir’s director of engineering, referring to a computer’s distinct address on the Internet. That same approach became the heart of Palantir, which Karp says is an “attribution” software, meaning it’s used to find the people behind pieces of data.
Palantir wanted to beat terrorists the way PayPal beat the Russian mafia. And that approach bore some striking similarities to the one Poindexter took with TIA.
Thiel, who’s something of a policy dilettante, also thought the idea could make a lot of money. He has said that Palantir will be the next Facebook; a top executive at JPMorgan Chase says it’s poised to do for information locked inside organizations what Google did for information on the Web. Palantir is considered one of the most valuable start-ups in America, with an estimated market capitalization as high as $2.5 billion.
“I told them I thought they had an interesting idea,” Poindexter says. He looked at an early mockup of the product and thought it had “an absolutely beautiful user interface.”
With little training, Palantir gives users the ability to turn text documents into charts and graphs and to layer intelligence reports onto maps or into interactive timelines.
Karp says Poindexter was one of many experts whom Palantir’s founders consulted in their early days. Many of them opened doors that usually remain closed to small companies with no experience in Washington.
In a short time, Palantir has assembled a legion of advocates from the most influential strata of government. Karp counts former CIA director George Tenet as a friend as well as Tenet’s employer, Herb Allen, who runs the enigmatic investment bank Allen & Co., a Palantir investor. Michael Leiter, former National Counterterrorism Center director, is a senior counselor to Palantir. Another top adviser, Bryan Cunningham, was a CIA intelligence officer and a senior staffer to former national-security adviser Condoleezza Rice.
Palantir has hired powerful lobbyists, including onetime senator John Breaux and ex–Senate majority leader Trent Lott, both of whom now work for Patton Boggs. And Palantir’s advocates in Congress include representatives Buck McKeon and Adam Smith, respectively the chairman and ranking member of the House Armed Services Committee, as well as Norm Dicks, the ranking member of the House Appropriations Committee, all of whom have urged the Defense Department to consider buying more of Palantir’s software.
“They provide a product that actually does what they say it can do, and at the price and on the schedule they promise,” Leiter says. “Regrettably, that is often rare among many government contractors.” This fact also helps explain how a start-up launched by five Washington outsiders found such remarkable success here. And it provides another explanation for why the government didn’t build something like Palantir immediately after 9/11. The fact is, the way Washington works made a Palantir-like success practically impossible. So Palantir decided not to play by Washington’s rules.
Next: “We believed it would work. No one else did.”
Big companies have dominated defense contracting for more than 60 years, but Palantir is an insurgent. Its founders don’t obey the hierarchical order of the defense-industrial complex, which they think has failed the government and ripped off taxpayers.
They have a point. The files of inspectors general overflow with accounts of how big-name companies over-promised and under-delivered on multibillion-dollar technology “modernization” projects meant to cure the intelligence failures that led to the 9/11 attacks. The FBI and the National Security Agency, both of which wasted billions on custom-built computer systems, offer some of the most dramatic examples of high-priced failure. But the IRS, the Interior Department, and the Pentagon, among others, can claim a place in that undistinguished company.
During a speech in London, where geographic and cultural distance may have put Karp in a more candid mood, he gave a scathing assessment of how big defense contractors—his competition—traditionally do business in Washington.
“People charge you every time you want something fixed, every time you want to build something, before it’s built, after it’s built,” he told the London crowd. “You’re paying by the hour. You’re paying 20 people. You think you’ve hired five.”
Palantir wanted to upend the system. But before Palantir could get to the point that it could offend Washington’s sensibilities, it had to get out of Silicon Valley.
Prospective investors were skeptical. Venture-capital firms were looking for the next Facebook or Google, but most presumed it would come in the form of a consumer technology, probably a new social-media Web site. No one wanted to back an expensive software platform for large organizations. The Palantir founders met with several venture-capital firms, and all turned them down. “We believed it would work,” Karp says. “No one else did.”
Potential investors were also leery of Washington. “The government was unpopular in Silicon Valley,” Karp says.
“Everyone told us we should stay away from Washington because it was corrupt and we didn’t know how to play golf with senators,” says Joe Lonsdale, another Palantir cofounder and former editor-in-chief of the Stanford Review, the libertarian publication Thiel founded as an undergraduate. He recalls that one prospective backer ridiculed Karp for having a PhD in philosophy, not a technical field. Says Lonsdale: “It was kind of a horrible experience.”
According to Karp, at the end of one failed pitch meeting, an investor who’d just turned him down said there was a group he should talk to “that does this kind of thing.” The group was In-Q-Tel, the venture-capital arm of the CIA, which was set up in 1999 to bypass the cumbersome government procurement process and to fund technologies that might be useful to the spy agencies.
In-Q-Tel is a gatekeeper between small technology companies, which usually know Washington only by its location on a map, and the secretive regions of the intelligence community. “If you’re crazy enough to want to work with government, they’re the right people to talk to,” Stephen Cohen, one of the cofounders, told an audience at the Startup School conference held at Stanford in October. Palantir got a meeting with In-Q-Tel’s CEO, Gilman Louie, a former computer-game designer.
“If you guys aren’t able to help us with counterterrorism, you have a real future in video games.”