We're learning more about how that enormous cache of telephone metadata at the National Security Agency is actually used. According to two sources in the intelligence community who have worked with the system, it's one of many tools available to analysts working on terrorism investigations or providing intelligence for military forces overseas.
One former defense intelligence employee describes it this way:
The NSA makes a list of names and/or phone numbers available to analysts who are cleared to use the meta database. These names and/or numbers have been obtained by NSA through other collection programs, presumably legal ones. The analysts input those names and/or numbers to the meta database, which will then show any connections to phone numbers in it.
The meta database itself doesn't contain any names--it is only phone numbers. If there's a number that's based in the United States, the analyst only sees an "X" mark. If he wants to see the number underneath that X, he has to get clearance from higher authority--in this source's experience that was the general counsel of the organization where he worked.
The source also said the tool wasn't particularly useful. It's there to help analysts better understand the links between potential terrorists, and to help identify them. But the analyst said that searching for numbers and names on Google often led to better results.
An intelligence official who has used the meta dabtase confirmed the description of how it works. But he said the presence of so many "innocent" numbers in the system posed a challenge. Analysts have to weed through them to find only the numbers they're allowed to see without permission from a higher authority.
The meta database is one of dozens of different systems or intelligence streams available across the intelligence agencies. From the sources' descriptions, it sounds relatively mundane compared to the other tools that are available.
However, one of those other tools, which was revealed yesterday by the Guardian and the Washington Post, called PRISM, appears far more secretive and less widely used. Neither of these source had ever heard of it. The former defense intelligence employee expressed alarm that, according to reports, the system gives the NSA direct access to the central servers of some of the country's biggest Internet companies, including Facebook and Yahoo!, and then lets analysts obtain e-mails, video and audio files, photographs, and documents.
This individual said he couldn't explain how, based on his training and experience, the PRISM system complies with the law. It doesn't seem to be discriminate enough in separating US person's content--such as their e-mails--from those of foreigners. The government almost always needs a warrant to look at content.
Reportedly, PRISM sweeps up the information of US persons when analysts tap into those central servers. They're instructed to document these "incidental collections," but are told, according to a training manual reviewed by the Post, that "it's nothing to worry about" if Americans are caught up in the stream.
The intelligence official said that based on reports, the PRISM system would have to be collecting massive amounts of information, and that NSA was likely the only agency with the computing power and the storage space to handle it all. The agency has been running out of electronic storage at its Ft. Meade, Md., headquarters and has built a new 1-million square foot data center in the Utah desert.
Multiple officials are now confirming that the National Security Agency's practice of collecting all telephone metadata from Verizon, as first reported by the Guardian, is part of a program that has been active for years. A US intelligence official tells me that orders of the kind delivered to Verizon in April are routine. Sen. Dianne Feinstein said today that the collection of metadata from phone companies is a seven-year-old practice. And an unnamed source told the Washington Post that the order appears to be similar to one first issued by the Foreign Intelligence Surveillance Court in 2006, and that it is “reissued routinely every 90 days” and not related to any particular government investigation.
Here’s what else we know so far about this massive intelligence collection program, a few things we might infer, and some big unanswered questions.
What is the government doing with all this phone metadata?
According to a senior administration official, “Information of the sort described in the Guardian article has been a critical tool in protecting the nation from terrorist threats to the United States, as it allows counterterrorism personnel to discover whether known or suspected terrorists have been in contact with other persons who may be engaged in terrorist activities, particularly people located inside the United States.”
This is a description of standard link analysis. Say the government obtains the phone number for a suspected terrorist. It then runs that number against the huge metadatabase. If there’s a match, presumably the government then obtains some other authority to find out who the number in the metadatabase belongs to; according to the court order, and the administration official, the metadata does not contain the names of phone subscribers. It’s just phone numbers, lengths of calls, and other associated data that’s not considered “content.”
What can you learn with metadata but no content?
A lot. In fact, telephone metadata can be more useful than the words spoken on the phone call. Starting with just one target’s phone number, analysts construct a social network. They can see who the target talks to most often. They can discern if he’s trying to obscure who he knows in the way he makes a call; the target calls one number, say, hangs up, and then within second someone calls the target from a different number. With metadata, you can also determine someone's location, both through physical landlines or, more often, by collecting cell phone tower data to locate and track him. Metadata is also useful for trying to track suspects that use multiple phones or disposable phones. For more on how instructive metadata can be, read this.
Where is all that metadata being stored?
According to the court order, at the National Security Agency. The electronic spying agency is headquartered in Ft. Meade, Md. But it has been running out of digital storage space there, as well as electricity to keep all its systems up and running. The NSA has built a new facility in the Utah desert, called, appropriately, the Utah Data Center. And it recently broke ground on another facility at Ft. Meade.
How does that data get from the phone companies to the NSA?
We still know little about the physical infrastructure that transmits the metadata. But we do know, from the order, that Verizon is sending the information to the NSA “on an ongoing daily basis.” That’s an extraordinary amount of information considering it covers millions of customers making multiple calls a day. In simple terms, we’re talking about a lot of pipes and cables leading from Verizon locations—like switching stations—to NSA facilities. We know from a whistleblower at AT&T that surveillance equipment was set up at the company’s offices in San Francisco as part of the NSA’s efforts to monitor terrorists after the 9/11 attacks.
What else might the NSA or other government agencies be doing with this metadata?
As I wrote in my book, The Watchers, the NSA has long been interested in trying to find unknown threats in very big data sets. You’ll hear this called “data mining” or “pattern analysis.” This is fundamentally a different kind of analysis than what I described above where the government takes a known suspect’s phone number and looks for connections in the big metadatabase.
In pattern analysis, the NSA doesn’t know who the bad guy is. Analysts look at that huge body of information and try to establish patterns of activity that are associated with terrorist plotting. Or that they think are associated with terrorist plotting.
The NSA spent years developing very complicated software to do this, and met with decidedly mixed results. One such invention was a graphing program that plotted thousands upon thousands of pieces of information and looked for relationships among them. Critics called the system the BAG, which stood for “the big ass graph.” For data geeks, this was cutting edge stuff. But for investigators, or for intelligence officials who were trying to target terrorist overseas, it wasn’t very useful. It produced lots of potentially interesting connections, but no definitive answers as to who were the bad guys. As one former high-level CIA officer involved in the agency’s drone program told me, “I don’t need [a big graph]. I just need to know whose ass to put a Hellfire missile on.”
How big a database do you need to store all this metadata?
A very, very big one. And lots of them. That facility in Utah has 1 million square feet of storage space.
But just storing the data isn’t enough. The NSA wants a way to manipulate it and analyze it in close to real-time. Back in 2004, the agency began building “in-memory” databases, which were different than traditional databases that stored information on disks. In-memory was built entirely with RAM, which allows a computer to hold data in storage and make it ready for use at an instant. With disks, the computer has to physically go find the data, retrieve it, and then bring it into a program. If you’re trying to analyze entire telephone networks at once—and that is precisely what the NSA wanted to do—a disk-based system will be too slow. But the NSA’s in-memory databases could perform analytical tasks on huge data sets in just a few seconds.
The NSA poured oceans of telephone metadata into the in-memory systems in the hopes of building a real-time terrorist tracker. It was an unprecedented move for an organization of the NSA’s size, and it was extremely expensive.
That was 2004. The court orders issued to Verizon, we’re told, go back to as early as 2006. It appears that the NSA has had an uninterrupted stream of metadata for at least seven years. But the agency was getting access almost immediately after 9/11. That could mean there’s more than a decade’s worth of phone records stored at the NSA’s facilities.
With drones poised for takeoff in US airspace beginning in 2015, questions are mounting over how governments will use remotely-piloted eyes in the sky to monitor everything from traffic patterns to wastewater runoff.
One association that represents private forest landowners, and whose members include some of the country's largest owners of timberland, says the government has effectively admitted that it's using drones to "spy" on private owners in the name of preventing pollution. The Forest Landowners Association is now polling its members to ask "your thoughts on the government using satellite and drone technology to glean insight about your land."
The questions the group wants to answer show where future conflicts may erupt between government and private interests over the use of technology that, heretofore, has mainly been used to monitor enemies in combat.
1. Are you concerned that with advances in satellite and drone technology the government has the ability to take a closer look at your land?
2. What does it have a right to know?
3. Are there boundaries on your property rights?
4. Have you ever had an instance where you wondered how the government knew certain things about your property?
Scott Jones, the association's CEO, said his group is concerned about "our rights as private forest landowners" in light of Senate testimony in April by Bob Perciasepe, the acting administrator of the Environmental Protection Agency. Jones said that under questioning from lawmakers about the EPA's use of drones to monitor animal feedlots for signs of water pollution, the acting chief admitted that the agency "was utilizing drones to 'spy' on private lands."
At the hearing, Sen. Mike Johanns (R-Neb.) accused the EPA of using the remotely-piloted aircraft to monitor people, not just pollution. "You're flying at low altitudes, you're flying over law-abiding people who are trying to do everything they can to honor your rules and regulations and you're not coming down on the bad actors," Johanns said. "You're checking on everybody and it feels terrible. It feels like there's a federal agency out there spying and on American citizens."
Perciasepe rejected that characterization and said the drones allowed the EPA "a very efficient way for us to narrow where we go to on the ground, [to] talk to landowners about what they're doing."
But Sen. Roy Blunt (R-Mo.), who noted that his own parents were dairy farmers, wasn't persuaded. “You said it's not like you were spying on people. What term would you use?"
"We're looking for where there may be animals and their waste in the water. And so, we are not looking at people at all," Perciasepe replied.
"So you're spying on animals?" Blunt asked.
“Well, we're looking to see where we would send inspectors to see if there was a problem of water pollution. So I don’t know that the animals are what we're spying on. We're looking at the conditions that could be creating water-quality violations.”
Jones, the association CEO, called the agency's use of drones "heavy handed."
"We launched a poll to our membership, who own and operate more than 43 million acres of private forestland, so they could weigh in with their concerns, which we intend to share with lawmakers and the administration." The group's members include large companies such as Weyerhaeuser and Georgia-Pacific, as well as owners of small land tracts.
The EPA has attempted to mollify critics of its drone flights and says it wants to hear about objections. "I understand the perception that you're bringing up," Perciasepe told Johanns. "It's helpful for me to hear the intensity of it and I will bring that back."
A number of observers were perplexed by President Obama's grand-strategy speech yesterday at the National Defense University. Was it an apologia or an apology? Did the speech mark a hardening of counterterrorism policies or the beginning of their end? The President seemed to want to do both. He may end up satisfying no one.
Obama was at once on the side of some of his fiercest critics, particularly with regards to targeted killing. And yet he mounted what is surely the most full-throated defense to date by any president of the commander-in-chief's authority to order lethal drone strikes in the nation's self defense. He insisted that the war on terror, like all wars, must end--"That’s what history advises. That’s what our democracy demands." But there was little to hang onto in the way of commitments, timetables, or markers that will tell us how much closer we are to that end.
To the list of confusing and often contradictory propositions about the state of US national security, add these lines.
"Meanwhile, we strengthened our defenses--hardening targets, tightening transportation security, giving law enforcement new tools to prevent terror. Most of these changes were sound. Some caused inconvenience. But some, like expanded surveillance, raised difficult questions about the balance that we strike between our interests in security and our values of privacy."
This was one of the rare moments of understatement in the President's address. Decisions, often secret ones, to allow agencies of the federal government broader authorities to monitor the communications of Americans are among the most fateful actions undertaken in the war on terror. They have fundamentally transformed that balance of which the President spoke. And yet they have received precious little debate or reconsideration since the attacks of 9/11.
Perhaps it's because the questions raised by expanded surveillance are so difficult that the President spent practically no time answering them. His lengthy speech was devoted to profound matters, namely interrogation, detention, and targeted killing, but those are policies that have directly affected a vastly smaller number of Americans than has broadened monitoring of phone calls, e-mails, and other personal data of millions of people.
There was no talk of NSA warrantless wiretapping. No mention of the Patriot Act. No discussion of amendments to the Foreign Intelligence Surveillance Act--amendments that Obama once opposed as a presidential candidate, and that have been the subject of a Supreme Court challenge. Nor did the President speak a word about threats to the nation's cyber infrastructure, which has been a top action item for his national security team. Like expanded surveillance, strengthening the nation's cyber defenses through greater monitoring of the Internet, which is what the administration is calling for, is freighted with implications for privacy and civil liberties.
The only hint the President gave that he might be inclined to reexamine US surveillance policy came in a discussion of homegrown terror plots.
"[I]n the years to come, we will have to keep working hard to strike the appropriate balance between our need for security and preserving those freedoms that make us who we are. That means reviewing the authorities of law enforcement, so we can intercept new types of communication, but also build in privacy protections to prevent abuse.
"That means that--even after Boston--we do not deport someone or throw somebody in prison in the absence of evidence. That means putting careful constraints on the tools the government uses to protect sensitive information, such as the state secrets doctrine. And that means finally having a strong Privacy and Civil Liberties Board to review those issues where our counterterrorism efforts and our values may come into tension."
Building privacy protections into the fabric of surveillance systems is much easier said than done. Paradoxically, it is the government's deep-rooted obsession with secrecy and applying so many different levels of classification and control to intelligence that makes it hard to build a system that can uniformly protect personal information. Different agencies treat personal information according to different standards and regulations. There's really not a one-size-fits-all proposition, and the government doesn't have a viable plan to find one.
As for reviewing the authorities of law enforcement, there is no effort underway to repeal or curtail them. However, the administration is looking to expand the powers of law enforcement to monitor communications on the Internet.
And as for the Privacy and Civil Liberties Oversight Board, it's no secret this has been one of the slowest-going, and to many, one the least effective counterweights to a widening net of digital monitors.
The President's nod to "the balance that we strike" between security and privacy felt perfunctory. It seemed thrown in for appearances sake, like the also-confused passages about the need to prevent leaks of national security information without chilling journalists and their sources, which is precisely what the administration's clamp down on leaks is designed to do.
On surveillance, there was nothing in the speech that suggested a change of course, a ratcheting down, or a return to pre-wartime footing. Having written at length on the history of this subject, I'd already concluded that the surveillance state was here to stay. I suppose the President's speech makes it official. More or less.
More than a decade after the 9/11 terrorist attacks, US intelligence agencies are doing a much better job sharing information about terrorism and other national security threats. Their challenge now "is largely one of information overload," says a new report by the Congressional Research Service, published last week.
"Analysts now face the task of connecting disparate, minute data points buried within large volumes of intelligence traffic shared between different intelligence agencies. According to a [Director of National Intelligence] statement from July 2010, 'Terabytes of foreign intelligence information come in each day, vastly exceeding the entire text holdings of the Library of Congress, which is estimated at 10 terabytes.' In the additional views section of the Senate report on the Christmas day bombing attempt, Senators Saxby Chambliss and Richard Burr noted that analysts who could have connected the dots prior to the incident struggled to search the large volume of terrorism-related intelligence available to them. The same problem was identified at the FBI in the aftermath of the 2009 Fort Hood shooting."
The crippling dilemma of information overload is not news. (See here, here, here.) But in the context of the Boston Marathon bombings, and the inevitable questions that will follow about who knew what when, it's important to keep this salient fact in mind: The US intelligence community does not have a problem collecting information. It has a problem understanding much of it.
(Thanks, as always, to Steve Aftergood at the Federation of American Scientists for posting this CRS report, among many others.)
The Obama administration is about to pull US telecommunications companies even deeper into the ongoing cyber conflict with China.
Foreign Policy reports that in the coming weeks, the National Security Agency, in concert with the Homeland Security Department and the FBI, "will release to select American telecommunication companies a wealth of information about China's cyber-espionage program." The idea behind this reportedly classified operation is to give the telecoms more information about how Chinese cyber spies ply their trade, so that American companies can in turn get ahead of the threat and better defend themselves.
The information the government wil share with the companies includes "sophisticated tools that China uses, countermeasures developed by the NSA, and unique signature-detection software that previously had been used only to protect government networks," FP reports.
This marks an escalation in the so-called "public-private partnership" that has existed for a few years now in the ever-expanding cyber battlefield. The government has already been sharing with telecom companies some domain names and Internet addresses associated with suspected spies and hostile actors. The companies which run and manage the country's networks, in turn are expected to exercise some level of surveillance and defense, which theoretically redounds to the benefit of their customers.
This hasn't really made cyberspace any safer, nor has it significantly reduced cyber espionage and malware attacks against US companies. So now, the government is effectively giving the companies more cyber "ammo," in the form of richer, and more secretive intelligence, which it has traditionally guarded. In theory, the companies will have greater insight into how spies are trying to crack their networks.
The timing of this event doesn't seem coincidental. In February, computer security firm Mandiant released a report naming the Chinese military as a major source of espionage against U.S. companies. I'm told by knowledgable sources that the release of that report was coordinated with the Defense Department and the Homeland Security Department, which just a day earlier released much of the same threat information that's in the Mandiant report, but without attributing the source to China. Like the new information-sharing program, these are not rhetorical strategies, but rather tactical attempts to push back against cyber spying and give US companies more means to defend themselves.
The Obama administration has long understood that in order to defend cyberspace, it's going to have to enlist the cooperation and active participation of US companies. The US government, for all its technical intelligence prowess, simply cannot defend a network infrastructure that is almost entirely owned and operated by the private sector.
For their part, companies have been itching to get more information and to change the often one-way flow of threat information from the private sector to the government. Companies know they're networks are threatened, but they often don't know much about the sources of those intrusions, and what else the intruders are capable of doing. They need a government intelligence agency to obtain that information--mainly through espionage, which companies can't legally practice on their own.
Yesterday, the chief information officer for Dow Chemical Company told a Senate panel that he'd like to see more information sharing from the government to industry, and among different sectors of US companies. He's about to get some of what he asked for.
To some extent, this information exchange has been happening already. For the past few years, US defense contractors have been sharing threat information with the government and allowing government agencies to monitor their networks, so the intelligence community can gather information about US adversaries, and how they work.
Now, though, the administration is pushing this cooperation even deeper into the telecom sector, essentially taking the fight down to the level of the network operators. That's a significant development. Think of this as deputizing some companies in the new cyber war. We're going to see a lot more of this in the future.
Google is expanding its regular “transparency report” to include some broad statistics on the numbers of national security letters it receives from the US government. It’s a significant step for the company to publicly disclose what it privately tells the authorities about its users, and it gives us some more insight into how the government monitors the vastness of the Internet.
The numbers Google is reporting are broad. But the big takeaway here is that the FBI--the primary user of national security letters--appears to be interested not so much in the content of a person's email, but rather in what's known as "basic subscriber information," more high-level data such as a person's name, address, and the length of service on his account. This information is potentially more useful, and surely easier to get, than the written contents of an e-mail.
At first glance, the numbers of NSL requests Google is reporting look “awfully high” for one company, says Cato’s Julian Sanchez, who breaks down the report and places it the context of what we already know about how NSLs, which are notoriously opaque tools for secretly obtaining information, are used.
Comparing the Google numbers for NSLs to those released by the Justice Department, one might conclude that the company received one-seventh of all NSL requests, something Sanchez concludes “seems impossible.” Google is big, but not so big that it would account for an outsized share of all NSLs relative to every other company that receives them. Telecommunications companies, including phone and Internet service providers, as well as financial institutions regularly get NSLs, which require companies to hand over different kinds of information short of the actual content of a message.
So why are Google’s numbers so high? Sanchez persuasively argues that Google is counting requests for basic subscriber information, and that the Justice Department, in its own NSL reports, is not. Looking at Google’s numbers, it would appear that the “overwhelming majority” of NSL requests it receives are for this basic subscriber information, Sanchez writes, which suggests, troublingly, “that the total number of Americans affected by all NSLs is thus vastly, vastly larger than the official numbers would suggest.”
I think Sanchez is right. And it makes sense based on what we know about how law enforcement and intelligence agencies use electronic information to track people and monitor the Internet for various threats.
For instance, shortly before the 9/11 attacks, the National Security Agency asked Qwest Communications for subscriber information on its then-quickly expanding communications network. The NSA’s goal was to monitor the Internet for potential cyber threats against the government. (This was years before cyber security became de rigueur in national security circles, so this was a very foresighted move by the NSA.) After the attacks, the NSA again made the request, this time for tracking terrorists.
Qwest refused, however, after concluding that access to such detailed customer information was illegal without a warrant. Qwest executives and lawyers decided that even though the information wasn't technically "content," it was still revealing enough that giving it to the government required some legal approval.
This is an important point. Call logs and records of phone calls may be called “basic” information under the law, but they are full of rich, potentially illuminating information about a person. Today, government agencies, including the NSA, use basic data, particularly phone logs and Internet addresses, to create detailed pictures of a person’s communications and his associations. It doesn’t really matter, in this context, that the data doesn’t include the text of an e-mail or the spoken words of a phone call.
The Google disclosure underscores the extent to which the government is after this kind of general data, more so than actual content. National security letters are not warrants, but they’re being used today to obtain information of the kind that the NSA wanted from Qwest. This should come as no surprise, given how well the NSA, and the FBI, anticipated the ways that digital technology would transform communication, and how that would, in turn, give the government new opportunities for collecting information.
The way the FBI is using national security letters today, if Sanchez's analysis is correct, suggests that written e-mails aren’t really what investigators want most. It’s easier under the law to get basic information, and that information can tell them a lot about their targets, often more than the text of an e-mail itself. Think about it: How likely is a suspected terrorist to spell out his intentions in a message? You’d learn a lot more about his capability to do harm by positioning him within a bigger terrorist network, and you can understand and illuminate that network with the kinds of information that Google and other NSL recipients provide. This broad information is also useful to investigators when they're trying to identify individuals who they can scrutinize more closely with searches that require a warrant.
The other reason why a government agency would want this kind of basic information? “To effectively de-anonymize the otherwise unknown user of a particular account,” Sanchez says. That's just what investigators did when they determined that Paula Broadwell was sending anonymous e-mails to a friend of Gen. David Petraeus. Sanchez speculates that this digital de-cloaking may be “the primary reason” an agency would ask Google for basic subscriber information.
There’s an important wrinkle in all of this. Google also said that when it receives NSLs, it doesn’t disclose Internet protocol addresses. “Since these can be crucial to linking a wide array of online activity to a particular user, their exclusion would somewhat limit the potential of NSLs to undermine Internet anonymity,” Sanchez writes. But it could be that this exclusion is just a Google policy. Sanchez concludes that “it is not at all clear whether other providers will disclose IP addresses in response to NSLs.”
We should also keep in mind that NSLs are not the only means by which companies share information with the government, nor are IP addresses the only way to unmask someone or provide useful intelligence for investigators. Nevertheless, this is an enlightening report, and it adds to the ever-accreting body of details about how the government watches us, and what companies are doing to comply with the law and at the same time protect their customers’ information. Never an easy balance. It’ll only get harder.
More than a decade after the 9/11 terrorist attacks, a set of extraordinary and secretive surveillance programs conducted by the National Security Agency has been institutionalized, and they have grown.
These special programs are conducted under the code name Ragtime, and are divided into several subcomponents, according to the new book Deep State: Inside the Government Secrecy Industry, by Marc Ambinder and D.B. Grady. (I purchased a copy this morning.)
The authors, both journalists who cowrote a previous book about special operations in the military, have dug deep into the code names and operational nitty gritty of the NSA's secretive and hugely controversial surveillance programs, and they've come up with impressive new details.
Ragtime, which appears in official reports by the abbreviation RT, consists of four parts.
Ragtime-A involves US-based interception of all foreign-to-foreign counterterrorism-related data;
Ragtime-B deals with data from foreign governments that transits through the US;
Ragtime-C deals with counterproliferation actvities;
and then there's Ragtime-P, which will probably be of greatest interest to those who continue to demand more information from the NSA about what it does in the United States.
P stands for Patriot Act. Ragtime-P is the remnant of the original President’s Surveillance Program, the name given to so-called "warrantless wiretapping" activities after 9/11, in which one end of a phone call or an e-mail terminated inside the United States. That collection has since been brought under law, but civil liberties groups, journalists, and legal scholars continue to seek more information about what it entailed, who was targeted, and what authorities exist today for domestic intelligence-gathering.
Deep State has some answers.
Only about three dozen NSA officials have access to Ragtime's intercept data on domestic counter-terrorism collection. That's a tiny handful of the agency's workforce, which has been pegged at about 30,000 people.
As many as 50 companies have provided data to this domestic collection program, the authors report.
If the NSA wants to collect information on a specific target, it needs one additional piece of evidence besides its own "link-analysis" protocols, a computerized analysis that assigns probability scores to each potential target. This is essentially a way to use a computer data-mining program to help determine whether someone is a national security threat. But the authors find that this isn't sufficient if NSA wants to collect on said target. And while the authors found that the Foreign Intelligence Surveillance Court rarely rejects Ragtime-P requests, it often asks the NSA to provide more information before approving them.
How the surveillance is approved tells us a lot about the breadth of the NSA's intelligence gathering. The court and the Attorney General both certify a slate of approved targets under Ragtime-P, the authors find. That includes a certain amount of "bulk data"—such as phone call logs and records—that can be collected around those targets. An NSA official told the authors that Ragtime-P can process as many as 50 different data sets at one time.
What happens next looks like a 21st-century data assembly line. At the NSA's headquarters in Fort Meade, Maryland, a program called Xkeyscore processes all intercepted electronic signals before sending them to different "production lines" that deal with specific issues. Here, we find another array of code names.
Pinwale is the main NSA database for recorded signals intercepts, the authors report. Within it, there are various keyword compartments, which the NSA calls "selectors."
Metadata (things like the "To" and "From" field on an e-mail) is stored in a database called Marina. It generally stays there for five years.
In a database called Maui there is "finished reporting," the transcripts and analysis of calls. (Metadata never goes here, the authors found.)
As all this is happening, there are dozens of other NSA signals activity lines, called SIGADS, processing data. There's Anchory, an all-source database for communications intelligence; Homebase, which lets NSA analysts coordinate their searches based on priorities set by the Director of National Intelligence; Airgap, which deals with missions that are a priority for the Department of Defense; Wrangler, an electronic intelligence line; Tinman, which handles air warning and surveillance; and more.
Lest you get confused by this swirl of code names and acronyms, keep this image in mind of the NSA as a data-analysis factory. Based on my own reporting, the agency is collecting so much information every day that without a regimented, factory-like system, analysts would never have the chance to look at it all. Indeed, they don't analyze much of it. Computers handle a chunk, but a lot of information remains stored for future analysis.
So who is monitoring this vast production to ensure that the communications of innocent Americans aren't spied on? Ambinder and Grady report that for the NSA's terrorism-related programs, the agency's general counsel's office regularly reveals "target folders," which contain the identities of those individuals who are under surveillance, "to make sure the program complied with the instruction to surveil those reasonably assumed to have connections to al-Qaeda."
That the NSA is policing itself may come as small comfort to many critics of the Obama administration's intelligence programs. The size of the "compliance staff" that monitors this activity is only about four or five people, depending on what's available in the budget at any moment, the authors report. They also say that we cannot know whether the program is pushing beyond the boundaries of the law.
However, outside the closed circle of about three dozen NSA employees who are read in to Ragtime, there more than 1,000 people "outside the NSA are privy to the full details of the program." If NSA is breaking the law, "how much longer can that secret last?" the authors ask.
We have a preceding example to test this hypothesis, albeit in a limited fashion. In 2004, the senior leadership of the Justice Department and the FBI threatened to resign over what they saw as illegal collection activities at the NSA, collection activities that are still going on under Ragtime and under new surveillance law.
Back then, James Comey, acting as Attorney General while John Ashcroft was in the hospital, refused to sign a set of certifications provided by the Justice Department to Internet, financial, and data companies, the authors report. Why? Comey believed that the justification for providing bulk data to the NSA wasn't sufficient.
The administration's tortured logic "drove him bonkers. There was just no way to justify this," the authors report, quoting people who have spoken to Comey, who has never publicly said why he objected. Interestingly, the authors find that the parts of the program he was objecting to didn't implicate the Foreign Intelligence Surveillance Act.
This comports with my own reporting in my book, The Watchers. The NSA was making "mirrors" of telecommunications databases, so that analysts could go through the data and mine it for clues. As it has been explained to me, the problem here dealt with how the government viewed its legal authorities to access data stored in computers, and whether analysts could dip back into it without specific authorizations. Importantly, this data consisted of that so-called "bulk data." It wasn't recorded phone calls or the text of e-mails. That information was governed by FISA--or should have been--because it was considered "content" under law, and that requires a warrant to obtain.
The White House panicked when Comey and Ashcroft refused to sign off, Ambinder and Grady report, fearing that the companies on which NSA was depending for information would cut the agency off if they didn't get a signed order from the Attorney General himself. It took six months for the administration to reshape the program so that it comported with "interpretation of the metatdata provisions" that were promulgated by the Justice Department's Office of Legal Counsel.
Had these officials resigned, it's unthinkable that the secrets of NSA's intelligence gathering activities would have stayed hidden. A year later, in 2005, they were revealed in part by the New York Times. Here, too, Ambinder and Grady have some new insights. It turns out that while the NSA's director, General Michael Hayden, was publicly excoriating the newspaper for disclosing the classified activities, he was privately glad that they withheld what he considered key operational details.
The sordid tale of the Marine, the Hostess, the Rival, and her Lover is entering its second act: The quest for public redemption.
The FBI has launched a new Web site that lets you search for bank robberies by neighborhood and see surveillance shots of wanted crooks. You can also search by date of the robbery or the robber's nickname.