Bates also complained that the government's submissions make clear that the NSA was gathering Internet data years before it was authorized by the USA Patriot Act's Section 702 in 2008.
The NSA had moved to revise its Internet surveillance in an effort to separate out domestic data from its foreign targeted metadata — which includes email addresses and subject lines. But in his October 2011 ruling, Bates said the government's "upstream" collection of data — taken from internal U.S. data sources — was unconstitutional.
Three senior U.S. intelligence officials said Wednesday that national security officials realized the extent of the NSA's inadvertent collection of Americans' data from fiber optic cables in September 2011. One of the officials said the problem became apparent during internal discussions between the NSA and Justice Department officials about the program's technical operation.
The problem, according to the officials, was that the top secret Internet-sweeping operation, which was targeting metadata contained in the emails of foreign users, was also amassing thousands of emails that were bundled up with the targeted materials. Because many web mail services use such bundled transmissions, the official said, it was impossible to collect the targeted materials without also sweeping up data from innocent domestic U.S. users.
Officials said that when they realized they had an American communication, the communication was destroyed. But it was not clear how they determined to whom an email belonged and whether any NSA analyst had actually read the content of the email. The officials said the bulk of the information was never accessed or analyzed.
As soon as the extent of the problem became clear, the officials said, the Obama administration provided classified briefings to both Senate and House intelligence committees within days. At the same time, officials also informed the FISA court, which later issued the three 2011 rulings released Wednesday — with sections blacked out — as part of the government's latest disclosure of documents.
The officials briefed reporters on condition of anonymity because they were not authorized to do so by name.
The gathering of innocent Americans' communications was happening when the NSA accessed Internet information "upstream," meaning off fiber optic cables or other channels where Internet traffic traverses the U.S. telecommunications system.
The NSA disclosed that it gathers some 250 million Internet communications each year, with some 9 percent from these "upstream" channels, amounting to 20 million to 25 million emails a year. The agency used statistical analysis to estimate that of those, possibly as many as 56,000 Internet communications collected were sent by Americans or people in the U.S. with no connection to terrorism.
Under court order, the NSA resolved the problem by creating new ways to detect when emails by people within the U.S. were being intercepted and separated those batches of communications. It also developed new ways to limit how that data could be accessed or used. The agency also agreed to only keep these bundled communications for possible later analysis for a two-year period, instead of the usual five-year retention period.
That means the U.S. material is still gathered and kept, but is treated with stricter protocols.
The agency also, under court order, destroyed all the bundled data gathered between 2008, when the FISA court first authorized the collection under Section 702 of the Patriot Act, and 2011, when the new procedures were put in place.
The court signed off on the new procedures.
The once-secret documents were posted on a new website that went live Wednesday afternoon. The front page of the site said it was "created at the direction of the president of the United States (and) provides immediate, ongoing and direct access to factual information related to the lawful foreign surveillance activities carried out by the U.S. intelligence community."