To most visitors, Cheltenham is a charming spa town on the edge of the Cotswolds. They admire its handsome regency terraces, visit its racecourse and throng to a thriving festival scene. Less visibly, Cheltenham is also a company town built around one industry: spying.
The Government Communications Headquarters seems to be very good at what it does. Its 6,400 employees include many bright computer engineers who work tirelessly to invent ever more imaginative ways to collect vast amounts of data on hundreds of millions of people.
Some find what they do reassuring, others menacing. As for the people who work at GCHQ, they have found themselves, for the first time, under intense scrutiny. This is, to put it mildly, unwelcome to them. They would like it to stop – and they have friends in politics, the law and even the press who agree.
According to those who study such things for a living, we live in a golden age of surveillance. The mobile phones we carry around betray us – our movements, our search terms, our health, our intentions, our friends, our emails, our texts. A bland name for it is "metadata". But, as one former lawyer with the US National Security Agency told me: "Metadata absolutely tells you everything about somebody's life."
GCHQ is, along with the NSA, a world leader. Over the past five years GCHQ's access to what they call "light" (a sweeter name for metadata) has increased by 7,000%, according to documents leaked by the former NSA contractor Edward Snowden. The amount of material being analysed or processed is up by 3,000%. That's a lot of light.
Some accuse GCHQ of being little more than the Cheltenham branch office of the NSA. This may be unfair, but Whitehall officials concede that there is a high degree of co-operation and sharing between the two agencies. One of Cheltenham's senior legal advisers put the possible attraction for their American counterparts this way: "We have a light oversight regime compared with the US."
GCHQ, which receives tens of millions of pounds from the NSA every year, used, in some minds, to be the Cinderella of the intelligence world. The public imagination was more easily captured by James Bond, George Smiley and the cold war pitting of agent against agent.
It's now clear that GCHQ and the NSA have risen without trace to the top of the intelligence pecking order. Increasingly an asymmetry has developed: they potentially know virtually everything about us, but we know virtually nothing about them.
This raises three questions. First, is it right that they are able to master all civil and commercial forms of communication in order to collect, store and analyse information about entire populations? Who knew?
Secondly, is it right that we should know so little about who they are or what they do – that this dramatic loss of individual privacy, unprecedented in history, could be done without any kind of public knowledge or consent? Who agreed?
Finally, is this new infrastructure sustainable?
Everyone agrees we need intelligence agencies and that their work will, to a significant degree, be secret. But technology today is so potentially intrusive it raises sharp questions about what sort of democratic controls or accountability can be exercised over the talented engineers at Cheltenham or their counterparts in Fort Meade, Maryland.
The traditional answer has been secret courts and secret committees. But all come with further questions. Are the trusted few invariably told the truth, the whole truth and nothing but? Do they know what questions to ask? Do they truly understand cutting-edge technologies? How much auditing can they really do on secret bureaucracies and processes? Do they have privacy advocates or digital advisers at hand? How much do they themselves share with fellow lawmakers who might reasonably need to have some insight into the state's capabilities before approving still more powers?
The secrecy regulators themselves thus come under scrutiny – and it is clear that the Snowden documents raise issues of concern. It is certain, for instance, that Congress was not always told the truth about what the NSA was up to. The independence of the foreign intelligence surveillance court – the one-sided secret mechanism for signing off state surveillance requests – has also been questioned. The Westminster parliamentarians asked to approve vastly expanded powers of data collection are furious they were denied information which has only now come to light.
The one striking point of agreement now is that virtually everyone concedes the need for a public debate about these issues. "Everyone" includes the US president, lawmakers, ambassadors, academics, tech companies, cryptologists, journalists, lawyers, the oversight committees – and even the spies themselves.
The congressman who was responsible for the post-9/11 Patriot Act was horrified to discover (from the press) what it was being used for and is moving another bill to stop what he considers to be unwarranted snooping. He says he wants to "put the [NSA's] metadata program out of business".
The intelligence agencies must be asking themselves how such programmes are sustainable without that debate – and without meaningful consent from the people whose data is being harvested. After Chelsea Manning and Snowden it is surely apparent the agencies cannot securely keep their precious secrets if their own employees are troubled enough to leak them. Their engineers and analysts must believe that the law is in tune with the technologies they create and employ.
Finally, there is the role of the media in all this. This debate would not have been possible without journalists – and, indeed, without Snowden himself. The issues which, it's now acknowledged, have to be discussed were not revealed through secret oversight processes but by newspapers.
A former NSA general counsel, Stewart Baker, stated what he understood to be the constitutional settlement in the US regarding the position of the press in such matters: "Snowden violated the law, but once he's given it to the reporters, the reporters are protected."
In America there is no prior restraint. No journalist is prosecuted under the Espionage Act. There is virtually no criticism of the New York Times or Washington Post for pursuing a story which is so clearly of public importance.
In Britain the picture is far less clear. Here, the state can threaten to close down debate, impound documents, call in the police, misuse terror laws and even get parliament to investigate a newspaper which shines a light on institutions more comfortable with darkness.
"When a government calls journalists traitors the questions should begin, not end," wrote the New Yorker's Amy Davidson recently. She's right.
Alan Rusbridger is the editor of the Guardian
