It’s a calm summer afternoon off Fraserburgh in northern Scotland, and Luke Duthie is hunting mackerel.

His 8-metre-long (26ft) boat is equipped with sonar that soon shows a large red cloud about 15 metres below the surface. He dashes out of the cabin and on to the deck, lowers the twin fishing lines and watches them dance in the waves. But the fish aren’t biting.

Back in the cabin, Duthie, 25, says the catch has been shrinking for years – and so have the fish. Often, the fish are too small to meet the legal requirement, and he has to throw them back into the sea.

“When I was 12, we caught maybe three or four small mackerel,” Duthie recalls of fishing trips with his father. “Today, 60% are too small.”

Mackerel is the most economically valuable fish in the north-east Atlantic Ocean, and along with herring and blue whiting at No 2 and No 3, it has been systematically overfished – in some cases, for more than a decade.

The three fish are pelagic: they spend their lives in the upper layers of the sea, just below the surface, and migrate from Portugal to as far as Norway. As the shoals travel, large industrial vessels wait for them. Duthie is just one small fisherman in a billion-dollar business.

To avoid a wild west scenario where everyone takes whatever they can, the coastal states – Norway, Russia, Iceland, the Faroe Islands, along with the EU and, since Brexit, the UK – meet every year in London at the North-East Atlantic Fisheries Commission (NEAFC), where they must try to agree on catch quotas. (Russia was not invited this year after it invaded Ukraine.)

In most cases, the countries formally accept scientists’ recommendations for the maximum amount of fish that should be caught. When it comes to sharing the pie, however, the negotiators don’t agree on who gets what. They almost always end up allocating themselves quotas that are far too high – and, collectively, end up fishing too much.

The result has been chaos. An investigation by journalists from the German broadcasters NDR and WDR, the German newspaper Süddeutsche Zeitung and the Guardian shows that in recent years, fishers in the north-east Atlantic have caught between 66% and 86% more than the amount scientists – and even the countries themselves – agree is safe.

“The reality is that everyone is only defending their own national interests – and nobody cares about the environment,” says Javier Lopez, a campaign director at Oceana, an NGO focused on ocean health.

The effect could be catastrophic. Mackerel, blue whiting and even the famously plentiful herring may soon be in real danger should this systematic overfishing continue.

Anyone watching the negotiations in London earlier this month would have no doubt that the situation is likely to continue. The delegations left without an agreement. According to sources involved in the negotiations, an agreement for 2023 is unrealistic too.

“When you understand how the negotiations are really going, you can’t believe it,” says Anna Heiða Ólafsdóttir, a marine biologist at the Marine and Freshwater Research Institute in Iceland. Along with many experts, she describes the current system of negotiations as unsustainable.

The Marine Stewardship Council agrees: it has withdrawn its well-known “blue tick” sustainability seal for all three fish species. The supermarket chain Aldi is among those calling for greater political efforts to protect fish stocks.

Activists have also complained for years that quota negotiations are heavily influenced by the fishing industry. Business representatives have joined the annual meetings for years, and NGOs have only been allowed in since 2021. Responding to a freedom of information request, both the NEAFC and the European Commission declined to list the negotiation participants for the past five years. The UK, which had the chairmanship for mackerel negotiations for 2022, also declined to provide the list.

However, documents obtained in the investigation, and discussions with several insiders, suggest that the negotiations are heavily influenced by the fishing industry. In a previous round of quota negotiations in the north-east Atlantic last year, this time between the UK, Norway and the EU, about 40 of the 75 participating EU negotiators were industry representatives; about 30 were from various governments, and only three came from NGOs.

Then Brexit happened, and things went from bad to worse.

Contested quotas

Fish like mackerel know no national borders. A blue-and-green-striped fish with a silvery belly, many mackerel are born in the spring in Irish waters, and in the summer swim in their millions to the waters of the Faroes and Norway, and sometimes even as far as Iceland and Greenland. When they’re done feeding, they head back south to spawn.

Because no single country can be said to “own” the fish, the 1982 United Nations Convention on the Law of the Sea stipulates that all countries must take into account the interests of other countries. If one coastal state fishes too much, it puts everyone else at a disadvantage. Therefore, they meet every year to determine the respective shares.

There are negotiations of this kind all over the world, and like many of them, the NEAFC has a problem: it has no way of forcing the countries to agree, and no way of punishing them for not agreeing. As long as the countries meet, they have, officially, done their duty.

The negotiations follow a similar pattern. Each country sends a delegation to London, usually accompanied by industry representatives and marine biologists, and together they sit down in the NEAFC offices, in a nondescript building like any other in white-collar Baker Street. Between rounds of negotiations, the representatives occasionally step outside for a coffee, but are otherwise confined to the closed-door meetings.

To open the dance, scientists from the International Council for the Exploration of the Sea – an independent organisation that studies the sustainable use of the ocean – recommend a maximum catch quota for each species. Mostly, everybody agrees. Then the negotiations begin. In round after round of backroom talks, each country jostles for the highest quotas, leaning on historical or territorial arguments in an attempt to justify why it alone should get a different share than the proportion recommended by scientists.

“It’s been very complicated for years,” says Ann Kristin Westberg, the Norwegian delegation’s chief negotiator.

Norway, for example, argues that herring spawn in its national waters, so it should get a particularly high quota. Russia doesn’t even have a north-east Atlantic coastline, but claims historical fishing rights for its vessels because, well, it always has. The chief negotiator for the Faroes, Herluf Sigvaldsson, declined to comment on specific numbers but said: “I have never entered negotiations when someone wanted a smaller quota.”

The departure of the UK from the European Union complicated this situation even further, with two major effects: first, it severed a large part of the EU’s coastal waters; and second, it threw a new and fractious negotiator into the mix.

The EU still argues that it has a historical right to catch large quotas. However, the UK government insists British fishers need higher quotas than they received while in the EU, particularly for mackerel, which it claims have high “zonal attachment” because the fish spend a lot of time in UK waters – and with 773,676 sq km (298,718 sq miles) of those territorial waters, it is a bulky new player at the table.

Britain is also playing Brexit politics, argues Dr Bryce Stewart, a marine ecologist at the University of York. Fishing was a key plank of the campaign to leave the EU, and fishers were targeted with such slogans as “seas of opportunity” and “take back control of our waters”.

“I was constantly amazed by the sort of high profile of fisheries in the [Brexit] debates,” Stewart says, noting that the industry accounts for only 0.12% of the UK’s economy. In the end, small-scale fishers strongly supported the Brexit campaign.

Many now feel betrayed. “We’ve probably lost more than we’ve gained,” says David Milne, the chairman of the Scottish White Fish Producers’ Association.

He would have liked the UK government to “trade” some of its substantial mackerel, herring and blue whiting quotas as currency to access cod in Norwegian waters, in what is a relatively common form of bilateral agreement. So far, however, he hasn’t had any luck in getting his message across. “You’ve got some really, really rich families controlling the pelagic, and the government intends to listen to them.”

Ian Gatt, the chief executive of the Scottish Pelagic Fishermen’s Association, counters that the pelagic quotas aren’t high enough to justify trading them away. “We just don’t buy into the thing that because somebody’s quota is poor in a particular deal, that somebody else should pay their bill,” he says. He says the biggest challenge is the lack of quota agreement, but, like everyone else, he isn’t satisfied with the quota given to the ship owners he represents. “If you asked me, do you think these stocks are going to collapse? No, I don’t.”

Norway, meanwhile, scoffs at Britain’s position. Its chief negotiator, Ann Kristin Westberg, points out that each nation has a different perspective on its fisheries – and matter differently to their economies. For Norway, she says zonal attachment – time the fish spend in Norwegian waters – is the most important point. When she argued that point last year, she says, matters got heated. No agreement was reached. “That’s why everyone sets their own quotas.”

Warning signs

The result: collective overfishing. Last year, north-east Atlantic countries caught almost 20% more mackerel, 50% more herring and one-third more blue whiting than scientists say is sustainable.

For mackerel, the International Council for the Exploration of the Sea recommended a quota of 922,000 tonnes in 2021, but the states allocated themselves a collective quota that was 168,000 tonnes higher (a total of 1.1m). For herring, the recommendation was 650,000, while the allocated quota hit 880,000; the year before, in 2020, the recommendation for blue whiting was more than 1.1m tonnes, but the allocated quota topped 1.47m.

When asked for comment, the European Commission said its proposals always followed the scientific recommendations. It said it paid attention to how much pressure fish stocks have come under in order to protect them accordingly. A spokesperson for the UK’s Department for Environment, Food and Rural Affairs said: “Using our powers as an independent coastal state, our objectives across these negotiations are to secure a deal which puts in place a stable quota-sharing arrangement, promotes the long-term sustainability of the pelagic stocks, and ensures the long-term profitability of the UK’s pelagic industry.”

The three fish species can, in a sense, consider themselves lucky so far; despite the overfishing, no stock has yet vanished. But the warning signs are everywhere, and other fish stocks in European waters, such as cod, have already collapsed.

Ólafsdóttir, who has advised the Icelandic delegation on numerous occasions, says reform is desperately needed. “The day will come when overfishing will have consequences,” she says. So far, the schools of fish have come back every year. “But that doesn’t mean we’ll be so lucky for ever.”

• The main image on this article was replaced on 1 November 2022, as the original image showed a species of fish that does not feature in the article.

• This article was a collaboration between the Guardian, NDR, WDR and Süddeutsche Zeitung and developed with the support of journalismfund.eu