Anti-racist campaigners have said their protest rally was more than twice as popular as a pro-Brexit demonstration led by Ukip and the far-right figurehead Tommy Robinson on opposing marches through central London on Sunday afternoon.

Thousands of people massed outside BBC headquarters at Portland Place under the banner “No to Tommy Robinson; no to Fortress Britain” before marching through central London, via Trafalgar Square to the north end of Whitehall.

They had gathered to oppose a “Brexit betrayal” march led by Robinson – real name Stephen Yaxley-Lennon – and Ukip, in response to the deal brokered by Theresa May with the European commission. It began at Park Lane and made its way to the south end of Whitehall via Parliament Square.

The Labour grassroots group Momentum, one of the organisers of the Stop Tommy demonstration, claimed it had brought 15,000 on to the streets, compared with about 3,000 who attended Robinson’s protest. Other sources within the anti-racist movement said they counted their protest at closer to 8,000.

There were verbal confrontations between anti-racist protesters and hecklers in Trafalgar Square, prompting police to move in and more tightly manage the crowd.

One man shouted “leftwing scum, off our streets”, at protesters making their way into Whitehall. He was confronted by another man who shouted in his face: “We’re black, white, Asian and we’re Jewish. Fuck off!”

On Haymarket, a group of Brexit rally supporters had to be moved by police after they tried to block the path of the anti-racist demonstration.

Anti-racist protesters arrived at Whitehall to a large sound system on a curtain-side lorry playing hip-hop music, behind which a police cordon stretched across the street to keep the opposing demonstrations apart. The Metropolitan police reported three arrests, including one for assault, one for possession of an offensive weapon and one for a public order offence, all on the anti-racist side.

The Met had imposed strict conditions on timings and routes of both demonstrations. Hundreds of officers in riot gear and units of mounted police were deployed to keep order.

Laura Parker, national coordinator for Momentum, said: “It’s really important that everyone from Labour, from the broader Labour movement, from the trade union movement comes out and makes it clear that we will not have fascists on the streets of Britain in 2018.

“We have seen a terrifying rise of the right across Europe, in Hungary, in Poland. We have got Trump in America, Bolsonaro in Brazil and Tommy Robinson wandering around London. It has just got to be stopped.”

Weyman Bennett, co-convener of Stand Up to Racism, said he felt it was important for anti-racists to appeal to both remain and Brexit supporters, rather than tar those who want to leave the EU with the brush of racism. He said: “The main danger is that for the first time we have seen Ukip, the BNP and the EDL coming together as a mainstream organisation, linking together electoral and street politics, which has not happened for many years. It’s a racist criminal conspiracy against black people in this country. An urgent task is now to do this in every town.”

On the other side of the police cordon on Whitehall, several thousand pro-Brexit supporters rallied beneath union flags, purple Ukip flags and the lambda insignia of radical right group Generation Identity, listening to speeches from Yaxley-Lennon, the Ukip leader Gerard Batten, the YouTuber Carl Benjamin and the Tory-turned-Ukip politician Neil Hamilton.

Addressing the crowd, Yaxley-Lennon said he was among the 30-40% of people who don’t vote. “I’ve never voted because there is no one to vote for,” he said.

Now, with a move to the right from Ukip, whose leader recently appointed Yaxley-Lennon as an adviser, Britain would have a populist party of its own to vote for, he said. “Ukip can be a voice for the working class communities,” he said, adding that “no other politician will talk about Islam; no other politician will talk about the issues affecting our country.”

Among those watching was Madeleine Jenkins, 72, from Brentwood. She had joined the march to “get us out of the EU”, she said.

“It’s a totally fake deal and we want to go to WTO rules. It shouldn’t be frightening because we don’t want to belong to a communist state. We want to be independent and I think you will find the eastern European countries also wouldn’t want to be in what it’s become, because they just got out of a totalitarian state.”