When Keir Starmer shifted Labour’s position on Kashmir in the first weeks of his leadership last year, he may not have realised that his stated intention of preventing “issues of the subcontinent to divide communities here” could cost the support of some of its most loyal voters.

Canvassing outside Jamia Masjid in Heckmondwike on Friday, the Labour candidate in the Batley and Spen byelection, Kim Leadbeater, received a hostile reception from voters who are unhappy with the party’s stance on foreign policy issues such as Palestine and Kashmir, amid a perception that the party takes some forms of racism more seriously than others.

The accusation “you’ve taken our votes for granted” was repeatedly levelled at Leadbeater and Lisa Nandy, the shadow foreign secretary and Wigan MP, who joined her on the campaign trail.

“Keir took the time to condemn two idiots for being antisemitic last month but he won’t condemn the Israeli government for killing innocent people,” said Wajjad Hussain, 33, referring to an incident in north London last month where antisemitic abuse appeared to be shouted from a car.

“I’ve voted Labour my whole life but I won’t be blindly giving them my vote any more. And that’s not just about Palestine. It’s everything locally. They’ve been in power here for 25 years but only now they’re under threat do they care about Asians,” he added.

While many voters were glad that antisemitism was tackled seriously by the party, they felt allegations of Islamophobia were not given the same priority. On Sunday, the Labour Muslim Network wrote to Starmer and others to complain about an anonymous briefing to the Mail on Sunday from a “senior Labour official” claiming that the reason the party was losing votes was a “backlash” from Muslim voters over “what Keir has been doing on antisemitism”.

LMN said the “obvious implication” was that the Muslim community themselves harbour antisemitic views, which it described as a “common racist trope perpetuated onto Muslims”. The letter said the official had breached the all-party parliamentary groups’ definition of Islamophobia. Labour deputy leader Angela Rayner has promised an investigation into the comments by the party official.

Even before the Mail on Sunday story, five Muslim organisations across Batley and Spen had written to Starmer on Wednesday expressing their disaffection. The letter-writers said that while Muslims in the area had been “proud to support the Labour party, for a long time the Labour party has not been proud of our support”. By Sunday, according to Dr Wajid Khan, the organisers had not had a response.

The sentiment is not just in the constituency: a recent poll carried out by Survation on behalf of the Labour Muslim Network showed falling support for the party among Muslim voters.

Analysis from the Muslim Council of Britain (MCB), using 2018 data from the ONS, suggests Batley and Spen is one of the top 15 seats where Muslim voters have high impact. With turnout estimates, MCB say that about 8,600 voters on 1 July will be Muslim. Labour is defending a 3,525 majority.

One of those is Bashir Karolia, the owner of Mehrab Bakery in Batley. Between chatting to customers buying sweets and cakes, Karolia explains that his unhappiness with the Labour leader means that for the first time, he will be placing his vote elsewhere.

“He’s never spoken up about Batley or immigration or about Muslims. Never. Now he’s speaking up because George [Galloway] is here,” said Karolia. “So why should I support a Labour leader that has never spoken for me or my community until he wants the vote?”

Galloway, the divisive politician who delivered a shock defeat to Labour in nearby Bradford West in 2012, has set up a campaign base in a caravan festooned with Palestinian flags a few streets away. He has now expanded into an empty travel agents, one of three campaign headquarters in the Batley and Spen constituency covered with banners of Galloway in trademark fedora hat and posters in Labour red.

Karolia’s frustrations are nothing to do with Leadbeater, the sister of Jo Cox MP, who was murdered in her constituency, but Starmer, who he perceives as failing to challenge Islamophobia. But if not for Galloway – who wants to “bring about a convulsion in the Labour party that saw [Starmer] thrown out” – he would have voted Conservative.

But what of the prime minister’s comparison of Muslim women in burqas to letter boxes, and other allegations of Islamophobia in the Conservative party?

In that case, Karolia is motivated by “different reasons”, he says, pointing to neighbouring Dewsbury, who voted in a Conservative MP in 2019 and has just received £24.8m from the government’s towns fund. “If we have a Labour MP, we’re not going to get anything – and Batley needs reviving,” he said.

It’s a feeling the Tories hope to capitalise on. Boris Johnson visited Batley on Friday with the Conservative candidate Ryan Stephenson, a councillor in Leeds, who he said would help to target funding whether through the towns fund, the levelling up fund “or the many other funds that we have available”.

Galloway launched his campaign under the Workers Party of Great Britain banner just three weeks ago, fresh from an unsuccessful bid to get elected to Holyrood for anti-Scottish independence party All for Unity. Despite accusations of alleged antisemitic remarks and minimising sexual assault, his charisma and reputation for what a Batley deli-owner describes as “support for humanity” means Galloway enjoys local celebrity status in some areas. He is regularly stopped for selfies, and local food outlets offer him refreshments on the house.

WhatsApp groups within the Asian community have been buzzing with video endorsements for Galloway from local former Labour activists and even councillors, as well as the prominent Palestinian youth activist Ahed Tamimi. Hasan Ali, a 19-year-old first time voter, said his main issue was Palestine, and that Galloway would get his vote despite mistakenly believing he was standing for Labour.

While his campaign began in the mosques and Muslim community centres, Galloway, who supported Brexit, also aims to capitalise on what he describes as a “critical mass of unhappiness with Labour amongst white working class people”, hoping to chase a few of the 12% of vote share that fell to Paul Halloran in 2019, an independent candidate who is not standing.

One local Labour activist said that there was no doubt that the strength of feeling against Starmer, and Galloway’s “massive” support on the doorstep in Asian areas, would cost them the seat, resulting in a Conservative MP for Batley and Spen for the first time since 1997. It’s a feeling matched by other dejected door knockers, who feel the party has been slow to wake up to the unhappiness among Muslim voters, instead concentrating on stemming the tide of lost votes in white, working class areas. Another poll conducted last week by Survation showed the Conservatives on course to win the seat, with Labour trailing six points behind.

Others are less pessimistic, hoping that Galloway’s popularity among men is not universally matched by Muslim women and that the party’s targeted campaigning, local messaging and Leadbeater’s strong ties within the constituency, could mean that Labour manages to hold on to the seat.

The fightback includes a leaflet calling for a two-state solution in Palestine, self-determination for Kashmir and a promise to tackle Islamophobia. Naz Shah and Afzal Khan, northern MPs who have both faced Galloway and defeated him, are two of several Labour politicians canvassing regularly.

It may be enough to hold on to the seat, but the perception that Labour equivocates on issues important to its core voters may continue to be a headache.

“Everyone knows that what Israel is doing is wrong,” said Alieu Sey, 40, speaking after Friday prayers in Heckmondwike. “Politicians nowadays don’t speak their minds but George will say what needs to be said.”

• This article was amended on 20 June 2021. The voter quoted was Wajjad Hussain, 33, not “Wajid Mohammed, 35” as an earlier version said.