The home secretary, Theresa May, is to introduce a "comprehensive package" reforming the use of police stop-and-search powers after telling MPs that as many as a quarter of a million street searches last year were probably carried out illegally.
The home secretary said she wanted the battery of revised codes, best practice schemes and new methods of accountability to lead to a significant reduction in the use of stop and search, more intelligence-led targeted operations, and better arrest ratios.
Black people are still seven times more likely to be stopped by the police than white people, with only about one in 10 of those stops leading to an arrest.
But during a Commons statement on Wednesday it became clear that May has lost her battle to persuade Downing Street that legislation to curb the use of stop and search should be included in this summer's Queen's speech – the annual legislative programme – the last before next year's general election.
"I want to make myself absolutely clear: if the numbers do not come down, if stop and search does not become more targeted, if those stop-to-arrest ratios do not improve considerably, the government will return with primary legislation to make those things happen," May told the Commons.
"Nobody wins when stop and search is misapplied. It is a waste of police time. It is unfair, especially to young, black men. It is bad for public confidence in the police," she told MPs.
May justified the need for a comprehensive reform package by citing the findings of a study by Her Majesty's Inspectorate of Constabulary that showed 27% of stop-and-search records they looked at did not contain reasonable grounds to search people, even though they had been endorsed by supervising officers
"If the HMIC sample is accurate, that means more than a quarter of the million or so stops carried out under the Police and Criminal Evidence Act last year could have been illegal," May said.
A Home Office consultation launched last July on the future of stop-and-search powers has generated more than 5,000 responses. They showed that young people and black people had very different attitudes towards stop and search than white and older age groups.
The package of reforms proposed by May now includes:
• A revised code of practice making clear what constitutes "reasonable grounds for suspicion" under the Police and Criminal Evidence Act, which covers the vast majority of stops carried out. Officers who fail to use their powers properly will face disciplinary proceedings.
• Telling chief constables and police commissioners they must open up police stop-and-search records to public scrutiny, under threat of statutory legislation.
• Review national police training to include "unconscious bias awareness training" to reduce possibility of prejudice in officers' approach to stop and search.
• Official assessment of individual officer's fitness to use stop-and-search powers.
• Launch a new "best use of stop and search scheme" which will include better recording and improved public accountability and will limit the use of section 60 "no suspicion" stop-and-search powers to 15 hours.
May launched the consultation last July saying she wanted to introduce proposals to scale back the use of police stop-and-search powers and ensure their "fair and effective use".
At the time, the home secretary said it was no longer sustainable that black people were still seven times more likely to be searched on the street than white people. She also said it had been seen as sharply divisive in Britain's black and minority ethnic communities.
May had planned to announce cabinet approval of her proposals before last Christmas, but she ran into strong Downing Street concerns that the move could leave the Tories looking "soft on crime".
Nick Clegg, who chairs the cabinet's home affairs committee, is said to have backed May's package but it was not sufficient to end the standoff.
The Metropolitan and West Midlands police forces have already massively scaled back their use of stop and search as part of a successful pilot scheme using a more "intelligence-led approach".
In the Met, the use of stop and search has fallen by 20% since 2012 and the use of section 60 "no suspicion" stops by 90% over the same period. At the same time stabbings have fallen by a third, shootings by 40%. The arrest ratio has gone up and complaints have gone down.
