An unprecedented snapshot of migrants held in British detention centres found more than half of the sample were either suicidal, seriously ill or victims of torture, a Guardian investigation has established.

The survey of almost 200 detainees held in seven deportation centres in England as of 31 August showed almost 56% were defined as an “adult at risk”. Such individuals are only supposed to be detained in extreme cases, suggesting that Home Office guidelines on detention have been breached.

The survey – conducted in association with 11 law firms and charities that work with those facing deportation – also found that a third had dependent children in the UK, and 84% had not been told when they would be deported – implying open-ended incarceration.

Almost half the detainees had not committed a crime, but the average detainee in the sample had been imprisoned for four months. The majority had lived in the UK for five years or more and some had been in the country for more than 20 years.

The sample amounts to 8% of all those held in detention at the time of survey, according to the most recent Home Office figures. A Home Office spokesperson insisted detention was “an important part of the immigration system”, but said that it must be “fair, dignified and protect the most vulnerable”, adding that further improvements could still be made to the system.

While it is not sufficiently scientific to be extrapolated across the entire removal population, the survey suggests many hundreds of extremely vulnerable people are being held indefinitely, in one of the most severe manifestations of the Conservatives’ “hostile environment” policy.

Roland Adjovi a member of the UN Office of the Human Rights Commissioner’s arbitrary detention working group, said that states must ensure that detention ‘is truly a measure of last resort’

“Detention in the context of migration must be a measure of last resort,” he said. “Such detention can never be of unlimited duration and the national legislation must clearly prescribe the maximum permitted duration of detention.”

The former prisons and probation ombudsman Stephen Shaw, who has conducted two comprehensive reviews for the government into immigration detention, added: “Although the overall use of detention has fallen by one third in the last three years, far too many people are still being detained for long periods when there is no realistic prospect of their removal from the UK.”

The shadow home secretary, Diane Abbott, said: “This snapshot is truly shocking, but not entirely surprising.

“There have been repeated assurances that vulnerable people, victims of trafficking and children would not been detained. But this investigation shows that those assurances are worthless. People are even being detained even though there is no instruction for their removal. This is a scandalously inhumane and unjustifiable system.”

The government detains just over 25,000 people every year pending deportation, at an annual cost of £108m. The practice of indefinite incarceration has been criticised by high court judges, local authorities, parliamentary committees and the UN.

More than half of all detainees are in any case ultimately released back into British society, not deported. Some have taken legal action over their imprisonment. The Home Office’s latest annual report acknowledges that government has paid out £3m to 118 people unlawfully detained in the 2017/18 financial year.

The UK is the only country in Europe to detain people without a time limit. It was Guardian revelations about government’s removal targets which forced Amber Rudd to resign as home secretary in April. Detention centres are instrumental to that policy.

In July, the new home secretary, Sajid Javid, promised to make changes to immigration detention. But the Guardian investigation revealed very little had changed and many vulnerable people were still being detained.

Eleven law firms and charities entered anonymised data on 188 people to build a snapshot of people in deportation centres on 31 August. The data included how long they were held, whether they were considered an adult at risk and whether they had been told when they would be deported.

The survey found:

• Children were held in adult detention centres, while 30% of detainees had dependent children in the UK.

• More than half were defined as an adult at risk due to being victims of torture, having suicidal thoughts or being unwell.

• While the government claims detainees are held briefly before being deported, 84% had not been given removal directions.

• Detention ranged from under five days to nearly three years, with a median of four months, despite Home Office guidance that it should be used sparingly and for the shortest period necessary.

• Detainees came from 56 countries, most commonly Nigeria and Algeria.

An adult at risk should be given special protection because they are particularly vulnerable. They should not usually be imprisoned, though they can be if the Home Office believes they pose a risk to the public or have a history of non-compliance with immigration law.

Of those represented in the Guardian survey, 27% had been tortured, 24% had serious health conditions and 4% were at risk of suicide.

The survey found just over half of detainees had served a prison sentence.

Alieu, a refugee from Gambia who was tortured in his home country, says that seven years after being detained in Harmondsworth immigration removal centre, near Heathrow, he is still suffering trauma.

“I kept asking the Home Office: ‘Am I a criminal, am I a prisoner?’ I was locked up in a very small space and was too scared to sleep. I’m still scared of people in uniform. The trauma from being locked up in detention after I’d already experienced torture will stay with me for the rest of my life.”

The investigation also uncovered multiple cases of children being held in the adult estate, despite this being banned in all but exceptional circumstances. Almost a third of adult detainees had dependent children in the UK, prompting concerns their removal would lead to families being separated.

Bail for Immigration Detainees, a charity that assists with detainees’ bail applications, condemned such separations, saying it causes children extreme distress.

“Many of our clients’ children have lost weight, suffered from recurring nightmares and experienced insomnia during their parents’ enforced absence,” said Celia Clarke, director of BID.

Kate Allen, director of Amnesty International UK, deplored the fact that the vast majority of detainees face open-ended imprisonment, adding: “That lack of an end date is causing serious harm, not only to those detained but also to their loved ones.”

Migration Watch, which monitors migration into the UK and has called for the detention estate to be expanded, said: “If people are here legally and they are being detained that’s a serious flaw in the system. It goes without saying that people who are here legally should not be detained.”

James Price, campaign manager at the TaxPayers’ Alliance expressed concern about the cost of detention: “Detention should only be used when there is a high chance of returning the individual in a short space of time, because a bureaucratic and lengthy wait is bad for the welfare of those detained, as well as costing taxpayers and meaning less money for essential services.”

The Home Office spokesman said: “We have made significant improvements to our approach in recent years, but it is clear we can go further.

“The home secretary has made clear that he is committed to going further and faster to explore alternatives to detention, increase transparency around immigration detention, further improve the support available for vulnerable detainees and initiate a new drive on detainee dignity.”

Regarding children, he added: “We ended the routine detention of children for immigration purposes in 2010, and fundamentally changed the system to ensure that the welfare of the child is at the heart of every decision we make.”

The methodology

The Guardian sent a series of questions to 15 organisations who work with detainees – law firms with Home Office contracts to represent detainees and specialist NGOs. We received responses from 11.

Our partner organisations provided anonymised data about a series of key metrics, including age, length of residence and family ties in the UK, length of detention and specific vulnerabilities.

We asked them to enter data about their entire client list on a single day, 31 August, but some did not have the resources to capture every detainee on their books.

After excluding a handful of potential double counts where an NGO and a law firm may have been working with the same detainee, we were left with 188 unique responses.

We then calculated the proportion of the group with certain characteristics, such as suicidal tendencies, dependent children and long-term residency.

The data should be treated as a snapshot and not as a sample representative of the whole population in immigration detention. Many detainees never have contact with any legal representative or NGO, and will not have been captured in our sample.

• In the UK, Samaritans can be contacted on 116 123 or email jo@samaritans.org. In the US, the National Suicide Prevention Lifeline is 1-800-273-8255. In Australia, the crisis support service Lifeline is 13 11 14. Other international suicide helplines can be found at www.befrienders.org.