The abduction of a former Libyan intelligence operative accused of preparing the bomb that brought down Pan Am flight 103 over Lockerbie, Scotland, in 1988 and his transfer into US custody was agreed about three months ago after conversations between US government and Libyan officials, the Guardian has been told.

Mohammed Abouagela Masud appeared briefly in court in Washington DC on Monday, accused of having set the timer for the bomb that destroyed the Boeing 747, killing 270 people in the most deadly terrorist attack to have taken place on British soil.

The US Department of Justice announced it had custody of Masud at the weekend, but gave no details of how he had arrived in the US.

Libyan officials with knowledge of the case said Masud was seized at his home in Tripoli’s Abu Salim neighbourhood on 17 November by gunmen loyal to Abdel Ghani al-Kikli, an infamous local militia commander known as “Gheniwa”, then handed over to a second militia group who detained him for two weeks before final transfer to US government agents.

Analysts have raised concern that the US choice of such notorious local partners in a clandestine operation to bring Masud to the US against his will may undermine efforts to promote the rule of law in Libya.

The officials said Trump administration officials had been in discussions with local authorities about bringing Masud to the US to stand trial since 2019, and that these “conversations” had continued under President Joe Biden.

Until six months ago, Masud was serving a 10-year sentence for crimes committed as an intelligence operative under the regime of Muammar Gaddafi, who was ousted in 2011.

While in prison, Masud, who is about 75 and seriously ill, could have been transferred relatively easily to US custody by the Tripoli-based Government of National Unity (GNU), but he was released earlier this year having served his term.

“He was basically a free man. He … was just at home. There was no warrant out for him or anything of that nature. The Americans knew this of course,” one Libyan official said.

The lack of legal justification for the detention of Masud initially appeared to have blocked the US efforts, until powerful individuals within the GNU, thought to be the influential nephew of the Libyan prime minister, Abdul Hamid Dbeibeh, offered to detain Masud and hand him over to the US government anyway.

“The Libyan leaders knew the Americans were interested, regardless of whether or not the individual was in custody. The legal process from the Libyan side was not of great concern,” said Jalel Harchaoui, associate fellow at the Royal United Services Institute.

Only a small number of senior officials in Washington, mainly in the ministry of justice and the state department, were aware of the fate planned for Masud, though the CIA and the White House are thought to have been notified.

After being abducted, Masud was transferred to a heavily armed paramilitary unit called the Joint Force in the port city of Misrata. The force was set up a year ago by Dbeibeh to act as a personal praetorian guard and is described by experts as “very well-armed, small, very nasty and able to get things done”.

“It seems he was … basically warehoused while they waited for the US to be ready and accept the gift,” said Harchaoui.

On around 1 December, US officials collected Masud and flew their captive to Malta against his will, the officials said.

Masud’s nephew, Abdulmenam Marimi, told Reuters on Monday that the family only learned he had been moved to the US from the media.

Libya is split between two administrations, and is the site of a proxy struggle for influence between countries including Egypt, Russia, the United Arab Emirates, Turkey and western nations. The GNU is based in Tripoli.

Dbeibeh’s mandate expired last December and so has been under pressure to justify remaining in power to the US and other western allies since.

“It’s very likely that Masud was seized on the orders of the current Tripoli-based prime minister, who needs to bolster his personal position with the US. Resentment from the international community towards him was building. He is supposed to be delivering the country to elections, and to be rooting out corruption networks, but there’s no sign of any genuine commitment from his side,” said Alia Brahimi, a north Africa expert at the Atlantic Council.

Earlier this year, Dbeibeh appointed al-Kikli, the notorious militia commander, to command a force that was given responsibility for arresting individuals in “national security” cases.

Al-Kikli, who has been accused repeatedly of human rights abuses, controls the poor, populous Abu Salim neighbourhood, where Masud lived.

Amnesty International said earlier this year that militias under al-Kikli’s command terrorised people “through enforced disappearances, torture, unlawful killings and other crimes under international law”. Al-Kikli denied Amnesty’s allegations, saying that he was “very keen to apply Libyan law first, which takes into account standards of justice and human rights”.

Jake Sullivan, the US national security adviser, told reporters at Monday’s White House press briefing: “Today is a good day because Masud will be facing justice for his alleged role in the 1988 Lockerbie bombing. I will say that this was done in a lawful manner according to established procedures.”

In a statement, Michael Glasheen, the acting assistant director in charge of the FBI Washington field office, said: “The lawful arrest and presentment in court of the alleged bombmaker … is the product of hard work and partnerships across the globe.”

Tim Eaton, an expert at Chatham House in London, said it was possible Washington had overruled concerned diplomats on the ground.

“The US has this idea that there is no moratorium on pursuing those responsible for crimes against America … but this episode … is completely at odds with what the US wants in Libya. The pitch is for democracy, accountability and transparency. You’d be very hard pushed to say this was an accountable and transparent process,” Eaton said.

Harchaoui said many Libyans would be dismayed by the US choice of local collaborators.

“What are you going to think if you live in Abu Salim and you are terrified of this militia and then you find out that the most powerful state in the world is working with it, accepting its illegal operation and giving it legitimacy? What are you going to say about the US?” he asked.

The Pan Am flight from Heathrow to John F Kennedy airport in New York exploded at 31,000ft over Scotland on 21 December 1988. Two hundred and 59 people were killed onboard, while fiery debris from the exploded plane killed another 11 on the ground in Lockerbie.

According to US legal documents, Masud was a key figure in the plot, along with Abdelbaset al-Megrahi, and Al Amin Khalifa Fhimah. Al-Megrahi was jailed for life for mass murder by three Scottish judges at a special court sitting in the Netherlands in 2001. Fhimah was later acquitted at a trial.

Investigators say Masud met the other two in Malta, where he had been directed to fly by a senior Libyan intelligence official with a prepared suitcase. He was asked to set the timer by the other two men, and the suitcase travelled via feeder flights to the hold of the Boeing 747.

At the time, Gaddafi was in conflict with the west, but under his leadership Libya later renounced terrorism and accepted responsibility for the aircraft bombing in 2003 in return for relief from economic sanctions.

After Gaddafi’s fall, Masud, a longtime explosives expert for the country’s intelligence service, was taken into custody by Libyan law enforcement. In 2017, US officials received a copy of an interview with Masud carried out by Libyan authorities soon after his arrest.

In that interview, US officials said, Masud admitted building the bomb used in the Pan Am attack and working with the two men charged earlier to plant it on the plane. He said the operation had been ordered by Libyan intelligence and that Gaddafi had thanked him and others after the attack, according to an FBI affidavit.

In late 2020, the US Justice Department announced charges against Masud. With Masud in Libyan custody, though, his prosecution remained largely theoretical.