Andrew Purcell talks to Herbie Hancock

As a Nichiren Buddhist, Herbie Hancock starts and ends each day with an invocation of the mystic law of cause and effect. He sits facing a sacred scroll, rings a bell, recites two chapters of the Lotus Sutra and chants "nam myoho renge kyo" - meaning that in life there are threads unseen and every action has consequences. His cadence is a calming, measured monotone you could set your watch or tune your piano by.

I know this because I can hear him through the door of his hotel room. When his daughter Jessica lets me in, he is seated at a makeshift altar on the breakfast bar, eyes closed, singing his pitch-perfect mantra. His faith and his musical philosophy are indivisible. "All you have to do is play one note," he tells me. "But it needs to be the right note." He relates this adage in a rasping Miles Davis whisper. Hancock is an accomplished composer, a musical pioneer of rare courage and a phenomenally gifted soloist, but to many jazz fans he will always be a sideman in Davis's second great quintet. He knows it, too. He quotes his mentor six times in the hour we spend together.

Hancock's bearing is serene. Until you see him play live, attacking the keys in energetic runs and arpeggios, you would never know his pulse rises above andante. Discreet wires connect the hearing aids he has worn since he "started losing the top end". His matte-black hair has grey roots but he looks nothing like 68 years old. His hands are young and supple, thanks to his regime of playing scales for hours on end at home in Los Angeles. He's in New York rehearsing with the band for his European tour: trumpeter Terence Blanchard, harmonica player Gregoire Maret, bassist James Genus, Lionel Loueke on guitar and Kendrick Scott on drums.

Hancock started learning the piano at the age of seven, along with his brother. Waymon was three years older, could throw an American football, hit a baseball and flick marbles expertly. "Before, I was the inept one," Herbie recalls. "For the first time, it was an even playing field. From that point on, I didn't go out and play with my brother any more." Within four years, he was good enough to perform the first movement of a Mozart piano concerto with the Chicago Symphony Orchestra.

By the time he joined Davis in 1963, Hancock already had a growing reputation as a band leader in his own right. One of the six originals on his 1962 debut album, Takin' Off, caught the ear of Cuban percussionist Mongo Santamaria, who rode its gospel groove into the hit parade. Watermelon Man has since become a standard, recorded hundreds of times, but the royalties from that first cover set Hancock up for a career in which he has alternately taken risks and played it safe, releasing bold, experimental music as well as straightforwardly commercial pop.

Outside jazz, he is best known for the 1973 funk album Head Hunters and 1983's electro smash Rockit, with its video of headless mannequins goose-stepping and moonwalking through a house. To purists who want retreads of his early artistic statements, this obsession with new technologies is a dead end. If Miles were still alive, the demands for a reunion would be unbearable.

The first time Hancock met Davis, at the trumpeter's house, Davis barely played at all, preferring to hide upstairs and listen to Hancock, bassist Ron Carter, drummer Tony Williams and saxophonist George Coleman through the intercom. Four rehearsals later, he set a date at Columbia studios to record Seven Steps to Heaven. "That's when I finally asked the question, 'Does that mean I'm in the band?' Miles gave me this sly smile and said, 'We're making a record, motherfucker.'" The coarse whisper dissolves into a laugh. Herbie enjoys his impersonation too much to keep a straight face.

The young rhythm section soon developed a telepathic understanding. For a while, performing the 1950s material that made Davis famous, Hancock was content to imitate his predecessors on the piano stool. "It was like I was holding back," he says. "Something wasn't getting satisfied in me." One night in Chicago, he decided to play his own way, clashing with the soloists, who had come to expect a more mannered style.

"I thought Miles was gonna fire me. But when we're walking off the stage, he says, 'Why didn't you play like that before?' and I realised if he wanted Bill Evans or Wynton Kelly, he would have hired them. His attitude was total encouragement for us to bring out our own personalities."

Over the course of six albums, with the addition of saxophonist Wayne Shorter, the Davis band elevated modal improvisation to new heights, playing with breathtaking originality at breakneck speed, in a system that came to be known as "time no changes" for the way they dispensed with chord schemes altogether.

The five men pushed and challenged each other, keeping structure to a bare minimum, in one of the greatest expressions of democracy in jazz. As Davis put it in his autobiography: "You get the right guys to play the right things at the right time and you got a motherfucker."

That is the lesson Hancock values most. He has trusted every one of his bands to express themselves. In Mwandishi, the fusion group he formed in 1969 after Davis sacked him (his replacement in the Davis group was Chick Corea), he gave synth programmer Patrick Gleeson licence to experiment. For Future Shock, the album that spawned Rockit, he ceded to Bill Laswell's electro-inspired production. As an instinctive collaborator, Hancock is well suited to the current vogue for vast corporate projects fronted by a parade of marketable singers. On the album Possibilities, he backed Christina Aguilera, Damien Rice, Joss Stone, Paul Simon and Annie Lennox. He makes no distinction between that and his most avant-garde work, insisting that both involve stepping out of his "comfort zone".

When his collection of Joni Mitchell covers, River, was named best album at the Grammys last year, it was only the second jazz disc to win the accolade, four decades after Stan Getz's Getz/Gilberto combined soft saxophone with Brazilian cool. Just like Getz's bossa nova classic, a wine bar on vinyl, River is an exquisitely palatable record. Hancock's arrangements are harmonically inventive, as always, but the drums are never louder than a murmur and the singers - including Norah Jones, Corinne Bailey Rae, Tina Turner and Mitchell herself - give restrained, sensitive performances.

"The reason it sounds comfortable is because I'm comfortable being uncomfortable," Hancock says. "It's not easy to play in a framework that requires simplicity and to tastefully find ways to interject the kind of freedom that we have in playing jazz. My desire was to walk that tightrope and that's not easy to do."

Accusations of selling out have been a constant feature of Hancock's career from the moment Maiden Voyage was commissioned as a jingle for Yardley aftershave. When he recorded the three Mwandishi albums, people said he was responding to the way Davis's 1970 double album Bitches Brew had crossed over with rock audiences. When he abandoned that fiercely experimental, all-Buddhist band to make Head Hunters, critics charged that this too was a purely commercial decision. Hancock admits there is some truth in this. "We did that - carving out new territory, exploring, but I got tired of it," he says. "No, I didn't express that well. I got tired of it only being that. There was something in me that wasn't quite being satisfied." Head Hunters was a way to make dance music, to let in the James Brown and Sly Stone influence he had previously filtered out, but it was also a search for wider relevance. In the sleeve notes for the Mwandishi box set, he confessed he was tired of seeing his records on the shelves but never hearing them played. Nevertheless, the last two of the Mwandishi albums, Crossings and Sextant, are among his most influential, respected by fans of prog rock, electronica, hip-hop and acid jazz.

His new greatest hits collection, The Definitive Herbie Hancock, merely demonstrates that there is no such thing as a definitive reading of the man. He has had far too many incarnations to cram on to one coherent playlist. It includes Cantaloupe Island, Chameleon, a funk workout he wrote for a Bill Cosby cartoon show, an appearance by Stevie Wonder, a Nirvana cover and a live version of Rockit stranded halfway between jazz and electro.

Hancock supported Barack Obama's presidential campaign because he values an open mind above all else. He doesn't talk about civil rights or the historic nature of Obama's candidacy. Nor does he mention the Democrat's policies. But he knows a fellow consensus-builder when he sees one. "If you ask him a question about something and he talks about his viewpoint, he also talks about the opposing viewpoint, shows where it came from, what value it could have," Hancock says. "That's the kind of guy the world needs. In my own way, I try to be that kind of person, too."

So, I ask him, when it is his name in lights outside the theatre, how does he ensure that equality reigns? There must be a tendency for the other players - even musicians as renowned as Blanchard - to subconsciously defer to him as band leader. Who pushes Herbie the way he pushed Miles?

"Me. I do that," he says. "So does everybody else. That's what's democratic. I encourage everyone to participate. I trust our ability to find a solution to whatever happens musically. And if we don't, it's not that big a deal. Nobody's going to die." His voice drops to a ragged whisper one last time. "Like Miles used to say, 'It's only My Funny Valentine.'"

• The Herbie Hancock Sextet play the Barbican on November 19 as part of the London jazz festival. Then and Now: The Definitive Herbie Hancock is released on Universal Classics on November 10