Holley was born in Detroit, Michigan, United States. He attended the prestigious Cass Technical High School in Detroit. Holley played violin and tuba when young. He started playing bass while serving in the Navy, playing in the Ships Company A Band at Camp Robert Smalls, which was led by Leonard Bowden and included Clark Terry, and several other musicians recruited from civilian dance bands.In the latter half of the 1940s, he played with Dexter Gordon, Charlie Parker, and Ella Fitzgerald; in 1950 he and Oscar Peterson recorded duets, and he also played with Peterson and Charlie Smith as a trio. He was married to Minnie Walton (born Millicent Aitcheson). In the mid-1950s, he moved to England and worked at the BBC. Upon his return to America, he toured with Woody Herman in 1958 and with Al Cohn/Zoot Sims between 1959 and 1960. A prolific studio musician, he played with Duke Ellington in 1964 and with the Kenny Burrell Trio, Coleman Hawkins, Lee Konitz, Roy Eldridge, Michel Legrand, Milt Buckner, Jay McShann and Quincy Jones in the 1960s and 1970s. From 1967 to 1970, he taught at the Berklee College of Music. Holley was known for singing along with his arco (bowed) bass solos, a technique Slam Stewart also used. Holley and Stewart recorded two albums together. Holley died of a heart attack in Maplewood, New Jersey, at the age of 66.