Williams played with many of the top big bands of the '20s, '30s, and '40s, yet beginning in 1940 he did some of his finest work as a member of Sidney Bechet-led small bands. Williams moved to Washington, D.C. as a small child. After the death of his parents he was sent to an industrial school in Delaware, where he played in the band. He returned to Washington after two years and studied trombone with Juan Tizol and James Miller, Jr. He played locally around Washington, including a residency in the pit band at the Lincoln Theater. He played with Claude Hopkins in New Jersey in 1927, just before moving to New York, where he stayed only briefly. He worked back in Washington for a time, then joined Horace Henderson's band in 1929. He played regularly with that band for the next three years while also working occasionally with Hopkins, Cliff Jackson, and Horace's brother Fletcher Henderson. He joined Fletcher's band full-time in 1932, staying for about a year before moving on to Chick Webb's orchestra.

Williams stayed with the Webb outfit after the drummer's death in 1939 when Ella Fitzgerald assumed the leadership. He left in early 1940. Over the next year he spent time with both the Benny Carter and Coleman Hawkins bands; he also worked with Sidney Bechet, recording with the clarinetist/soprano saxophonist for the first time in June, 1940. Williams had limited experience in small groups; his solos were typically fine, but it reportedly took him some time to get the hang of the ensemble work. Nevertheless, his fresh approach to the early jazz repertoire satisfied Bechet, who used him on sessions and gigs over the next few years.

In the early '40s Williams worked for other leaders, including Lucky Millinder, Cootie Williams, Mezz Mezzrow, Pete Brown, and Wild Bill Davison. In 1943 he spent time with Duke Ellington, temporarily replacing Lawrence Brown. Over the next several years he also worked with Oran Hot Lips Page, Don Redman, Rex Stewart, Claude Hopkins, and Roy Eldridge. On December 15, 1945 Williams played with Bechet's band at a Town Hall concert celebrating the Blue Note record label. Health problems caused him to eventually stop performing by the early '50s. He began gigging again occasionally in the late '50s, but from the '60s dental problems hindered his musical activity. ~ Chris Kelsey, Rovi