Jimmy Rogers was born James A. Lane in Ruleville, Mississippi, on June 3, 1924 and was raised in Atlanta and Memphis. He adapted the professional surname 'Rogers' from his stepfather's last name. Rogers learned the harmonica alongside his childhood friend Snooky Pryor, and as a teenager took up the guitar and played professionally in East St. Louis, Illinois, where he played with Robert Lockwood, Jr. among others, before moving to Chicago in the mid 1940s. By 1946 he had recorded as a harmonica player and singer for the Harlem record label run by J. Mayo Williams. Rogers' name did not appear on the record, which was mislabeled as the work of "Memphis Slim and his Houserockers."

In 1947, Rogers, Muddy Waters and Little Walter began playing together as Muddy Waters' first band in Chicago (sometimes referred to as "The Headcutters" or "The Headhunters" due to their practice of stealing jobs from other local bands), while the band members each recorded and released music credited to each of them as solo artists. The first Muddy Waters band defined the sound of the nascent "Chicago Blues" style (more specifically "South Side" Chicago Blues). Rogers made several more sides of his own with small labels in Chicago, but none were released at the time. He began to enjoy success as a solo artist with Chess Records in 1950, scoring a hit with "That's All Right", but he stayed with Muddy Waters until 1954. In the mid 1950s he had several successful releases on the Chess label, most featuring either Little Walter Jacobs or Big Walter Horton on harmonica, most notably "Walking By Myself", but as the 1950s drew to a close and interest in the blues waned, he gradually withdrew from the music industry.

In the early 1960s Rogers briefly worked as a member of Howling Wolf's band, before quitting the music business altogether for almost a decade. He worked as a taxicab driver and owned a clothing store that burned down in the Chicago riots that followed the assassination of Dr. Martin Luther King, Jr. in 1968. He gradually began performing in public again, and in 1971 when fashions made him a reasonable draw in Europe, Rogers began occasionally touring and recording, including a 1977 reunion session with his old bandleader Muddy Waters. By 1982, Rogers was again a full time solo artist.

In 1995 Rogers was inducted into the Blues Hall of Fame.

He continued touring and recording albums until his death from colon cancer in Chicago in 1997. He was survived by his son, James D. Lane, who is also a guitarist and a record producer and recording engineer for Blue Heaven Studios and APO Records.