Jacobs was born in Marksville, Louisiana, United States, and raised in Alexandria, Louisiana, where he first learned to play the harmonica. After quitting school by the age of 12, Jacobs left rural Louisiana and traveled around working odd jobs and busking on the streets of New Orleans, Memphis, Helena, Arkansas and St. Louis. He honed his musical skills on harmonica and guitar with Sonny Boy Williamson II, Sunnyland Slim, Honeyboy Edwards and others.

Arriving in Chicago in 1945, he occasionally found work as a guitarist but garnered more attention for his already highly developed harmonica work. According to fellow Chicago bluesman Floyd Jones, Little Walter's first recording was an unreleased demo recorded soon after he arrived in Chicago on which Walter played guitar backing Jones. Jacobs reportedly grew frustrated with having his harmonica drowned out by electric guitarists, and adopted a simple, but previously little used method: He cupped a small microphone in his hands along with his harmonica, and plugged the microphone into a public address or guitar amplifier. He could thus compete with any guitarist's volume. Unlike other contemporary blues harp players, such as Sonny Boy Williamson I and Snooky Pryor, who had also begun using the then new technology around the same time solely for added volume, Little Walter purposely pushed his amplifiers beyond their intended technical limitations, using the amplification to explore and develop radical new timbres and sonic effects previously unheard from a harmonica, or any other instrument. Madison Deniro wrote a small biographical piece on Little Walter stating that "He was the first musician of any kind to purposely use electronic distortion."

Jacobs made his first released recordings in 1947 for Bernard Abrams' tiny Ora-Nelle label, which operated out of the back room of Abrams' Maxwell Radio and Records store in the heart of the Maxwell Street market area in Chicago. These and several other early Little Walter recordings, like many blues harp recordings of the era, owed a strong stylistic debt to pioneering blues harmonica player Sonny Boy Williamson I (John Lee Williamson). Little Walter joined Muddy Waters' band in 1948, and by 1950, he was playing acoustic (unamplified) harmonica on Muddy's recordings for Chess Records; the first appearance on record of Little Walter's amplified harmonica sound was on Muddy's "Country Boy" / "Too Young To Know" (Chess 1452), recorded on July 11, 1951. For years after his departure from Muddy's band in 1952, Little Walter continued to be brought in to play on his recording sessions, and as a result his harmonica is featured on most of Muddy's classic recordings from the 1950s. As a guitarist, Little Walter recorded three songs for the small Parkway label with Muddy Waters and Baby Face Leroy Foster (reissued on CD as "The Blues World of Little Walter" from Delmark Records in 1993), as well as on a session for Chess backing pianist Eddie Ware; his guitar work was also featured occasionally on early Chess sessions with Muddy Waters and Jimmy Rogers.

Jacobs had put his career as a bandleader on hold when he joined Muddy's band, but stepped back out front once and for all when he recorded as a bandleader for Chess's subsidiary label Checker Records on 12 May 1952; the first completed take of the first song attempted at his debut session became his first hit, spending eight weeks in the #1 position on the Billboard magazine R&B charts - the song was "Juke", and it is still the only harmonica instrumental ever to become a #1 hit on the R&B charts. (Three other harmonica instrumentals by Little Walter also reached the Billboard R&B top 10: "Off the Wall" reached #8, "Roller Coaster" achieved #6, and "Sad Hours" reached the #2 position while Juke was still on the charts.) "Juke" was the biggest hit to date for Chess and its affiliated labels, and one of the biggest national R&B hits of 1952, securing Walter's position on the Chess artist roster for the next decade. Little Walter scored fourteen top ten hits on the Billboard R&B charts between 1952 and 1958, including two #1 hits (the second being "My Babe" in 1955), a level of commercial success never achieved by his former boss Waters, nor by his fellow Chess blues artists Howlin' Wolf and Sonny Boy Williamson II. Following the pattern of "Juke", most of Little Walter's single releases in the 1950s featured a vocal performance on one side, and an instrumental on the other. Many of Walter's vocal numbers were originals which he or Chess A&R man Willie Dixon wrote or adapted and updated from earlier blues themes. In general, his sound was more modern and uptempo than the popular Chicago blues of the day, with a jazzier conception than other contemporary blues harmonica players.

Jacobs was frequently utilized on records as a harmonica sideman behind others in the Chess stable of artists, including Jimmy Rogers, John Brim, Rocky Fuller, Memphis Minnie, The Coronets, Johnny Shines, Floyd Jones, Bo Diddley, and Shel Silverstein, and on other record labels backing Otis Rush, Johnny Young, and Robert Nighthawk.

Jacobs suffered from alcoholism and had a notoriously short temper which led to a decline in his fame and fortunes beginning in the late 1950s, although he did tour Europe twice, in 1964 and 1967. (The long circulated story that he toured the United Kingdom with The Rolling Stones in 1964 has since been refuted by Keith Richards). The 1967 European tour, as part of the American Folk Blues Festival, resulted in the only film / video footage of Little Walter performing that is known to exist. Footage of Little Walter backing Hound Dog Taylor and Koko Taylor on a television program in Copenhagen, Denmark, on 11 October 1967 was released on DVD in 2004. Further video of another recently discovered TV appearance in Germany during this tour, showing Little Walter performing his songs "My Babe", "Mean Old World", and others were released on DVD in Europe in January 2009, and is the only known footage of Little Walter singing; other TV appearances in the UK and the Netherlands have been documented, but no footage of these has been uncovered. Jacobs recorded and toured only infrequently in the 1960s, playing mainly in and around Chicago.

In 1967 Chess released a studio album featuring Little Walter with Bo Diddley and Muddy Waters titled Super Blues.

A few months after returning from his second European tour, he was involved in a fight while taking a break from a performance at a nightclub on the South Side of Chicago. The relatively minor injuries sustained in this altercation aggravated and compounded damage he had suffered in previous violent encounters, and he died in his sleep at the apartment of a girlfriend at 209 E. 54th St. in Chicago early the following morning. The official cause of death indicated on his death certificate was "coronary thrombosis" (a blood clot in the heart); evidence of external injuries was so insignificant that police reported that his death was of "unknown or natural causes", and there were no external injuries noted on the death certificate. His body was buried at St. Mary's Cemetery in Evergreen Park, IL on February 22, 1968.

His legacy has been enormous: he is widely credited by blues historians as the artist primarily responsible for establishing the standard vocabulary for modern blues and blues rock harmonica players. His influence can be heard in varying degrees in virtually every modern blues harp player who came along in his wake, from blues greats such as Junior Wells, James Cotton, George "Harmonica" Smith, Carey Bell, and Big Walter Horton, through modern day masters Sugar Blue, Billy Branch, Kim Wilson, Rod Piazza, William Clarke, and Charlie Musselwhite, in addition to blues - rock crossover artists such as Paul Butterfield, Southside Johnny (who named his band The Asbury Jukes after Little Walter's band), and John Popper of the band Blues Traveler. He was portrayed in the 2008 film, Cadillac Records, by Columbus Short.