Raff was born in Lachen in Switzerland. His father, a teacher, had fled there from Württemberg in 1810 to escape forced recruitment into the military of that southwestern German state that had to fight for Napoleon in Russia. Joachim was largely self taught in music, studying the subject while working as a schoolmaster in Schmerikon, Schwyz and Rapperswil. He sent some of his piano compositions to Felix Mendelssohn who recommended them to Breitkopf & Härtel for publication. They were published in 1844 and received a favourable review in Robert Schumann's journal, the Neue Zeitschrift für Musik, which prompted Raff to go to Zürich and take up composition full time.

In 1845, Raff walked to Basel to hear Franz Liszt play the piano. After a period in Stuttgart where he became friends with the conductor Hans von Bülow, he worked as Liszt's assistant at Weimar from 1850 to 1853. During this time he helped Liszt in the orchestration of several of his works, claiming to have had a major part in orchestrating the symphonic poem Tasso. In 1851, Raff's opera König Alfred was staged in Weimar, and five years later he moved to Wiesbaden where he largely devoted himself to composition. From 1877 he was the first Director of, and a teacher at, the Hoch Conservatory in Frankfurt. There he employed Clara Schumann and a number of other eminent musicians as teachers, and established a class specifically for female composers (this was at a time when women composers were not taken very seriously). His pupils there included Edward MacDowell and Alexander Ritter. He died in Frankfurt on the night of June 24/25, 1882.

Raff was very prolific, and by the end of his life was one of the best known German composers, though his work is largely forgotten today (only one of his pieces, a cavatina for violin and piano, is performed with any regularity today, sometimes as an encore). He drew influence from a variety of sources - his eleven symphonies, for example, combine the Classical symphonic form, with the Romantic penchant for program music and contrapuntal orchestral writing which harks back to the Baroque. Most of these symphonies carry descriptive titles including In the Forest (No. 3), Lenore (No. 5) and To the Fatherland (No. 1), a very large scale work lasting around seventy minutes. His last four symphonies make up a quartet of works based on the four seasons. Arturo Toscanini conducted some performances of the Symphony No. 3 In the Forest in 1931.

The Lenore symphony (No. 5), famous in its time, was inspired by a ballad of the same name by Gottfried August Bürger that also inspired works by several other composers, including Maria Theresa von Paradis (1789), Henri Duparc, Franz Liszt (late 1850s, mentioned by Alan Walker in his Liszt biography vol. 2), for example. The world premiere recording of Lenore was made in 1965, by the London Philharmonic Orchestra conducted by Bernard Herrmann, who championed Raff's orchestral music. He described it as "one of the finest examples of the Romantic Programme School - it deserves a place alongside the Symphonie fantastique of Berlioz, Liszt's Faust Symphony and the Manfred Symphony of Tchaikovsky".

Richard Strauss was a pupil of Hans von Bülow, a friend of Raff's, and it has been said that Strauss was influenced in his early works by Raff. For example, Raff's Symphony No. 7 In the Alps (1877) could be compared with Strauss's An Alpine Symphony (1915). Much of Raff's music has been said to forecast the early works of Jean Sibelius.

Raff also composed in most other genres, including concertos, opera, chamber music and works for solo piano. His chamber works include two piano sonatas, five violin sonatas, a cello sonata, a piano quintet, two piano quartets, a string sextet and four piano trios. Many of these works are now commercially recorded. He also wrote numerous suites, some for smaller groups (there are suites for piano solo and suites for string quartet), some for orchestra and one each for piano and orchestra and violin and orchestra.