Hartmann came from a musical family of German descent. Although he received his music lessons initially from his father, he taught himself as much as possible. He complied with his father's wishes for him to study jurisprudence and consequently worked as a civil servant from 1829 to 1870, but also pursued an extensive musical career. By 1824, he was already the organist at the Garnisons Kirke in Copenhagen, and in 1832, he made his debut as a composer with the opera Ravnen (The Raven).

In 1836, he made his first study tour to Germany and France, where he made the acquaintance of such significant musical figures as Frédéric Chopin, Gioachino Rossini, Luigi Cherubini, and Louis Spohr. Spohr and the Danish composer Christoph Ernst Friedrich Weyse were Hartmann's most important mentors. Further journeys to Germany followed in the next few years, during which he also founded the Danish Musical Association in 1836, remaining its chairman until the end of his life. In 1843, he transferred from the Garnisons Kirke to play the organ for the Vor Frue Kirke, and became the director of the Student Choral Association. He also held both these posts until his death.

In 1867, after having taught at the Conservatory of Giuseppe Siboni since 1827, Hartmann helped co-establish and direct the Copenhagen Conservatory with Niels Gade and Holger Simon Paulli. During his lifetime, Hartmann occupied a central place in Danish musical life and was considered by many to be an absolute authority on musical matters. His son, Emil Hartmann, also became a composer; Niels W. Gade married J.P.E. Hartmann's daughter.

Hartmann's works were characterized by artistic seriousness, dramatic vitality, and in particular, by national coloring which appealed deeply to Danish audiences. The Nordic elements, which could be discerned in the themes based on folksongs, modulations, and the tendency towards rather dark sounds, emerged strongly after the 1830s. Hartmann united these Romantic influences with a strong control over both form and theme, acquired through his Classical training and often reminiscent of Felix Mendelssohn.