Olivier was the son of the teacher Prof. Ferdinand Olivier, the Elder, and his wife, opera singer Louise Niedhart. Like his brothers Frederick Olivier and Olivier Heinrich Ferdinand Olivier, he enjoyed in the years 1801 - 1802 art lessons by Karl Wilhelm Kolbe and Johann Christian Haldenwang (1777 - 1831).

In the years 1802 to 1803 he studied in Berlin woodcut techniques by Johann Friedrich Unger. From summer 1804 to summer 1806 Olivier went along with his brother Henry to Dresden, where they mostly copied the landscapes of Claude Lorrain and Jacob van Ruisdael Izaaksoon. In addition, Olivier studied Jakob Wilhelm Mechau and Karl Ludwig Kaaz. Through his acquaintance with Friedrich August von Klinkowstrom, he became acquainted with Philipp Otto Runge and Caspar David Friedrich.

Olivier was Dessau when the war of 1807 started. In the same year he joined as Secretary the Prussian representative in Paris. 

In the summer of 1810 Olivier undertook together with his brother Frederick a painting trip . The resulting works show clearly the influence of Caspar David Friedrich. In 1811 Olivier traveled via Dresden to Vienna. There he met Joseph Anton Koch. The following year he married Margaret Heller, a widow with three children. In Vienna Olivier had a religious awakening, a deepening of his Protestantism faith. His studio became the center of Protestant artists in Vienna, where they established friendly exchanges with the Catholic Nazarenes of the circle of August Wilhelm Schlegel. In 1816 Olivier was made member of the Guild of St Luke. With Olivier's family lived also his brother Henry Olivier, and Julius Schnorr von Carolsfeld.

After the liberation wars Olivier traveled together with Philipp Veit through Salzburg, and they discovered this land through the German landscape painting. On a second trip to this country he was accompanied by his brother Friedrich Olivier, Carl Ludwig Frommel, Johann Christian Rist and Julius Schnorr von Carolsfeld.

During 1830 Olivier went to Munich and was at the intercession of Peter Cornelius appointed as the successor of Theobald Schorn (1866 - 1913) as Secretary of the Munich Academy. In 1833 Olivier became a professor of art history.

He died at the age of 56 on 11 February 1841 in Munich.

Although Olivier never resided in Rome, he is still regarded as a leading exponent of the Nazarene movement. In his later works he also idealized themes of Dughet Gaspard and Nicolas Poussin.