Neumann was born in Joachimsthal, Margraviate of Brandenburg, located not far from Berlin. In 1815 he interrupted his studies at Berlin to serve as a volunteer in the Hundred Days against Napoleon, and was wounded in the Battle of Ligny. Subsequently he entered Berlin University as a student of theology, but soon turned to scientific subjects. His earlier papers were mostly concerned with crystallography, and the reputation they gained him led to his appointment as Privatdozent at the University of Königsberg, where in 1828 he became extraordinary, and in 1829 ordinary, professor of mineralogy and physics. His 1831 study on the specific heats of compounds included what is now known as Neumann's law: the molecular heat of a compound is equal to the sum of the atomic heats of its constituents.

Devoting himself next to optics, he produced memoirs which entitle him to a high place among the early searchers after a true dynamical theory of light. In 1832, by the aid of a particular hypothesis as to the constitution of the ether, he reached by a rigorous dynamical calculation results agreeing with those obtained by Augustin Louis Cauchy, and succeeded in deducing laws of double refraction closely resembling those of Augustin-Jean Fresnel. In studying double refraction, with his deduction of the elastic constants (on which the optical properties depend) Neumann employed the assumption that the symmetry of the elastic behavior of a crystal was equal to that of its form. In other words, he assumed that the magnitudes of the components of a physical property in symmetric positions are equivalent. This assumption substantially reduced the number of independent constants and greatly simplified the elastic equations. However, four decades passed before Neumann elaborated his application of symmetry in a course on elasticity in 1873. This principle was later formalized by his student Woldemar Voigt (1850 – 1918) in 1885: ‘‘the symmetry of the physical phenomenon is at least as high as the crystallographic symmetry,’’ which became a fundamental postulate of crystal physics known as ‘‘Neumann’s principle’’. In 1900, Voigt attributed this principle to Neumann’s 1832 paper even though, at most, all that was present in that work was an implicit assumption that the symmetry of the phenomenon was equal to that of the crystal. Bernhard Minnigerode (1837 – 1896), another student of Neumann, first expressed this relation in written form in 1887 in the journal Neues Jahrb. Mineral Geol. Paleontol. (Vol. 5, p. 145).

Later, Neumann attacked the problem of giving mathematical expression to the conditions holding for a surface separating two crystalline media, and worked out from theory the laws of double refraction in strained crystalline bodies. He also made important contributions to the mathematical theory of electrodynamics, and in papers published in 1845 and 1847 established mathematically the laws of the induction of electric currents. His last publication, which appeared in 1878, was on spherical harmonics (Beitrage zur Theorie der Kugelfunctionen).

With the mathematician Carl Gustav Jacobi, he founded in 1834 the mathematisch-physikalisches seminar which operated in two sections, one for mathematics and one for mathematical physics. Not every student took both sections. In his section on mathematical physics Neumann taught mathematical methods and as well as the techniques of an exact experimental physics grounded in the type of precision measurement perfected by his astronomer colleague Friedrich Wilhelm Bessel. The objective of his seminar exercises was to perfect one's ability to practice an exact experimental physics through the control of both constant and random experimental errors. Only a few students actually produced original research in the seminar; a notable exception was Gustav Robert Kirchhoff who formulated Kirchhoff's Laws on the basis of his seminar research. His seminar was the model for many others of the same type established after 1834, including that of Gustav Robert Kirchhoff at Heidelberg University.

Neumann retired from his professorship in 1876, and died at Königsberg (now Kaliningrad, Russia) in 1895 at the age of 96.

His children were talented. His son, Carl Gottfried Neumann (1832 - 1925), became in 1858 Privatdozent, and in 1863 extraordinary professor of mathematics at Halle. He was then appointed to the ordinary chair of mathematics successively at Basel (1863), Tübingen (1865) and Leipzig (1868).