Diesel was born in Paris, France in 1858 as the second of three children to Theodor and Elise Diesel. Diesel's parents were German-born immigrants living in France. Theodor Diesel, a bookbinder by trade, had left his home town of Augsburg, Kingdom of Bavaria, in 1848. Theodor met his wife, Elise Strobel, daughter of a Nuremberg merchant, in Paris in 1855 and himself became a leather goods manufacturer there. Diesel spent his early childhood in France, but as a result of the outbreak of the Franco-Prussian War in 1870, the family (like many foreigners) was forced to leave and emigrated to London. Before the end of the war, however, Diesel's mother sent 12-year-old Rudolf to Augsburg to live with his aunt and uncle, Barbara and Christoph Barnickel, so that he might learn to speak German and visit the Königliche Kreis-Gewerbsschule [Royal County Trade School], where his uncle taught mathematics. At age 14, Rudolf wrote to his parents that he wanted to become an engineer, and after finishing his basic education at the top of his class in 1873, he enrolled at the newly-founded Industrial School of Augsburg. Later, in 1875, he received a merit scholarship from the Royal Bavarian Polytechnic of Munich which he accepted against the will of his perennially impecunious parents who would rather have seen him begin earning money.

In Munich, one of his professors was Carl von Linde. Diesel was unable to graduate with his class in July 1879 because of a bout of typhoid. While he waited for the next exam date, he gathered practical engineering experience at the Gebrüder Sulzer Maschinenfabrik [Sulzer Brothers Machine Works] in Winterthur, Switzerland. Diesel graduated with highest academic honors from his Munich alma mater in January 1880 and returned to Paris, where he assisted his former Munich professor Carl von Linde with the design and construction of a modern refrigeration and ice plant. Diesel became the director of the plant a year later.

In 1883, Diesel married Martha Flasche, and continued to work for Linde, garnering numerous patents in both Germany and France. In early 1890, Diesel moved his wife and their three children (Rudolf Jr, Heddy, and Eugen) to Berlin to assume management of Linde's corporate research and development department and to join several other corporate boards there. Because he was not allowed to use the patents he developed while an employee of Linde's for his own purposes, Diesel sought to expand into an area outside refrigeration. He first toyed with steam, his research into fuel efficiency leading him to build a steam engine using ammonia vapor. During tests, this machine exploded with almost fatal consequences and resulted in many months in the hospital and a great deal of ill health and eyesight problems. He also began designing an engine based on the Carnot cycle, and in 1893, soon after Gottlieb Daimler and Karl Benz had invented the automobile in 1887, Diesel published a treatise entitled Theorie und Construktion eines rationellen Wärmemotors zum Ersatz der Dampfmaschine und der heute bekannten Verbrennungsmotoren [Theory and Construction of a Rational Heat-engine to Replace the Steam Engine and Combustion Engines Known Today] and formed the basis for his work on and invention of the diesel engine.

Diesel understood thermodynamics and the theoretical and practical constraints on fuel efficiency. He knew that even very good steam engines are only 10-15% thermodynamically efficient, which means that up to 90% of the energy available in the fuel is wasted. His work in engine design was driven by the goal of much higher efficiency ratios. He tried to design an engine based on the Carnot Cycle. However, he gave up on this and tried to develop his own approach. Eventually he designed his own engine and obtained a patent for his design. In his engine, fuel was injected at the end of compression and the fuel was ignited by the high temperature resulting from compression. In 1893, he published a book in German with the title "Theory and design of a rational thermal engine to replace the steam engine and the combustion engines known today" (English translation of the original title in German) with the help of Springer Verlag, Berlin. He managed to build a working engine according to his theory and design. His engine and its successors are now known as diesel engines. From 1893 to 1897, Heinrich von Buz, director of MAN AG in Augsburg, gave Rudolf Diesel the opportunity to test and develop his ideas. Rudolf Diesel obtained patents for his design in Germany and other countries, including the USA.

In the evening of 29 September 1913, Diesel boarded the post office steamer Dresden in Antwerp on his way to a meeting of the Consolidated Diesel Manufacturing company in London. He took dinner on board the ship and then retired to his cabin at about 10 p.m., leaving word for him to be called the next morning at 6:15 a.m. He was never seen alive again. Ten days later, the crew of the Dutch boat "Coertsen" came upon the corpse of a man floating in the sea. The body was in such an advanced state of decomposition that they did not bring it aboard. Instead, the crew retrieved personal items (pill case, wallet, pocket knife, eyeglass case) from the clothing of the dead man, and returned the body to the sea. On 13 October these items were identified by Rudolf's son, Eugen Diesel, as belonging to his father.

There are various theories to explain Diesel's death. Grosser (1978) presents a credible case for suicide. There are conspiracy theories that suggest that various people's business interests may have provided motives for homicide. Evidence is limited for all explanations.