He was born in Dresden, Saxony, in 1887, the son of an Alsatian pastor of the French Protestant Church. He entered the artillery in 1907. In World War I, he served on the Western Front until 1916, when he was appointed as Captain to the German General Staff. After the war, he was thought well enough to be kept in the reduced Reichswehr, whose officer corps was limited to 4,000 by the Treaty of Versailles. However, he had to resign from the army in 1932, when he got into trouble because of an indiscretion during the Carnival in the demilitarized zone of the Rhineland, where Reichswehr officers were prohibited. He soon found a job in a new organization which Hermann Göring set up under the Prussian police. He transferred to the Abwehr in October 1933. It was in this connection that he met future conspirators Hans Bernd Gisevius and Arthur Nebe, who were then working in the Gestapo. Oster also became a close confidant of Wilhelm Canaris, the head of the Abwehr.

Like many other army officers, Oster initially welcomed the Nazi regime, but his opinion soon soured after the 1934 "Night of the Long Knives", in which the Schutzstaffel (SS) extrajudicially murdered many of the leaders of the rival Sturmabteilung (SA) and their political opponents, including General Kurt von Schleicher, last Chancellor of Weimar Republic, and Major-General Ferdinand von Bredow, former head of the Abwehr. In 1935, Oster was allowed to reenlist in the army, but never on the General Staff. By 1938, the Blomberg-Fritsch Affair and Kristallnacht (state-sanctioned pogrom of Jews in Germany) turned his antipathy into a hatred of Nazism. In the course of the Fritsch crisis, Oster met General Ludwig Beck, Chief of General Staff, for the first time.

Oster's position in the Abwehr was invaluable to the conspiracy. The Abwehr could provide false papers and restricted materials, provide cover by disguising conspiratorial activities as intelligence work, link various resistance cells that were otherwise disparate, and supply intelligence to the conspirators. He also played a central role in the first military conspiracy to overthrow Hitler, which was rooted in Hitler's intention to invade Czechoslovakia. In August 1938, Beck spoke openly at a meeting of army generals in Berlin about his opposition to a war with the western powers over Czechoslovakia. When Hitler was informed of this, he demanded and received Beck’s resignation. Beck was highly respected in the army and his removal shocked the officer corps. His successor as Chief of Staff, Franz Halder, remained in touch with him and also with Oster. Privately, he said that he considered Hitler “the incarnation of evil.” During September, plans for a move against Hitler were formulated, involving Field Marshal Erwin von Witzleben, the army commander of the Berlin Military Region, and thus well-placed to stage a coup.

Oster, Gisevius and Schacht urged Halder and Beck to stage an immediate coup against Hitler, but the army generals argued that they could mobilize support among the officer corps for such a step only if Hitler made overt moves towards war. Halder nevertheless asked Oster to draw up plans for a coup. It was eventually agreed that Halder would instigate the coup when Hitler committed an overt step towards war. Therefore, emissaries of the conspirators traveled to Great Britain with assistance of Oster's Abwehr to urge the British to stand firm against Hitler over the Sudeten crisis. On 28 September, however, Prime Minister Neville Chamberlain agreed to a meeting in Munich, at which he accepted the dismemberment of Czechoslovakia. Hitler's diplomatic triumph undermined and demoralized the conspirators. Halder would no longer support a coup. This was the nearest approach to a successful conspiracy against Hitler before the 20 July plot of 1944.

As war again grew more likely in mid 1939, the efforts for a pre-emptive coup were revived. Oster was still in contact with Halder and Witzleben. But many officers, particularly those from Prussian landowning backgrounds, were strongly anti-Polish and saw a war to regain Danzig and other lost eastern territories as justified.

After the outbreak of World War II, resistance in the army became more problematic since it could lead to defeat of Germany. However, when Hitler decided to attack France soon after the Polish campaign in 1939, Halder along with other ranking generals thought it to be hopelessly unrealistic and again entertained the idea of coup, urged by Oster and Canaris. However, when Hitler vowed to destroy the "spirit of Zossen" (Zossen was where headquarters of Army High Command was located), by which he meant defeatism, Halder feared that their conspiracy was about to be discovered and destroyed all incriminating documents. Meanwhile, in an act that even other conspirators would have regarded as treason, Oster informed his friend Bert Sas, the Netherlands' military attaché in Berlin, more than twenty times of the exact date of the repeatedly delayed invasion of the Netherlands. Sas passed the information to his government, but was not believed. Oster calculated that his "treason" could cost lives of 40,000 German soldiers and wrestled with his decision, but concluded that it was necessary to prevent millions of deaths that would occur in what would be undoubtedly a protracted war should Germany achieve an early victory.

The period between 1940 and 1942 was the nadir of German resistance. Some officers were pleased to be wrong in their fear of military disaster. Others still opposed Hitler and the Nazi regime, but felt that his enormous popularity with the people made any action impossible. Tireless, Oster nevertheless succeeded in rebuilding an effective resistance network. In 1941, when the systematic extermination of Europeans Jews began with the invasion of Soviet Union, his Abwehr group established contact with Henning von Tresckow's resistance group in Army Group Center. In 1942, his most important recruit was General Friedrich Olbricht, head of the General Army Office headquartered at the Bendlerblock in central Berlin, who controlled an independent system of communications to reserve units all over Germany. Oster's Abwehr group supplied English-made bombs to Tresckow's group for their various attempts to assassinate Hitler in 1943.

However, later in 1943, the Abwehr group's efforts to rescue Jews were discovered by the Gestapo and Oster was dismissed from his post. Hans von Dohnanyi, who joined the Abwehr shortly before war, and Dietrich Bonhoeffer, the famous Lutheran theologian and Dohnanyi's brother-in-law, helped 14 Jews to flee to Switzerland disguised as Abwehr agents in Operation U-7. Dohnanyi and Bonhoeffer were arrested on charges of alleged breach of monetary exchange laws, amongst others with the leading German insurance brokers Jauch & Hübener, Captain Walter Jauch of the Jauch family being a first cousin-in-law of Oster and Otto Hübener later being hanged. Oster was placed under house arrest. Their involvement in the German resistance was discovered after the failure of the 20 July Plot. Oster was arrested one day after the failed 20 July Plot to assassinate Hitler. On April 4, 1945, the diaries of Admiral Canaris were discovered and, in a rage upon reading them, Hitler ordered that the conspirators be executed.

On April 8, 1945, Oster, Dietrich Bonhoeffer, Wilhelm Canaris, and other anti-Nazis were convicted and sentenced to death by an SS drumhead court-martial presided over by Otto Thorbeck. At dawn the next day, Oster, Bonhoeffer and Canaris were hanged in Flossenbürg concentration camp. They were forced to strip naked before being taken to the gallows. The camp was liberated two weeks later by American forces.

Fabian von Schlabrendorff, one of the few major coordinators of anti-Nazi activities to survive the war, described Oster as "a man such as God meant men to be, lucid and serene in mind, imperturbable in danger."