In 1920 Nebe joined the Berlin detective force Kriminalpolizei, or Kripo (literally, "Criminal Police", a similar organization to the United Kingdom's Criminal Investigation Department) and attained the rank of Police Commissioner in 1924. Nebe joined the Nazi party on 1 July 1931. He became a contributing member of the SS on the same day. Nebe became the Nazis' liaison in the Berlin criminal police, with links to the SS group led by Kurt Daluege. In early 1932 Nebe and other Nazi detectives formed the NS (National Socialist) Civil Service Society of the Berlin Police. Following the Nazi seizure of power, Daluege recommended Nebe, in April 1933 to be Chief Executive of the State Police.

In October 1933 Nebe was ordered by Rudolf Diels, the then head of the Gestapo, to arrange the liquidation of Hitler's rival Gregor Strasser. This began the process of turning Nebe against the Nazis. When the Kripo was absorbed into the Security Police (Sicherheitspolizei) Nebe became an SS Gruppenführer. His aversion to Reinhard Heydrich and Himmler grew even though he continued to regularly lunch with them. In 1933 he came to know Hans Bernd Gisevius, then an official in the Berlin Police Headquarters and Gisevius introduced him to Hans Oster.

In July 1936, the Kripo became the Criminal Police Department for the entire Reich. It was merged, along with the Gestapo into the Sicherheitspolizei (SiPo). At that point, Reinhard Heydrich was in overall command of the SiPo (Gestapo and Kripo) and the SD. Nebe was appointed head of the Kriminalpolizei (Kripo). As head of the Kripo, Nebe reported to Heydrich. In September 1939, the Kripo would become the Criminal Police of Department V in the RSHA.

In 1938, Nebe joined forces with future fellow conspirator Dr. Karl Sack (Judge Advocate - General of the Wehrmacht) to spoil Himmler's plot against General Werner von Fritsch. That same year, Hans Oster recruited Nebe into the conspiracy for the September 1938 coup attempt, a plot to overthrow Hitler if he went to war with Czechoslovakia over the Sudetenland. Nebe supplied the conspirators with information regarding SS strength, logistics, and safehouses throughout the Berlin area.

In 1941, just prior to Operation Barbarossa, Heinrich Himmler selected Nebe to command Einsatzgruppe B behind Army Group Center in the east. Of his work in the east, historian Gerald Reitlinger stated:

Foreseeing the crimes in which he would be involved, he tried to escape by asking for a move to the International Police Commission but is said to have been persuaded by Ludwig Beck and Hans Oster to accept the appointment, which would place him in a position where he could give them information on what was happening inside the SS and the Gestapo. He worked with Henning von Tresckow and Fabian von Schlabrendorff to reduce the atrocities committed, and often massaged the numbers reported to his superiors (including one claim that his task force was responsible for more than 45,000 killings). He returned from Russia convinced that the war would end with the military defeat of Germany.

In late 1942 after the Wannsee Conference, Nebe informed his fellow conspirators of Himmler's plans for the so-called Final Solution.

In March 1944, after the 'Great Escape' from Stalag Luft III POW camp, Nebe was ordered by Heinrich Müller, Chief of the Gestapo (Amt IV, or Department 4, of the RSHA), to choose 50 of the 73 captured prisoners to be executed in the Stalag Luft III murders. It is reputed that this selection caused Nebe distress. Guenter Lewy, however, in his book The Nazi Persecution of the Gypsies, describes a different picture of Nebe. He describes how Nebe held a demonstration, on Himmler's request, of a mass shooting of 100 people in Minsk. On learning it was distasteful to the troops, Himmler wanted Nebe to come up with something less distressing. Nebe decided to try experimenting by murdering Soviet mental patients first with explosives near Minsk and then with automobile exhaust at Mogilev. Lewy also lays out other issues that question Nebe's innocence. Among other things:

• "As head of the RKPA," (the Kripo, or Criminal Police), "Nebe played a leading role in the formulation of Gypsy policy".
• Nebe told Eichmann to put Gypsies with the Jews on the transports to Nisko, in October 1939.
• In 1944, Nebe suggested to Grawitz that the Gypsies interned at Auschwitz would be good people to use for medical experiments at the Dachau concentration camp (Himmler had asked Grawitz for advice on the question).
• In September 1941, Nebe helped give a course named 'The Jewish Question with special attention to the partisan movement', which included the murdering of 32 people at Mogilev.
• Bernd Wehner of the RKPA claimed Nebe was worried the Allies would punish him for his crimes, and that this was the only reason he joined the resistance.