After the Civil War Rokossovsky studied at the Frunze Military Academy and became a senior cavalry commander in the Red Army. During the 1920s his division was stationed in Mongolia. In 1929 — by agreement with the Chinese government — he took part in defending the Chinese Eastern Railway against warlords.

It was in the early 1930s that Rokossovsky's life first became intertwined with Georgy Zhukov (later Marshal of the Soviet Union), when Rokossovsky was the commander of the 7th Samara Cavalry Division, and Zhukov a brigade commander under him. A sense of the nature of the beginning of their famous World War II rivalry can be gathered from reading Rokossovsky's comments on Zhukov in an official report on his character:

"Disciplined. Demanding and persistent in his demands. A somewhat ungracious and not sufficiently sympathetic person. Rather stubborn. Painfully proud. In professional terms well trained. Broadly experienced as a military leader."

Rokossovsky was among the first to realize the potential of armored assault. He was an early supporter of the creation of a strong armored core for the Red Army as championed by Marshal Mikhail Tukhachevsky in his theory of "deep operations".

Rokossovsky held senior commands until 1937, when he became caught up in Joseph Stalin's Great Purge and accused of "connections with foreign intelligence". His association with the cutting edge methods of Marshal Tukhachevsky may have been the real cause of his conflict with more traditional officers such as Semyon Budenny who still favored cavalry tactics, and whose policy disagreements with Tukhachesvky triggered the Great Purge of the Red Army that resulted in the execution of the latter, among many others. Rokossovsky, however, survived. After interrogations that included torture resulting in nine missing teeth, three cracked ribs, the removal of his fingernails, and three mock shooting ceremonies, he was sent to the Kresty Prison in Leningrad, where he remained until March 1940, when he was released without explanation. Rokossovsky first revived in the so-called "Villa of ecstasy" in the spa of Sochi on the coast of the Black Sea. After a brief talk with Stalin he was restored to the rank of a Major General under Kirponos in the Kiev Military Region — renamed the Southwestern Front at the outbreak of war.

When Nazi Germany attacked the Soviet Union in June 1941 Rokossovsky was serving as the commander of the 9th Mechanized Corps, where his command participated in the Battle of Dubno -- an early Soviet counter-attack that is credited with significantly delaying the rapid advance of the Von Rundstedt's Army Group South into the Ukraine. Notably on this occasion, Rokossovsky unilaterally resolved a dispute between Kirponos, the commander of the Southwestern Front, and his superior, Chief of General Staff G.K. Zhukov, by refusing a direct order, saying: "We had once again received an order to counterattack. However, the enemy outnumbered us to such a degree, that I took on the personal responsibility of ordering to halt the counteroffensive and to meet the enemy in prepared defenses." Zhukov had wanted the counteroffensive to continue. Nonetheless, Rokossovsky was promoted and became the commander of the 4th Army stationed near Smolensk in July under Timoshenko and then in September 1941 appointed to the command of 16th Army which was composed almost entirely of soldiers serving in penal battalions and charged with defending the approach to Moscow. Rokossovsky was now under the direct command of General Georgy Zhukov, his former subordinate.

The 16th Army (later renamed the 11th Guards Army) played a key role in the Battle of Moscow when it was deployed along the main axis of the German advance along the Volokolamsk Highway that was a central junction of the bitter fighting during the German winter offensive of 1941, (Operation Typhoon) as well as the subsequent Russian counter attack of 1941 - 42. On November 18, during the desperate last ditch efforts of the Wehrmacht to encircle Moscow in 1941, General Rokossovsky, his soldiers under heavy preassure from Hoepner's 4th Panzer Group, asked his immediate superior, Zhukov, if he could withdraw the 16th Army to more advantageous positions. Zhukov categorically refused. Rokossovsky went over Zhukov's head, and spoke directly to Marshal Boris Shaposhnikov, now Chief of the General Staff in Zhukov's place; reviewing the situation Shaposhnikov immediately ordered a withdrawal. Zhukov reacted at once. He revoked the order of the superior officer, and ordered Rokossovsky to hold the position. In the immediate aftermath, Rokossovsky's army was pushed aside and the 3rd and 4th Panzer Groups were able to gain strategically important positions north of Moscow, but this marked the high point of the German advance upon Moscow. Throughout Operation Typhoon, Rokossovsky's 16th army had taken the brunt of the German effort to capture Moscow.

In early 1942 Rokossovsky was transferred to the Bryansk Front. He commanded the right flank of the Soviet forces as they fell back before the Germans towards the Don and Stalingrad in the summer of 1942. During the Battle of Stalingrad Rokossovsky, commanding the Don Front, led the northern wing of the Soviet counter attack that encircled Paulus' Sixth Army and won the decisive victory of the Soviet - German war.

In 1943, after becoming commander of the Central Front, Rokossovsky successfully conducted defensive operations in the Kursk salient, and then led the counterattack west of Kursk which defeated the last major German offensive on the eastern front and allowed the Soviet armies to advance to Kiev. The Central Front was then renamed 1st Belorussian Front, which he commanded during the Soviet advance through Byelorussia (Belarus) and into Poland.

In a famous incident during the planning in 1944 of Operation Bagration, Rokossovsky conflicted with Stalin who demanded in accordance with Soviet war practice a single breakthrough of the German frontline. Rokossovsky held firm in his argument for two breakthroughs. Stalin ordered Rokossovsky to "go and think it over" three times, but every time he returned and gave the same answer "Two breakthroughs, Comrade Stalin, two breakthroughs." After the third time Stalin remained silent, but walked over to Rokossovsky and put a hand on his shoulder. A tense moment followed as the whole room waited for Stalin to rip the epaulette from Rokossovsky's shoulder; instead, Stalin said "Your confidence speaks for your sound judgement," and ordered the attack to go forward according to Rokossovsky's plan. The battle was successful, and Rokossovsky's reputation was assured. After crushing German Army Group Centre in Belarus, Rokossovsky's armies reached the east bank of the Vistula opposite Warsaw by mid 1944. For these victories he gained the rank of Marshal of the Soviet Union. Stalin once said: "I have no Suvorov, but Rokossovsky is my Bagration."

While Rokossovsky's forces stood stalled on the Vistula, the Warsaw Uprising (August - October, 1944) broke out in the city, led by the Polish Home Army (AK) on the orders of the Polish government in exile in London. Rokossovsky did not order reinforcements to the insurgents. Soviet assistance was limited to airdrops. There has been much speculation about Rokossovsky's personal views on this decision. He would always maintain that, with his communications badly stretched and enemy pressure against his northern flank mounting, committing forces to Warsaw would have been disastrous.

In November 1944, Rokossovsky was transferred to the 2nd Belorussian Front, which advanced into East Prussia and then across northern Poland to the mouth of the Oder at Stettin (now Szczecin). At the end of April he linked up with British Field Marshal Bernard Montgomery's forces in northern Germany while the forces of Zhukov and Ivan Koniev captured Berlin. It has been speculated that he was not allowed to capture Berlin because he was Polish.

With the end of the war Rokossovsky remained in command of Soviet forces in Poland (Northern Group of Forces). In October 1949, with the establishment of a fully Communist government under Bolesław Bierut in Poland, Rokossovsky, on Stalin's orders, became the Polish Minister of National Defense, with the additional title of Marshal of Poland. Together with Rokossovsky, several thousands of Soviet officers were put in charge of almost all Polish military units, either as commanding officers or as their advisors.

In 1952 he became Deputy Chairman of the Council of Ministers of the People's Republic of Poland. Although Rokossovsky was nominally Polish, he had not lived in Poland for 35 years, and most Poles regarded him as a Russian and Soviet emissary in the country, especially as he spoke poor Polish and even ordered Polish soldiers to address him in Russian instead. As Rokossovsky himself bitterly put it: "In Russia, they say I'm a Pole, in Poland they call me Russian".

Rokossovsky took part in the suppression of the Polish independence movement and stalinization and sovietization of Poland in general and the Polish Army in particular. As the superior commander of the Polish Army, he introduced various ways of suppression of anti-Soviet activity. Among the most notorious were the labour battalions of the army, to which all able-bodied men found socially or politically insecure, or guilty of having their families abroad were drafted. It is estimated that roughly 200,000 men were forced to work in labour camps in hazardous conditions, often in quarries, coal and uranium mines, and 1,000 died in their first days of "labour", while tens of thousands would become crippled. Other groups targeted by the repressions were former soldiers of the pre-war Polish Army and wartime Home Army.

In 1956 during the Poznań 1956 protests against Soviet occupation of Poland, Rokossovsky approved the order to send military units against protesters. As a result of the action of over 10,000 soldiers and 360 tanks, at least 74 civilians were killed.

When Communist reformers under Władysław Gomułka tried to come to power in Poland in 1956, Rokossovsky went to Moscov and tried to convince Nikita Khrushchev to use force against the Polish state. After Gomułka managed to negotiate with the Soviets, Rokossovsky left Poland. He returned to the Soviet Union, which restored his Soviet ranks and honours; and in July 1957, following the removal from office of Defense Minister Zhukov, Nikita Khrushchev appointed him Deputy Minister of Defense and Commander of the Transcaucasian Military District. In 1958 he became chief inspector of the Ministry of Defense, a post he held until his retirement in April 1962.

He died in August 1968, aged 71, and lies buried in the Kremlin Wall Necropolis on Red Square.