Xu was born in Wutai county, Shanxi province, China. He was admitted to the Whampoa Academy in 1924 and held various officer ranks in the National Revolutionary Army between 1925 and 1927. Xu joined the Communist Party of China in 1927 and later became a commander in the 4th Front Army of the Red Army, led by Zhang Guotao. He served as Zhang's principle commander with Ye Jianying as Chief of Staff. During this time, he helped Zhang to establish new communist bases and expanded the 4th Front Army of the Chinese Red Army despite the fact that his wife was executed by Zhang Guotao in his political purges. While under suspicion and the surveillance of Zhang's political commissars, Xu Xianqian led the 80,000 strong 4th Front Army of the Chinese Red Army in Sichuan to glorious victories against the nationalist warlords' troops that numbered more than 300,000, killing more than 100,000 of them, and defeated and scattered the rest 200,000. Xu remained loyal to Zhang Guotao despite Zhang's distrust and unlike Ye Jianying who defected to Mao Zedong after Mao and Zhang disagreed, Xu faithfully carried out Zhang's unrealistic orders which obviously ended up in the inevitable disaster, which eventually contributed to Zhang Guotao's failure and downfall from power.

During the Sino - Japanese War (1937 - 1945), Xu fought against the Japanese invaders, and established communist bases in the northern China. The communist bases proved to be a communist stronghold and when the communist headquarter in its base in Shaanxi was forced to be evacuated under the Kuomintang's military pressure, it was evacuated to the base established by Xu. After World War II, Xu participated in the struggles against the nationalists and he had demonstrated his brilliant military skills, often contradictory to Mao's military doctrine. For example, when the enemy is much stronger, Mao's military doctrine emphasized achieving local victories by concentrating the force to form absolute numerical superiority over the enemy in a particular local battle, often at least 3 or 4 times of enemy strength (better 5 or 6 times), and accumulate small victories into large ones. This way, the technical disadvantage and the numerical disadvantage in overall strength can be effectively resolved.

Xu, in contrast, during his fight against his fellow Shanxi warlord Yan Xishan who was on Chiang Kai-shek's side, defied Mao's military doctrines by his daring attack on the numerically and technically superior Kuomintang force in battles and achieving astonishing success: Xu's force only numbered a mere 60,000 at the beginning of the campaigns and within a mere 18 months, his force completely defeated Yan Xishan's 350,000 numerically and technically superior force, killed 300,000 of them, with only 50,000 out of the original 350,000 being able to escape back to the fortress city of Taiyuan. During the final assault on Taiyuan, Xu's numerically and technically inferior force of 100,000 once again defeated Yan Xishan's numerically and technically superior force of 130,000, taking the city. Mao and other communist leaders were so impressed by the result that long after the battles were over, they often joked on how Xu achieved that whenever they met Xu.

After the communist revolution in China in 1949, Xu Xianqian served as the General Chief of Staff of the People's Liberation Army, as vice chairman of the CMC in 1954, and was awarded the rank of marshal in 1955. He served as vice premier in 1965. Xu protected Deng Xiaoping when Deng was purged from the government in 1976. He was one of the military supporters of Hua Guofeng's coup against the Gang of Four. He later served as Defense Minister from 1978 to 1981. In 1978, Xu was almost killed in an accident of Chinese HJ-73 ATGM demonstration when the missile suddenly malfunctioned and turned 180 degrees after traveling several hundred meters, flying in opposite direction toward the observatory platform where Xu and other top ranking Chinese officers were sitting, and landed right in front of the platform. It was fortunate that the missile failed to explode and Xu and others on the platform survived and remained there until the completion of the demonstration. Xu did not originally planned to attend the demonstration, but because both Ye Jianying and Nie Rongzhen, who originally planned to attend, were hospitalized at the time, Xu was invited instead.

Xu led the preparations for PLA operations in the Sino - Vietnam War in 1979.