Gromyko was born to a Belarusian peasant family in the Belarusian village of Staryja Hramyki/ Gramyki, near Gomel. He studied agriculture at the Minsk School of Agricultural Technology and graduated in 1932. Later he worked as an economist at the Institute of Economics in Moscow 1936 – 1939.

Gromyko entered the department of the foreign affairs in 1939 after Joseph Stalin's purges in the section responsible for the Americas. He was soon sent to the United States and worked in the Soviet embassy there until 1943, when he was appointed the Soviet ambassador to the United States. He played an important role in coordinating the wartime alliance between the two nations and was prominent at events such as the Yalta Conference. He became known as an expert negotiator. In the West, Gromyko received a nickname "Mr. Nyet" (Mr. No) or "Comrade Nyet" or "Grim Grom" for his obstinate negotiating style. He was removed from his Washington post on April 10, 1946 in order to be able to devote his full attention to United Nations matters.

In 1946 he became the Soviet Union's representative on the United Nations Security Council. He served briefly as the ambassador to the United Kingdom in 1952 – 1953 and then returned to the Soviet Union, where he served as Foreign Minister of the USSR for 28 years. As Soviet foreign minister, Gromyko played a direct role in the Cuban Missile Crisis and met with U.S. President Kennedy during the crisis.

Gromyko also helped negotiate arms limitations treaties, specifically the ABM Treaty, the Nuclear Test Ban Treaty, SALT I and II, and the INF and START agreements. During the Brezhnev years, he helped construct the policy of détente between the superpowers and was active in drawing up the non-aggression pact with West Germany.

In 1966, he engaged in a dialog with Pope Paul VI as part of the pontiff's ostpolitik, which resulted in greater openness for the Roman Catholic Church in Eastern Europe.

Gromyko always believed in the superpower status of the Soviet Union and always promoted an idea that no important international agreement could be reached without its involvement.

Gromyko was minister of foreign affairs from 1957 until 1985, when he was replaced as foreign minister by Eduard Shevardnadze. Gromyko nominated Gorbachev for the Communist Party general secretary's post at the 11 March 1985 Politburo meeting. Gromyko entered the Politburo in 1973, eventually becoming chairman of the Presidium of the Supreme Soviet (i.e. head of state of the Soviet Union) in 1985. However, the position was largely ceremonial, and he was forced out three years later because of his conservative views during the Gorbachev era. Gromyko died in Moscow one year later.

Gromyko's dour demeanour, shown clearly during his first term in Washington, echoed throughout his tenure as Soviet foreign minister. There is a story that Gromyko was leaving a Washington hotel one morning and was asked by a reporter, "Minister Gromyko, did you enjoy your breakfast today?" His response was "Perhaps."

He had a wife named Lidiya (died 2004), a son named Anatoli (born 1932), and a daughter named Emiliya (born 1938).