In 1954, Andropov became the Soviet Ambassador in Hungary and held this position during the 1956 Hungarian Revolution. After these events, Andropov suffered from a "Hungarian complex", according to historian Christopher Andrew: "he had watched in horror from the windows of his embassy as officers of the hated Hungarian security service were strung up from lampposts. Andropov remained haunted for the rest of his life by the speed with which an apparently all-powerful Communist one-party state had begun to topple. When other Communist regimes later seemed at risk - in Prague in 1968, in Kabul in 1979, in Warsaw in 1981, he was convinced that, as in Budapest in 1956, only armed force could ensure their survival". Andropov played a key role in crushing the Hungarian Revolution. He convinced a reluctant Nikita Khrushchev that military intervention was necessary. He deceived Imre Nagy and other Hungarian leaders that the Soviet government did not order an attack on Hungary at the very moment of this attack. The Hungarian leaders were arrested and Nagy executed.

Andropov returned to Moscow to head the Department for Liaison with Communist and Workers' Parties in Socialist Countries (1957 – 1967). In 1961, he was elected full member of the CPSU Central Committee and was promoted to the Secretariat of the CPSU Central Committee in 1962. In 1967, he was relieved of his work in the Central Committee apparatus and appointed head of the KGB on recommendation of Mikhail Suslov.

During the Prague Spring events in Czechoslovakia, Andropov was the main proponent of the "extreme measures". He ordered the fabrication of false intelligence not only for public consumption, but also for the Soviet Politburo. "The KGB whipped up the fear that Czechoslovakia could fall victim to NATO aggression or to a coup". At this moment, Soviet intelligence officer Oleg Kalugin reported from Washington that he gained access to "absolutely reliable documents proving that neither the CIA nor any other agency was manipulating the Czechoslovak reform movement". However his message was destroyed because it contradicted the conspiracy theory fabricated by Andropov. Andropov ordered a number of active measures, collectively known as operation PROGRESS, against Czechoslovak reformers.

Andropov was personally obsessed with "the destruction of dissent in all its forms" and always insisted that "the struggle for human rights was a part of a wide-ranging imperialist plot to undermine the foundation of the Soviet state". In 1968 he issued a KGB Chairman's order "On the tasks of State security agencies in combating the ideological sabotage by the adversary", calling for struggle against dissidents and their imperialist masters. The brutal repression of dissidents included plans to maim the dancer Rudolf Nureyev, who had defected in 1961.

In 1973, Andropov was promoted to full member of the Politburo. Andropov played the dominant role in the decision to invade Afghanistan in 1979. He insisted on the invasion, although he expected that the international community would blame the USSR for this action; the decision led to the Soviet war in Afghanistan (1979 – 1988).

Andropov was the longest-serving KGB chairman and did not resign as head of the KGB until May 1982, when he was again promoted to the Secretariat to succeed the late Mikhail Suslov as secretary responsible for ideological affairs. Two days after Brezhnev's death, on 12 November 1982, Andropov was elected General Secretary of the CPSU being the first former head of the KGB to become General Secretary. His appointment was received in the West with apprehension, in view of his roles in the KGB and in Hungary. At the time his personal background was a mystery in the West, with major newspapers printing detailed profiles of him that were inconsistent and in many cases fabricated.

During his rule, Andropov attempted to improve the economy by raising management effectiveness without changing the principles of socialist economy. In contrast to Brezhnev's policy of avoiding conflicts and dismissals, he began to fight violations of party, state and labour discipline, which led to significant personnel changes. During 15 months in office, Andropov dismissed 18 ministers, 37 first secretaries of obkoms, kraikoms and Central Committees of Communist Parties of Soviet Republics; criminal cases on highest party and state officials were started. For the first time, the facts about economic stagnation and obstacles to scientific progress were made available to the public and criticised.

In foreign policy, the war continued in Afghanistan, although Andropov - who felt the invasion may have been a mistake - did half-heartedly explore options for a negotiated withdrawal. Andropov's rule was also marked by deterioration of relations with the  United States. U.S. plans to deploy Pershing missiles in Western Europe in response to the Soviet SS-20 missiles were contentious. But when Paul Nitze, the American negotiator, suggested a compromise plan for nuclear missiles in Europe in the celebrated “walk in the woods” with Soviet negotiator Yuli Kvitsinsky, the Soviets never responded. Kvitsinsky would later write that, despite his own efforts, the Soviet side was not interested in compromise, instead calculating that peace movements in the West would force the Americans to capitulate. On March 8, 1983, during Andropov's reign as General Secretary, U.S. President Ronald Reagan famously labeled the Soviet Union an "evil empire."

In August 1983 Andropov made a sensational announcement that the country was stopping all work on space-based weapons. One of his most notable acts during his short time as leader of the Soviet Union was in response to a letter from an American child from Maine named Samantha Smith, inviting her to the Soviet Union. Some people think he used Smith to make the Soviet Union less corrupt. Smith made friends with children in Moscow. This resulted in Smith becoming a well-known peace activist. Meanwhile, Soviet-U.S. arms control talks on intermediate range nuclear weapons in Europe were suspended by the Soviet Union in November 1983 and by the end of 1983, the Soviets had broken off all arms control negotiations.

Cold War tensions were exacerbated by the downing near Moneron Island by Soviet fighters of a civilian jet liner, Korean Air Flight KAL-007 with a complement of 269 passengers and crew, including a congressman from Georgia, Larry McDonald. KAL 007 had strayed over the Soviet Union on 1 September 1983 on its way from Anchorage, Alaska, to Seoul, South Korea. Andropov was advised by his Defense Minister Dmitri Ustinov and by the head of the KGB Victor Chebrikov to keep secret the fact that the Soviet Union held in its possession the sought-after "black box" from KAL 007. Andropov was encouraged to state that the Soviet Union engage in the deception that they too were looking for KAL 007 and the black box. Andropov agreed to this and the ruse continued until Boris Yeltsin disclosed the secret in 1992.

When he could no longer work in the Kremlin or attend the Politburo meetings, from September 1983, he adopted an original way of governing: he would suggest ideas to his assistants and speech writers, who would then prepare analytical "notes" for the Politburo.

On a Saturday preceding a Tuesday plenum of the Central Committee, Arkady Volsky, an aide to Andropov, came to Andropov's room at the Central Clinical Hospital in Kuntsevo to help him draft a speech. Andropov was in no shape to attend the plenum and he would have one of his men in the Politburo deliver the speech in his name. The last lines in the speech said that Central Committee staff members should be exemplary in their behavior, uncorrupted, responsible for the life of the country. Then Andropov gave Volsky a folder with the final draft and said, "The material looks good. Make sure you pay attention to the agenda I've written." Since the doctor walked him to the car, he did not have time to look right away at what he had written. Later, he got a chance to read it and saw that at the bottom of the last page Andropov had added in ink, in a somewhat unsteady handwriting, a new paragraph. It went like this: "Members of the Central Committee know that due to certain reasons, I am unable to come to the plenum. I can neither attend the meetings of the Politburo nor the secretariat. Therefore, I believe Mikhail Sergeyevich Gorbachev should be assigned to preside over the meetings of the Politburo and the secretariat (of the Central Committee)." Andropov was recommending that Gorbachev be his inheritor. Volsky made a photocopy of the document and put the copy in his safe. He delivered the original to the Party leadership and assumed that it would be read out at the plenum. But at the meeting neither Konstantin Chernenko, Viktor Grishin, Nikolai Tikhonov, Dmitriy Ustinov nor any of the other politburo members made mention of Andropov's stated wishes. Volsky thought there must have been some mistake: "I went up to Chernenko and said, 'There was an addendum in the text.' He said, 'Think nothing of any addendum.' Then I saw his aide Bogolyubov and said, 'Klavdy Mikhailovich, there was a paragraph from Andropov's speech....' He led me off to the side, and said, 'Who do you think you are, a wise guy? Do you think your life ends with this?' I said, 'In that case, I'll have to phone Andropov.' And he replied, 'Then that will be your last phone call'". Andropov was furious when he heard what had happened at the plenum, but there was little he could do.

In his memoirs, Mikhail Gorbachev recalled that when Andropov was the leader, he and Nikolai Ryzhkov, the chairman of Gosplan, asked Andropov for access to real budget figures. "You are asking too much," he responded. "The budget is off limits to you."

In February 1983, Andropov suffered total renal failure. In August 1983, he entered the Central Clinical Hospital in western Moscow on a permanent basis, where he would spend the remainder of his life. His aides would take turns visiting him in the hospital with important matters and paperwork. Shortly before he was due to leave for the Crimea, Andropov's health severely deteriorated. The lightly dressed Andropov had become tired, and had taken a rest on a granite bench in the shade; his body became thoroughly chilled, and he soon began shivering uncontrollably. The only ones who saw him on a regular basis were Politburo members Dmitry Ustinov, Andrey Gromyko, Konstantin Chernenko and Viktor Chebrikov. For the last two months of his life Andropov did not get out of bed, except when he was lifted onto a couch while his sheets were changed. He was physically finished but his mind was clear. Throughout his last days Andropov still worked, even if it meant little more than signing papers or giving his assent to his aides' proposals.

On 31 December 1983 Andropov celebrated the New Year for the last time. Vladimir Kryuchkov together with other friends visited Andropov. He was very thankful that his doctors let him drink a glass of champagne. They remained with him for about an hour and a half. After they had gone, Andropov remained alone with Kryuchkov and told him that he wished health and success to all the friends. At that moment, Kryuchkov understood that Andropov was going to die. In January, the future prime minister Nikolai Ryzhkov visited Andropov. Andropov kissed him and told him to go.

In late January 1984 the gradual decline in his health that characterized his tenure suddenly intensified due to growing intoxication in his blood, as a result of which he had periods of failing consciousness. On 9 February 1984, Andropov's last day, the nurse came to Boris Klukov, one of his many bodyguards, and said that he didn't want to eat. She asked him to try to convince Andropov to eat. Klukov came up to Andropov and convinced him that he must eat. Andropov finally agreed to eat and they ate together. Then, Boris Klukov left the room for some time. And after half an hour there was a sudden commotion. Doctors ran to Andropov's room and the assistant of the security director also went there. Klukov called the assistants. He came up to Andropov's room, looked at the display and observed his slowing pulse. Andropov died on that day at 16:50 in his hospital room. Few of the top people, not even all the Politburo members, learned of the fact on the same day. According to the Soviet medical report, Andropov suffered from several medical conditions: interstitial nephritis, nephrosclerosis, residual hypertension and diabetes, which were worsened by chronic kidney deficiency.

A four-day period of nationwide mourning was announced. Inside the hall, mourners shuffled up a marble staircase beneath chandeliers draped in black gauze. On the stage at the left side of the hall, amid a veritable garden of flowers, a complete symphony orchestra in black tailcoats played classical music. Andropov's embalmed body, dressed in a black suit, white shirt and black-and-red tie, lay in an open coffin banked with carnations, red roses and tulips, faced the long queue of mourners. At the right side of the hall, in the front row of seats reserved for the dead leader's family, his wife Tatyana Filipovna with her red hair held in place with a hairclip, sat alongside with their two children, Igor and Irina.

On 14 February, the funeral parade began. Two officers led the funeral parade, carrying a large portrait of him followed by numerous red floral wreaths. Then officers in tall Astrakhan hats appeared, carrying the late leader's 21 decorations and medals on small red cushions. Behind them, the coffin rested atop a gun carriage drawn by an olive-green military scout vehicle. Walking immediately behind were the members of Andropov's family. The Politburo leaders, almost indistinguishable from one another in their fur hats and look-alike overcoats with red armbands, led the last group of official mourners. As the coffin reached the middle of the Red Square, it was taken out of the carriage and placed on a red-draped bier facing the Lenin Mausoleum, with its lid removed. At exactly 12:45 p.m. Tuesday, Andropov's coffin was lowered into the ground as foghorns blared, joining with sirens, wheezing factory whistles and rolling gunfire in a mournful cacophony.

He was succeeded in office by Konstantin Chernenko.

Andropov's legacy remains the subject of much debate in Russia and elsewhere, both among scholars and in the popular media. He remains the focus of television documentaries and popular non-fiction, particularly around important anniversaries. As KGB head, Andropov was ruthless against dissent, and author David Remnick, who covered the Soviet Union for the Washington Post in the 1980s, called Andropov "profoundly corrupt, a beast". Alexander Yakovlev, later an advisor to Mikhail Gorbachev and the ideologist of perestroika, said "In a way I always thought Andropov was the most dangerous of all of them, simply because he was smarter than the rest." However, it was Andropov himself who recalled Yakovlev back to high office in Moscow in 1983 after a ten year de facto exile as ambassador to Canada after attacking Russian chauvinism. Yakovlev was also a close colleague of Andropov associate KGB General Yevgeny Primakov, later Prime Minister of Russia.

According to his former subordinate Securitate general Ion Mihai Pacepa,

"In the West, if Andropov is remembered at all, it is for his brutal suppression of political dissidence at home and for his role in planning the 1968 invasion of Czechoslovakia. By contrast, the leaders of the former Warsaw Pact intelligence community, when I was one of them, looked up to Andropov as the man who substituted the KGB for the Communist party in governing the Soviet Union, and who was the godfather of Russia's new era of deception operations aimed at improving the badly damaged image of Soviet rulers in the West."

Despite Andropov's hard-line stance in Hungary and the numerous banishments and intrigues for which he was responsible during his long tenure as head of the KGB, he has become widely regarded by many commentators as a humane reformer, especially in comparison with the stagnation and corruption during the later years of his predecessor, Leonid Brezhnev. Andropov, "a throwback to a tradition of Leninist asceticism", was appalled by the corruption during Brezhnev's regime, and ordered investigations and arrests of the most flagrant abusers. The investigations were so frightening that several members of Brezhnev's circle "shot, gassed or otherwise did away with themselves." He was certainly generally regarded as inclined to more gradual and constructive reform than was Gorbachev; most of the speculation centres around whether Andropov would have reformed the USSR in a manner which did not result in its eventual dissolution.

The Western media favored Andropov because of his supposed passion for Western music and scotch. However, these were unproven rumours. It is also questionable whether Andropov spoke any English at all. 

The short time he spent as leader, much of it in a state of extreme ill health, leaves debaters few concrete indications as to the nature of any hypothetical extended rule. As with the shortened rule of Lenin, speculators have much room to advocate their favourite theories and to develop the minor cult of personality which has formed around him.

Andropov lived in 26 Kutuzovski prospekt, the same building that Suslov and Brezhnev also lived in. He was first married to Nina Ivanovna. She bore him a son who died in mysterious circumstances in the late 1970s. In 1983 she was diagnosed with cancer and underwent a successful operation. He met his second wife, Tatyana Filipovna, during WW2 in the Karelian Front when she was Komsomol secretary. She had suffered a nervous breakdown during the Hungarian revolution. Andropov's chief guard informed Tatyana about the death of her husband. She was too grief-stricken to join in the procession and during the funeral her relatives helped her to walk. Before the lid could be closed on Andropov's coffin, she bent to kiss him. She touched his hair and then kissed him again. In 1985, a respectful 75-minute film was broadcast in which Tatyana (not even seen in public until Andropov's funeral) reads love poems written by her husband. Tatyana became ill and died in November 1991. Andropov had also a son, Igor (died June 2006) and a daughter, Irina (born 1946).