His mayoral victory came in the wake of the Great Depression and the deep resentment many Chicagoans had of Prohibition and the increasing violence resulting from organized crime's control of Chicago, typified by the St. Valentine's Day Massacre. The many ethnic groups such as Poles, Czechs, Ukrainians, Jews, Italians, and African Americans that began to settle in Chicago in the early 1900s were mostly detached from the political system, due in part to lack of organization which led to underrepresentation in the City Council. As an immigrant himself, Cermak recognized Chicago's relatively new immigrants as a significant population of disenfranchised voters and a large power base for Cermak and his local Democratic organization.

Before Cermak, the Democratic party in Cook County was run by the "Lace Curtain" Irish. They looked down on anyone who wasn't "Lace Curtain," even the Irish from the Back of the Yards and Bridgeport neighborhoods (referred to as "Pig Shit" Irish), and also non-Irish ethnics. As Cermak climbed the local political ladder, the resentment of the Lace Curtain group grew. When the bosses rejected his bid to become the mayoral candidate, Cermak swore revenge. He formed his political army from the non-Irish elements, and even persuaded black politician William L. Dawson to switch from the Republican to the Democratic Party. Dawson later became U.S. Representative (from the 1st District) and soon the most powerful black politician in Illinois.

Cermak's political and organizational skills helped create one of the most powerful political organizations of his day, and Cermak is considered the father of Chicago's Democratic machine. With support from Franklin D. Roosevelt on the national level, Cermak gradually wooed members of Chicago's growing black community into the Democratic fold. Walter Wright, the superintendent of parks and aviation for the city of Chicago also aided Cermak in stepping into office.

When Cermak challenged the incumbent "Big Bill" Thompson in the 1931 mayor's race, Thompson, representative of Chicago's existing power structure, responded with ethnic slurs:

I won't take a back seat to that Bohunk, Chairmock, Chermack or whatever his name is.

Tony, Tony, where's your pushcart at?

Can you picture a World's Fair mayor?

With a name like that?

Cermak's reply, "He doesn't like my name... it's true I didn't come over on the Mayflower, but I came over as soon as I could." It was a sentiment to which ethnic Chicagoans could relate and Thompson's slur largely backfired.

The flamboyant Thompson's reputation as a buffoon and the voters' disgust with the corruption of his machine and his inability or unwillingness to clean up organized crime in Chicago were cited as major factors in Cermak capturing 58% of the vote in the mayoral election on April 6, 1931. Cermak's victory finished Thompson as a political power and largely ended the Republican Party's power in Chicago – no Republican has held the office of mayor of Chicago since Thompson's exit in 1931.

For nearly his entire administration, Cermak had to deal with a major tax revolt. From 1931 to 1933, the Association of Real Estate Taxpayers mounted a "tax strike." At its height, ARET, which was headed by John M. Pratt and James E. Bistor, had over thirty thousand members. Much to Cermak's dismay, it successfully slowed down the collection of real estate taxes through litigation and promoting refusal to pay. In the meantime, the city found it difficult to pay teachers and maintain services.

While shaking hands with President-elect Franklin D. Roosevelt at Bayfront Park in Miami, Florida, on February 15, 1933, Cermak was shot in the lung and seriously wounded when Giuseppe Zangara, who was believed to have been engaged in an attempt to assassinate Roosevelt, hit Cermak instead. Later, rumors circulated that Cermak, not Roosevelt, had been the intended target, as his promise to clean up Chicago's rampant lawlessness posed a threat to Al Capone and the Chicago organized crime syndicate. According to Roosevelt biographer Jean Edward Smith, there is no proof for this theory. One of the first people to suggest the organized crime theory was reporter Walter Winchell, who happened to be in Miami the evening of the shooting. At the critical moment, Lilian Cross, a doctor's wife, hit Zangara's arm with her purse and spoiled his aim. In addition to Cermak, Zangara hit four other people, one of whom, a woman, also died of her injuries. Zangara told the police that he hated rich and powerful people, but not Roosevelt personally.

Long-time Chicago newsman Len O'Connor offers a different view of the events surrounding Cermak's death. He has written that aldermen "Paddy" Bauler and Charlie Weber informed him that relations between Cermak and FDR were strained because Cermak fought FDR's nomination at the Democratic convention in Chicago, and the legend that his last words were "I'm glad it was me instead of you" was, according to O'Connor, totally fabricated by Weber and Bauler.

Author Ronald Humble offers his view as to why Cermak was killed. In his book Frank Nitti: The True Story of Chicago's Notorious Enforcer, Humble contends that Cermak was as corrupt as Thompson and that the Chicago Outfit hired Zangara to kill Cermak in retaliation for Cermak's attempt to murder Frank Nitti.

Cermak died of his wounds on March 6 and was interred in a mausoleum at Bohemian National Cemetery in Chicago. The mayor's death was followed by a struggle for succession to his party chairmanship and to the mayor's office.

A plaque honoring Cermak still lies at the site of the assassination in Miami's Bayfront Park. It is inscribed with Cermak's alleged words to FDR after he was shot, "I'm glad it was me instead of you." Following Cermak's death, 22nd Street, a major east-west artery that traversed Chicago's West Side and the close-in suburbs of Cicero and Berwyn, areas with a significant Czech population, was renamed Cermak Road. In addition, Zangara was electrocuted in Florida's electric chair on March 20, 1933, for he could not be charged with murder until Cermak died. In 1943, a Liberty ship, the SS A. J. Cermak was named after Cermak. It was scrapped in 1964.

Cermak's son-in-law, Otto Kerner, Jr., was governor of Illinois and a federal circuit judge. Relatives of Cermak still live in the Chicago area.