He was born in Logierait, Perthshire, Scotland, to Alexander Mackenzie Sr. and Mary Stewart Fleming. He was the third of ten children. At the age of 13, Mackenzie's father died, and he was forced to end his formal education in order to help support his family. At the age of 16 he apprenticed as a stone mason and by the age of 20 he had reached journeyman status in this field. Mackenzie immigrated to Canada in 1842 in order to seek a better life as well as to follow his sweetheart, Helen Neil. Shortly thereafter, he converted from Presbyterianism to Baptist beliefs. Mackenzie's faith was to link him to the increasingly influential temperance cause, particularly strong in Ontario where he lived, a constituency of which he was to represent in the Parliament of Canada.

Mackenzie married Helen Neil (1826 – 1852) in 1845 and with her had three children, with only one girl surviving infancy. In 1853, he married Jane Sym (1825 – 1893).

In Canada, Mackenzie continued his career as a stone mason, building many structures that still stand today. He began working as a general contractor, earning a reputation for being a hard working, honest man as well as having a working man's view on fiscal policy. Mackenzie involved himself in politics almost from the moment he arrived in Canada. He campaigned relentlessly for George Brown, owner of the Reformist paper The Globe in the 1851 election, helping him to win a seat in the assembly. In 1852 Mackenzie became editor of another reformist paper, the Lambton Shield (now the Sarnia Observer). As editor, Mackenzie was perhaps a little too vocal, leading the paper to a suit of law for libel against the local conservative candidate. The paper lost the suit and was forced to fold due to financial hardship. He was elected to the Legislative Assembly as a supporter of George Brown in 1861.

When the Macdonald government fell due to the Pacific scandal in 1873, the Governor General, Lord Dufferin, called upon Mackenzie, who had been chosen as the leader of the Liberal Party a few months earlier, to form a new government. Mackenzie formed a government and then asked the Governor General to call an election for January 1874. The Liberals won, and Mackenzie remained prime minister until the 1878 election when Macdonald's Conservatives returned to power with a majority government. It was unusual for a man of Mackenzie's humble origins to attain such a position in an age which generally offered such opportunity only to the privileged. Lord Dufferin, the current Governor General, expressed early misgivings about a stonemason taking over government. But on meeting Mackenzie, Dufferin revised his opinions:

"However narrow and inexperienced Mackenzie may be, I imagine he is a thoroughly upright, well-principled, and well-meaning man."

Mackenzie also served as Minister of Public Works and oversaw the completion of the Parliament Buildings. While drawing up the plans, he included a circular staircase leading directly from his office to the outside of the building which allowed him to escape the patronage seekers waiting for him in his ante-chamber. Proving Dufferin's reflections on his character to be true, Mackenzie disliked intensely the patronage inherent in politics. Nevertheless, he found it a necessary evil in order to maintain party unity and ensure the loyalty of his fellow Liberals.

In keeping with his democratic ideals, Mackenzie refused the offer of a knighthood three times, and was thus the only one of Canada's first eight Prime Ministers not to be knighted. His pride in his working-class origins never left him. Once, while touring Fort Henry as prime minister, he asked the soldier accompanying him if he knew the thickness of the wall beside them. The embarrassed escort confessed that he didn't and Mackenzie replied, "I do. It is five feet, ten inches. I know, because I built it myself!".

As Prime Minister, Alexander Mackenzie strove to reform and simplify the machinery of government. He introduced the secret ballot; advised the creation of the Supreme Court of Canada; the establishment of the Royal Military College of Canada in Kingston in 1874; the creation of the Office of the Auditor General in 1878; and struggled to continue progress on the national railway. After his government's defeat, Mackenzie remained Leader of the Opposition until 1880, when he relinquished the party leadership to Edward Blake. However, he remained as a Member of Parliament until his death in 1892 from a stroke that resulted from hitting his head during a fall. He died in Toronto and was buried in Lakeview Cemetery in Sarnia, Ontario.

In their 1999 study of the Prime Ministers of Canada, which included the results of a survey of Canadian historians, J.L. Granatstein and Norman Hillmer found that Mackenzie was in the #11 place just after John Sparrow David Thompson.