Fleming was born in Kirkcaldy, Fife, Scotland, to Andrew and Elizabeth Fleming. In 1845, at the age of 17, he emigrated with his older brother David to Ontario (then the western half of the British province of United Canada, at that time called Canada West). Their route took them through many cities of the Canadian colonies, Quebec City, Montreal, and Kingston, Ontario, before settling in Peterborough, Ontario with their cousins two years later in 1847.

In 1849 he established the Royal Canadian Institute, which was formally incorporated on November 4, 1851. In 1851 he designed the Threepenny Beaver, the first Canadian postage stamp. Throughout this time he was fully employed as a surveyor, mostly for the Grand Trunk Railway. His work for them eventually gained him the position as Chief Engineer of the Northern Railway of Canada in 1855, where he tirelessly advocated the construction of iron bridges instead of wood for safety reasons.

Fleming served in the 10th Battalion Volunteer Rifles of Canada (later known as the Royal Regiment of Canada) and was appointed to the rank of Captain on January 1, 1862. He retired from the militia in 1865.

In 1858 he first proposed a coast to coast railway line spanning all of British North America. The timing was not quite right, but a few years later he was appointed as the sole engineer to supervise the survey of the proposed Intercolonial Railway, linking the Maritime provinces with Quebec. He moved for a time to Halifax, Nova Scotia, during construction, where he built a house at the seaward end of town. In 1872, the newly formed Canadian government decided to build a rail link to the Pacific Ocean, and naturally the job of surveying the route fell to Fleming. That same year he organized an expedition to the Pacific that included surveyors as well as the naturalist John Macoun, and his Church of Scotland clergyman from the St. Matthew's Presbyterian "kirk" from Halifax, George Monro Grant. Over the next few years he supervised both the Intercolonial and the Canadian Pacific Railway, a job he completed in 1876 before turning over the chief engineer position to his long term collaborator, Collingwood Schreiber. Fleming was present when Donald Smith drove in the "Last Spike" in Craigellachie, British Columbia, in 1885, now as a board member of the Canadian Pacific company. He published The Intercolonial: A Historical Sketch (1876).

As soon as he arrived in Peterborough in 1845, Fleming became friendly with the family of his future wife, the Halls, and was attracted to Jeanie Hall. However, it was not until a sleigh accident almost ten years later that the young people’s love for each other was revealed. A year after this incident, in January 1855, Sandford married Ann Jane (Jean) Hall. They were to have nine children of whom two died young. The oldest son, Frank Andrew, accompanied Fleming in his great Western expedition of 1872. A family man, deeply attached to his wife and children, he also welcomed his father Andrew Greig Fleming, Andrew's wife and six of their other children who came to join him in Canada two years after his arrival. The Fleming and Hall families saw each other often.

After missing a train in 1876 in Ireland because its printed schedule listed p.m. instead of a.m., he proposed a single 24-hour clock for the entire world, located at the centre of the Earth and not linked to any surface meridian. At a meeting of the Royal Canadian Institute on February 8, 1879 he linked it to the anti-meridian of Greenwich (now 180°). He suggested that standard time zones could be used locally, but they were subordinate to his single world time, which he called Cosmic Time. He continued to promote his system at major international conferences, including the International Meridian Conference of 1884. That conference accepted a different version of Universal Time, but refused to accept his zones, stating that they were a local issue outside its purview. Nevertheless, by 1929 all of the major countries of the world had accepted time zones.

In 1880 he retired from the world of surveying, and took the position of Chancellor of Queen's University in Kingston, Ontario. He held this position for his last 35 years, where his former Minister George Monro Grant was principal from 1877 until Grant's death in 1902. Not content to leave well enough alone, he tirelessly advocated the construction of a submarine telegraph cable connecting all of the British Empire, the All Red Line, which was completed in 1902. In 1880 he served as the vice president of the Ottawa Horticultural Society. In his later years he retired to his house in Halifax, later deeding the house and the 95 acres (38 hectares) to the city, now known as Sir Sandford Fleming Park (Dingle Park). He also kept a residence in Ottawa, and was buried there, in the Beechwood Cemetery. He was a freemason.

His accomplishments were well known world wide, and in 1897 he was knighted by Queen Victoria. Fleming Hall was built in his honour at Queen's in 1901, and rebuilt after a fire in 1932. It was the home of the university's Electrical Engineering department.

In Peterborough, Ontario, Fleming College, a Community College of Applied Arts and Technology bearing his name, was opened in 1967, with additional campuses in Lindsay/Kawartha Lakes, Haliburton, and Cobourg. Also, the main building of University of Toronto Faculty of Applied Science and Engineering is named after Fleming (Sandford Fleming building).