Polhem was born on the island of Gotland. His birth is not found in any official records such as church records, so the exact place of birth is unknown, but assumed to be Visby. Originally the Polheim family came from Austria to Pomerania, Germany, from whence his father, Wolf Christoph Polhammer traded with Visby, where he would eventually settle down to become a skipper. When Polhem was 8, his father died and his mother, Christina Eriksdotter Schening from Vadstena, Östergötland remarried. As a result of conflicts with his stepfather, his private tuition was no longer paid for and Polhem was sent to live with his uncle in Stockholm. In Stockholm he attended a German school until the age of 12 when his uncle died; once again Polhem was left without the possibility of education.

He took a job as a farmhand on Vansta, a property in Södertörn, Stockholm. He quickly rose to the position of supervisor, being responsible for supervision and accounting, for which he was well suited by his affinity for mathematics. He worked at Vansta for ten years, during which period he constructed a workshop where he made tools, repaired and constructed simple machinery to earn money.

Hungering for knowledge within his fields of interest, mathematics and mechanics, he soon realized that he would get no further without learning Latin. Self-studies were attempted, but given up; Polhem realized he needed a tutor. In exchange for constructing a complex clock, he was given Latin lessons by a local vicar.

Word of Polhem's mechanical skill spread quickly and a member of the clergy wrote the professor of mathematics at Uppsala University, Anders Spole to recommend Polhem. Spole presented two broken clocks to Polhem and offered to let him study under him if he could repair them, Polhem repaired the clocks with no difficulty and began recovering years of lost education in 1687, at the age of 26.

He married Maria Hoffman in 1691, together they had two children, Gabriel and Emerentia.

In 1716 he was ennobled in gratitude of his services to the nation by the king and changed his name from Polhammar to Polheim, which he later changed to Polhem.

He and his son Gabriel Polhem were both elected members of the Royal Swedish Academy of Sciences in 1739, the year the Academy was founded.

Polhem died of natural causes in 1751 in Stockholm.

According to Polhem's autobiography, the event that marked the beginning of his career was the successful repair of the unfinished medieval (16th century) astronomical clock by Petrus Astronomus at Uppsala Cathedral, which had remained unfinished and broken for more than a century.

In 1690 Polhem was appointed to improve upon the current mining operations of Sweden. His contribution was a construction for lifting and transporting ore from mines, a process that was rather risky and inefficient at the time. The construction consisted of a track system for lifting the ore, as opposed to wires; the construction was powered entirely by a water wheel. Human labor needed was limited to loading the containers. Being new and revolutionary, word of Polhem's work reached the reigning king, Charles XI, who was so impressed with the work that he assigned him to improve Sweden's main mining operation; the Falun Copper mine.

Funded by the Swedish mining authority, Polhem traveled throughout Europe, studying mechanical development, he returned to Sweden in 1697 to establish laboratorium mechanicum in Stockholm, a facility for training of engineers, as well as a laboratory for testing and exhibiting his designs, it is considered to be the predecessor of The Royal Institute of Technology. The laboratory was later moved from Stockholm to Falun and from there to Stjärnsund. In 1748 the collection of models used in the laboratory was returned to Stockholm.

His greatest achievement was an automated factory powered entirely by water; automation was very unusual at the time. Built in 1699 in Stjärnsund, the factory produced a number of products, deriving from the idea that Sweden should export fewer raw materials and process them within their own borders instead. The factory was a failure; it met great resistance among workers who feared they would be replaced by machinery. Eventually most of the factory was destroyed in a fire in 1734, leaving only the part of the factory that produced clocks left. The factory continued producing clocks, known for their high quality and low price. Although the popularity of the clocks diminished during the beginning of the 19th century, clock making continues to this day at Stjärnsund, still producing around twenty clocks of the Polhem design per year.

Another product from the factory was the Scandinavian padlock ("Polhem locks", Swedish: Polhemslås), essentially the first design of the variation of padlocks common today. Economically, the factory was unfeasible, but the king at the time, Charles XII, was supportive and gave Polhem freedom from taxes to encourage his efforts.

The factory of Stjärnsund was visited by one of his contemporaries, Carolus Linnaeus, who wrote about the factory in his diaries as Nothing is more optimistic than Stjärnsund ("Intet är spekulativare än Stjärnsund").

Polhem also contributed to the construction of Göta Canal, a canal connecting the east and west coasts of Sweden. Together with Charles XII of Sweden, he planned the construction of parts of the canal, particularly the canal locks in the 18th century; not until 1832, long after his death, was it finished under the supervision of his son, Gabriel Polhem.

Other major contributions made by Polhem were the constructions of dry docks, dams and as mentioned before, canal locks, which he designed together with his assistant and friend, Emanuel Swedenborg.

Polhem was not only active within the field of mechanics, he actively wrote essays concerning medicine, social criticism, astronomy, geology and economics.