Rennie, a farmer's younger son, was born at Phantassie, near East Linton, East Lothian, Scotland, and showed a taste for mechanics at a very early age, and was allowed to spend much time in the workshop of Andrew Meikle, millwright, the inventor of the threshing machine, who lived at Houston Mill on the Phantassie estate. After receiving a rudimentary education at the parish school of Prestonkirk Parish Church, he was sent to the burgh school at Dunbar, and in November 1780 he matriculated at the University of Edinburgh, where he remained until 1783. His older brother George remained to assist in the family agricultural business, achieving notability in this arena.

He seems to have employed his vacations in working as a millwright, and so to have established a business on his own account. At this early date the originality of his mind was exhibited by the introduction of cast iron pinions instead of wooden trundles. In 1784 he took a journey south for the purpose of enlarging his knowledge, visiting James Watt at Soho, Staffordshire. Watt offered him an engagement, which he accepted, and after a short stay at Soho he left for London in 1784 to take charge of the works at the Albion Flour Mills, Blackfriars, for which Boulton & Watt were building a steam engine. The machinery was all designed by Rennie, a distinguishing feature being the use of iron instead of wood for the shafting and framing. About 1791 he started in business as a mechanical engineer on his own account in Holland Street, Blackfriars, whence he and his successors long conducted engineering operations of vast importance. (In the same year, the Albion Flour Mills were destroyed by arson).

In 1791, he moved to London and set up his own engineering business, having by then begun to expand into civil engineering, particularly the construction of canals. His early projects included the Lancaster Canal (started 1792), the Chelmer and Blackwater Navigation (1793), the Crinan Canal (1794), Rudyard Lake (1797) and the Dundas Aqueduct on the Kennet and Avon Canal (1800). This was followed by the Rochdale Canal, which passes through difficult country between Rochdale and Todmorden. He subsequently in 1802 revised the plans for the Royal Canal of Ireland from Dublin to the Shannon near Longford. For many years he was engaged in extensive drainage operations in the Lincolnshire and Norfolk fens (1802 – 1810), and in the improvement of the River Witham. The Eau Brink Cut, a new channel for the River Ouse, was completed just before his death.

Over the next few years Rennie also attained a deserved reputation as a builder of bridges, combining stone with new cast iron techniques to create previously unheard - of low, wide, elliptical arches, at Leeds Bridge, and in London at Waterloo Bridge (1811 – 1817), with its nine equal arches and perfectly flat roadway (thought to be influenced by Thomas Harrison's design of Skerton Bridge over the River Lune in Lancaster). His later efforts in this line also show that he was a skilful architect, endowed with a keen sense of beauty of design. Waterloo Bridge, London Bridge – built from his design, though not completed until 1831, after his death – and Southwark Bridge (1815 – 1819) best attest his skill. He also designed the Old Vauxhall Bridge.

Rennie was also responsible for designing and building docks at Hull, Liverpool, Greenock, London (London, East India and West Indiadocks), and Leith and improving the harbours and dockyards at Chatham, Devonport, Portsmouth, Holyhead, Ramsgate and Sheerness. He devoted much time to the preparation of plans for a government dockyard at Northfleet, but they were not carried out. Rennie's last project was London Bridge, still under construction when he died in 1821 but completed by his son, also John Rennie.

Dunleary harbour of 'Asylum' was a very difficult and important project, because it was critical to maintain an effective communication link between Ireland and London, the seat of Government. An Act of Parliament of 1816 (56 Geo.III. Cap 62) authorised the commencement of work building the harbour. Originally it was intended that only one pier (the East Pier) would be built (3,500 feet long), and when John Rennie was appointed Directing Engineer for the work, he insisted that a single pier would result in sand drifting behind the pier and that a second West Pier (4,950 feet long) would prevent this from occurring. He was correct as the sand has built up behind the west pier. The harbour once built was renamed 'The Royal Harbour of Kingstown' in 1821 on the occasion of the visit of George IV. The material for the harbour is Dalkey Hill granite. The granite was provided by Richard Toucher (a long time campaigner for the new harbour) at no cost to the construction team. The foundations of the piers are 300'-0" wide and 24'-0" below low water level. Many options were considered for the width of the space between the two pier heads. Rennie Snr. wrote to the Harbour Commissioners that the opening should be 430'-0" wide with the pier heads turned into the harbour to control swells within the harbour. His demands were never met and the harbour opening was built at 1,066'-0". This was clearly too wide and was subsequently reduced to 760'-0". It is worth noting that the Bay formed between Kingstown Harbour and The Forty Foot is still known as Scotsman's Bay after the great John Rennie.

The Bell Rock Lighthouse, near the entrance to the Firths of Forth and Tay, was built during 1807 and 1810. Rennie was, by some, credited with the design and execution, but there seems little doubt that he was only nominally responsible for the great undertaking. Robert Stevenson, surveyor to the Commissioners of Northern Lights, drew the original plans and, at his suggestion, the commissioners called Rennie into counsel when the works were begun, bestowing on him the honorary title of chief engineer. Stevenson did not accept the modifications proposed by Rennie, but the two men remained on friendly terms. Rennie visited the lighthouse while it was being built. According to Robert Louis Stevenson, Stevenson's grandson, the Commissioners of Northern Lights paid Stevenson alone when the lighthouse was completed. When Stevenson died in 1850, the Commissioners put on record in their minutes that to him was 'due the honour of conceiving and executing the Bell Rock lighthouse'. But Rennie and his friends always claimed that the general advice which Rennie gave Stevenson entitled him to rank the building among his own achievements.

Of all Rennie's works, that which appeals most strongly to the imagination is perhaps the breakwater at Plymouth Sound, consisting of a wall a mile in length across the Sound, in deep water, and containing 3,670,444 tons of rough stone, besides 22,149 cubic yards (16,934 m³) of masonry on the surface. It was constructed in order to provide safe passage for naval vessels entering the river Tamar (Hamoaze) at Devonport. This colossal work was first proposed in a report by Rennie, dated 22 April 1806; an order in council authorising its commencement was issued on 22 June 1811, and the first stone was deposited on 12 August following. The work was completed by his son, Sir John Rennie, and by Joseph Whidbey.

Rennie was a man of unbounded resource and originality. During the improvement of Ramsgate harbour he made use of the diving - bell, which he greatly improved. He is generally credited with the invention of a form of steam dredging machine with a chain of buckets, but in this he seems to have been anticipated by Sir Samuel Bentham. But he was certainly the first to use it on an extensive scale, which he did during the construction of the Hull docks (1803 - 9), when he devised a steam dredger to overcome the difficulties of that particular work, and apparently without any knowledge of Bentham's invention. Another expedient was the use of hollow walls, which was suggested by the necessity of providing an extensive bearing surface for the foundations of a wall in loose ground. Walls built upon this plan were largely used by Rennie.

The distinguishing characteristics of Rennie's work were firmness and solidity, and it has stood the test of time. He was most conscientious in the preparation of his reports and estimates, and he never entered upon an undertaking without making himself fully acquainted with the local surroundings. He was devoted to his profession, and, though he was a man of strong frame and capable of great endurance, his incessant labours shortened his life. He was elected Fellow of the Royal Society on 29 March 1798, and in 1815 he served as a Manager of the newly built London Institution.

He died, after a short illness, at his house in Stamford Street, London, on 4 Oct. 1821, and was buried in St. Paul's Cathedral. He married early in life Martha, daughter of E. Mackintosh, who predeceased him; by her he left several children, two of whom, George and Sir John, went on to notability in their own rights.

The Ruislip Lido Railway, based at Ruislip Lido (one of the dams John Rennie built), has named one of its locomotives after him. The Bo - Bo type diesel, 'John Rennie' was built in 2004 and is one of the main engines of the line.