Born in Saint-Dié, in the Vosges département, France, he studied law, and was called to the bar at Paris, but soon went into politics, contributing to various newspapers, particularly to Le Temps. He attacked the Second French Empire with great violence, directing his opposition especially against Baron Haussmann, prefect of the Seine département. Elected republican deputy for Paris in 1869, he protested against the declaration of war with Germany, and on 6 September 1870 was appointed prefect of the Seine by the Government of National Defense.

In this position he had the difficult task of administering Paris during the siege, and after the Paris Commune was obliged to resign (5 June 1871). From 1872 to 1873 he was sent by Adolphe Thiers as minister to Athens, but returned to the chamber as deputy for the Vosges, and became one of the leaders of the republican party. When the first republican ministry was formed under WH Waddington on 4 February 1879, he was one of its members, and continued in the ministry until 30 March 1885, except for two short interruptions (from 10 November 1881 to 30 January 1882, and from 29 July 1882 to 21 February 1883), first as minister of education and then as minister of foreign affairs. A leader of the Opportunist Republicans faction, he was twice premier (1880 - 1881 and 1883 - 1885).

Two important works are associated with his administration, the non-clerical organization of public education, and the beginning of the colonial expansion of France. Following the republican programme he proposed to destroy the influence of the clergy in the university and found his own system of republican schooling. He reorganized the committee of public education (law of 27 February 1880), and proposed a regulation for the conferring of university degrees, which, though rejected, aroused violent polemics because the 7th article took away from the unauthorized religious orders the right to teach. He finally succeeded in passing his eponymous laws of 16 June 1881 and 28 March 1882, which made primary education in France free, non-clerical (laïque) and mandatory. In higher education, the number of professors, called the "hussards noirs de la République" ("Republic's black hussars") because of their Republican support, doubled under his ministry.

The education policies establishing French language as the language of the Republic have been contested in the second half of the 20th century insofar as, if they played an important role in unifying the French nation-state and the Third Republic, they also nearly provoked the extinction of several regional languages.

After the military defeat of France by Germany in 1870, Ferry formed the idea of acquiring a great colonial empire, principally for the sake of economic exploitation. In a speech before the Chamber of Deputies on 28 July 1885, he declared that "the superior races have a right because they have a duty: it is their duty to civilize the inferior races." Ferry directed the negotiations which led to the establishment of a French protectorate in Tunis (1881), prepared the treaty of 17 December 1885 for the occupation of Madagascar; directed the exploration of the Congo and of the Niger region; and above all, he organized the conquest of Annam and Tonkin in what became Indochina. The last endeavor led to a war with the Manchu Empire, whose Qing Dynasty had a claim of suzerainty over the two provinces. The excitement caused in Paris by the sudden retreat of the French troops from Lang Son during this war led to the Tonkin Affair: his violent denunciation by Clemenceau and other radicals, and his downfall on 30 March 1885. Although the treaty of peace with the Manchu Empire (9 June 1885), in which the Qing Dynasty ceded suzerainty of Annam and Tonkin to France, was the work of his ministry, he would never again serve as premier.

The desire for a monarchy was still strong in France even under the Third Republic — the comte de Chambord having made a bid early in its history. A committed republican, Ferry proceeded to a wide-scale "purge" by dismissing many known monarchists from top positions in the magistrature, army and civil and diplomatic service.

The key to understanding Ferry's unique position in Third Republic history is that until his political critic, Georges Clemenceau became Prime Minister twice in the 20th Century Ferry has the longest tenure as Prime Minister under that regime. He also played with political dynamite that eventually destroyed his success. Ferry (like his 20th Century equivalent Joseph Caillaux) believed in not confronting Wilhelmine Germany by threats of a future war of revenge. Most French politicians in the middle and right saw it as a sacred duty to one day lead France again against Germany to reclaim Alsace-Lorraine, and avenge the awful defeat of 1870. But Ferry realized that Germany was too powerful, and it made more sense to cooperate with Otto von Bismarck and avoid trouble. A sensible policy - but hardly popular.

Bismarck was constantly nervous about the situation with France. Although he had despised the ineptness of the French under Napoleon III and the government of Adolphe Thiers and Jules Favre, he had not planned for all the demands he presented the French in 1870. He only wished to temporarily cripple France by the billion franc reparation, but suddenly he was confronted by the demands of Marshals Albrecht von Roon and Helmut von Moltke (backed by Emperor Wilhelm I) to annex the two French provinces as further payment. Bismarck, for all his abilities regarding manipulating events, could not afford to anger the Prussian military. He got the two provinces, but he realized it would eventually have severe future repercussions.

Bismarck was able to ignore the French for most of the 1870s and early 1880s, but as he found problems with his three erstwhile allies (Austria, Russia, and Italy) he realized France might one day take advantage of this (as it did with Russia in 1894). When Ferry came up with a radically different approach to the situation and offered an olive branch Bismarck reciprocated. A Franco-German friendship would alleviate problems of siding with either Austria or Russia, or Austria and Italy. Bismarck approved of the colonial expansion that France pursued under Ferry. He only had some problems with local German imperialists who were critical that Germany lacked colonies, so he found a few in the 1880s, making certain he did not confront French interests. But he also suggested Franco-German cooperation on the imperial front against the British Empire, thus hoping to create a wedge between the two Western European great powers. It did as a result, leading to a major race for influence across Africa that nearly culminated in war in the next decade, at Fashoda in the Sudan in 1898. But by then both Bismarck and Ferry were dead, and the rapproachment policy died when Ferry lost office. As for Fashoda, while it was a confrontation, it led to Britain and France eventually discussing their rival colonial goals, and agreeing to support each other's sphere of influence - the first step to the Entente Cordiale between the countries in 1904.

Ferry remained an influential member of the moderate republican party, and directed the opposition to General Boulanger. After the resignation of Jules Grévy (2 December 1887), he was a candidate for the presidency of the republic, but the radicals refused to support him, and he withdrew in favour of Sadi Carnot.

In 1887, a man named Aubertin attempted to assassinate Jules Ferry, who later died from complications attributed to this wound on 17 March 1893. The Chamber of Deputies gave him a state funeral.