The son of a Sardinian landowning family, born in Sassari, Sardinia, he studied to become a lawyer with a degree in agricultural and commercial law. Segni joined the Italian People's Party (Italian: Partito Popolare Italiano) – the predecessor of the Christian Democratic Party – in 1919. In 1924 he was a member of the party’s national council, until all political organizations were dissolved by Benito Mussolini two years later in 1926. For the next 17 years Segni taught Agrarian Law at the Universities of Pavia, Perugia, and Cagliari; he was also rector of Sassari University.

In 1943 Segni was one of the organizers of the new Christian Democratic Party in Sardinia. He held ministerial positions in many Christian Democrat governments from 1944 onward, despite his frail physique. Time Magazine once quoted a friend: "He is like the Colosseum; he looks like a ruin but he'll be around for a long time." In 1946, he was elected to the Constituent Assembly after World War II and then to parliament in 1948.

Segni made his reputation as Minister of Agriculture (1946 – 1951) under Alcide de Gasperi. He favoured land reform legislation and ordered the expropriation of most of his own estate in Sardinia. He became known as a “white Bolshevik” for his introduction of agrarian reform.

He became Prime Minister in 1955, succeeding Mario Scelba. During Segni’s government the treaties instituting the European Economic Community (EEC) were signed on 25 March 1957, and Italy co-founded the community.

In March 1959, he became Prime Minister again, succeeding Amintore Fanfani, in whose government he had been Minister of Defense.

Segni was elected President of the Italian Republic on 6 May 1962 (854 to 443 votes). He suffered a serious cerebral hemorrhage while working at the presidential palace on 7 August 1964. At the time he was 73 years old and the first prognosis were not positive. He only partially recovered, and he retired from office on 6 December 1964. In the interim, the President of the Senate Cesare Merzagora served as acting president.

Politically, Segni was a moderate conservative opposed to "opening to the centre-left" enabling coalition governments between the Italian Socialist Party (PSI) and the Christian Democrats. Segni was later accused of having tried to instigate a coup d'état (known as Piano Solo) along with General Giovanni De Lorenzo during his presidency to frustrate the opening to the left.

Segni was also a professor of law at University of Sassari. Straightforward, witty and courteous, Segni was more at ease in the classroom or the law court than in the back rooms of Italian politics. He died on 1 December 1972, in Rome. The frail, often ailing Segni, was affectionately called malato di ferro — "the invalid with the iron constitution".

Segni's son, Mariotto Segni, is also a prominent Italian politician.