Rudolph was the son of Count Albert IV of Habsburg and Hedwig, daughter of Count Ulrich of Kyburg, and was born at Limburg Castle near Sasbach am Kaiserstuhl in the Breisgau region. At his father's death in 1239, he inherited large estates from his father around the ancestral seat of Habsburg Castle in the Aargau region of present day Switzerland as well as in Alsace. In 1245 Rudolph married Gertrude, daughter of Count Burkhard III of Hohenberg. As a result, he became an important vassal in Swabia, the former Alemannic German stem duchy.

Rudolph paid frequent visits to the court of his godfather, the Hohenstaufen emperor Frederick II, and his loyalty to Frederick and his son, King Conrad IV of Germany, was richly rewarded by grants of land. In 1254, he was excommunicated by Pope Innocent IV as a supporter of King Conrad, due to ongoing political conflicts between the Emperor, who held the Kingdom of Sicily and wanted to reestablish his power in the Imperial Kingdom of Italy, especially in the Lombardy region, and the Papacy, whose States lay in between and feared being overpowered by the Emperor.

The disorder in Germany during the interregnum after the fall of the Hohenstaufen dynasty afforded an opportunity for Count Rudolph to increase his possessions. His wife was a Hohenberg heiress; and on the death of his childless maternal uncle, Count Hartmann IV of Kyburg in 1264, he also seized his valuable estates. Successful feuds with the Bishops of Strasbourg and Basel further augmented his wealth and reputation, including rights over various tracts of land that he purchased from abbots and others.

These various sources of wealth and influence rendered Rudolph the most powerful prince and noble in southwestern Germany (where the tribal Duchy of Swabia had disintegrated, leaving room for its vassals to become quite independent) when, in the autumn of 1273, the prince - electors met to choose a king after Richard of Cornwall had died in England the year before. Rudolph's election in Frankfurt on 29 September, when he was 55 years old, was largely due to the efforts of his brother-in-law, the Hohenzollern burgrave Frederick III of Nuremberg. The support of Duke Albert II of Saxony and Elector Palatine Louis II had been purchased by betrothing them to two of Rudolph's daughters.

As a result, within the electoral college, King Ottokar II of Bohemia (1230 - 1278), himself a candidate for the throne and related to the late Hohenstaufen king Philip of Swabia (being the son of the eldest surviving daughter), was almost alone in opposing Rudolph. Other candidates were Prince Siegfried I of Anhalt and Margrave Frederick I of Meissen (1257 - 1323), a young grandson of the excommunicated Emperor Frederick II, who however did not yet even have a principality of his own as his father still lived. With the consent of the other electors, Ottokar's dissent was neglected, and by the admission of Duke Henry XIII of Lower Bavaria, Rudolph gained all seven votes.

Rudolph was crowned in Aachen Cathedral on 24 October 1273. To win the approbation of the Pope, Rudolph renounced all imperial rights in Rome, the papal territory, and Sicily, and promised to lead a new crusade. Pope Gregory X, in spite of Otakar's protests, not only recognized Rudolph himself, but persuaded King Alfonso X of Castile (another grandson of Philip of Swabia), who had been chosen German (anti-)king in 1257 as the successor to Count William II of Holland, to do the same. Thus, Rudolph surpassed the two heirs of the Hohenstaufen dynasty that he had earlier served so loyally.

In November 1274 it was decided by the Imperial Diet at Nuremberg that all crown estates seized since the death of the Emperor Frederick II must be restored, and that King Ottokar II must answer to the Diet for not recognizing the new king. Ottokar refused to appear or to restore the duchies of Austria, Styria and Carinthia with the March of Carniola, which he had claimed through his first wife, a Babenberg heiress, and which he had seized while disputing them with another Babenberg heir, Margrave Hermann VI of Baden. Rudolph refuted Ottokar's succession to the Babenberg patrimony, declaring that the provinces reverted to the Imperial crown due to the lack of male-line heirs (a position that however conflicted with the provisions of the Austrian Privilegium Minus). King Ottokar was placed under the imperial ban; and in June 1276 war was declared against him.

Having persuaded Ottokar's former ally Duke Henry XIII of Lower Bavaria to switch sides, Rudolph compelled the Bohemian king to cede the four provinces to the control of the royal administration in November 1276. Rudolf then re-invested Ottokar with the Kingdom of Bohemia, betrothed one of his daughters to Ottokar's son Wenceslaus II, and made a triumphal entry into Vienna. Ottokar, however, raised questions about the execution of the treaty, made an alliance with some Piast chiefs of Poland, and procured the support of several German princes, again including Henry XIII of Lower Bavaria. To meet this coalition, Rudolph formed an alliance with King Ladislaus IV of Hungary and gave additional privileges to the Vienna citizens. On 26 August 1278, the rival armies met at the Battle on the Marchfeld, where Ottokar was defeated and killed. The March of Moravia was subdued and its government entrusted to Rudolph's representatives, leaving Ottokar's widow Kunigunda of Slavonia, in control of only the province surrounding Prague, while the young Wenceslaus II was again betrothed to Rudolph's youngest daughter Judith.

Rudolph's attention next turned to the possessions in Austria and the adjacent provinces, which were taken into the royal domain. He spent several years establishing his authority there but found some difficulty in establishing his family as successors to the rule of those provinces. At length the hostility of the princes was overcome. In December 1282, in Augsburg, Rudolph invested his sons, Albert and Rudolph II, with the duchies of Austria and Styria and so laid the foundation of the House of Habsburg. Additionally, he made the twelve year old Rudolph Duke of Swabia, a merely titular dignity, as the duchy had been without an actual ruler since Conradin's execution. The 27 year old Duke Albert (married since 1274 to a daughter of Count Meinhard II of Gorizia - Tyrol (1238 - 95)) was capable enough to hold some sway in the new patrimony.

In 1286 King Rudolf fully invested the Duchy of Carinthia, one of the provinces conquered from Ottokar, to Albert's father-in-law Count Meinhard. The Princes of the Empire did not allow Rudolf to give everything that was recovered to the royal domain to his own sons, and his allies needed their rewards too. Turning to the west, in 1281 he compelled Count Philip I of Savoy to cede some territory to him, then forced the citizens of Bern to pay the tribute that they had been refusing, and in 1289 marched against Count Philip's successor, Otto IV, compelling him to do homage.

In 1281 his first wife died. On 5 February 1284, he married Isabella, daughter of Duke Hugh IV of Burgundy, the Empire's western neighbor in the Kingdom of France.

Rudolph was not very successful in restoring internal peace. Orders were indeed issued for the establishment of landpeaces in Bavaria, Franconia and Swabia, and afterwards for the whole Empire. But the king lacked the power, resources, or determination, to enforce them, although in December 1289 he led an expedition into Thuringia where he destroyed a number of robber-castles.

In 1291, he attempted to secure the election of his son Albert as German king. However, the electors refused claiming inability to support two kings, but in reality, perhaps, leery of the increasing power of the House of Habsburg. Upon Rudolph's they elected Count Adolf of Nassau.

Rudolph died in Speyer on 15 July 1291, and was buried in the Speyer Cathedral. Although he had a large family, he was survived by only one son, Albert, afterwards the German king Albert I. Most of his daughters outlived him, apart from Katharina who had died in 1282 during childbirth and Hedwig who had died in 1285/6.

Rudolph's reign is most memorable for his establishment of the House of Habsburg as a powerful dynasty in the southeastern parts of the realm. In the other territories, the centuries long decline of the Imperial authority since the days of the Investiture Controversy continued, and the princes were largely left to their own devices.

In the Divine Comedy, Dante finds Rudolph sitting outside the gates of Purgatory with his contemporaries, who berate him as "he who neglected that which he ought to have done".