Hall was born in Hingham, Massachusetts, the oldest of four children. His parents, James Hall Sr. and Sousanna Dourdain Hall, had emigrated from England two years earlier. Hall developed an early interest in science and enrolled in Rensselaer Polytechnic Institute, a recently established college that emphasized student participation and focused on science. He was a student of Amos Eaton and Ebenezer Emmons, both notable geologists. Hall graduated with honors in 1832, received his masters degree in 1833, and remained at Rensselaer to teach chemistry and later geology.

In 1836 a multi-year survey was established to collect information on the geology and natural history of New York. For purposes of the survey, the state was divided into four districts and Hall became assistant geologist for Ebenezer Emmons, chief of the Second District. Hall’s initial assignment was to study iron deposits in the Adirondack Mountains. The following year the survey was reorganized and Hall put in charge of the Fourth District in western New York. Other notable geologists working on the survey included Lardner Vanuxem and Timothy Conrad. Working together, the survey staff developed a stratigraphy for New York and set a precedent for naming stratigraphic divisions based on local geography.

At the end of the survey in 1841, Hall was named the first state paleontologist. In 1843 he made his final report on the survey of the fourth geological district, which was published as "Geology of New York," Part IV. (1843). It was received with much acclaim and became a classic in the field. Hall had built a solid reputation and was to devote the rest of his life to stratigraphic geology and invertebrate paleontology.

Hall built a laboratory in Albany, New York, which became an important center of study and training for aspiring geologists and paleontologists. Many notable scientists began their career serving an apprenticeship with Hall, including Fielding Meek, Charles Walcott, Charles Beecher and Josiah Whitney. Now known as the James Hall Office, the laboratory was designated a National Historic Landmark in 1976.

Among his many works, James Hall identified that stromatolite fossils discovered at Petrified Sea Gardens, a site near Saratoga Springs, New York, that is now another National Historic Landmark, were originally organic.

After his work in New York, Hall extended his studies to other regions of the country. In 1850 Hall participated in a geological survey of northern Michigan and Wisconsin where he identified the first fossil reefs ever found in North America. He was appointed state geologist for Iowa (1855 – 1858) and Wisconsin (1857 – 1860). In addition, several other state survey programs sought out Hall for his expertise and advice. In 1866 he was made director of New York’s Museum of Natural History in Albany. In 1893 he was appointed the State Geologist of New York.

Between 1847 and 1894 Hall published 13 volumes of The Paleontology of New York, his principal contribution in the field. This massive work consisted of over 4500 pages and 1000 full-page illustrations. In addition, Hall wrote more than 30 other books, published over 200 papers, and contributed sections to several federal and state publications on geology.

He was a founding member of the National Academy of Sciences and the first president of the Geological Society of America. He was one of the founders of the International Geologic Congress and served as a vice-president at their sessions in Paris, Bologna, and Berlin. He was elected one of the fifty foreign members of the Geological Society of London in 1848, and in 1858 was awarded its Wollaston Medal. In 1884 he was elected correspondent of the French Academy of Sciences.

At the age of 85 he traveled to St. Petersburg to attend the International Geological Congress and also participate in an expedition to the Ural Mountains. Hall died two years later in Bethlehem, New Hampshire. He is buried at the Albany Rural Cemetery, Albany, New York.

There is a residence hall at Rensselaer Polytechnic Institute named after James Hall. It is officially known as Hall Hall.