Remsen was born in New York City and earned an M.D. from the College of Physicians and Surgeons in 1867 to please his parents. He then traveled to Germany to study chemistry - his true passion. He earned a Ph.D. from University of Göttingen in 1870. In 1875, after researching pure chemistry at University of Tübingen, Remsen returned to the United States and became a professor at Williams College, where he wrote the popular "Theoretical Chemistry". His book and reputation brought him to the attention of Daniel Coit Gilman who invited him to become one of the original faculty of Johns Hopkins University. He accepted and founded the department of chemistry there, where he ran his own laboratory. In 1879 he founded the American Chemical Journal which he edited for 35 years.

In 1879 he made the greatest discovery of his career by accident. When he ate rolls at dinner after a long day in the lab researching coal tar derivatives, he noticed that the rolls tasted initially sweet but then bitter. Since his wife tasted nothing strange about the rolls, Remsen tasted his fingers and noticed that the bitter taste was probably from one of the chemicals in his lab. The next day at his lab he tasted the chemicals that he had been working with the previous day and discovered that it was the oxidation of o-toluenesulfonamide he had tasted the previous evening. He named the substance saccharin and he and his research partner Constantin Fahlberg published their finding in 1880. Later Remsen became angry after Fahlberg patented saccharin, claiming that he had discovered saccharin.

In 1901 Remsen was appointed the president of Johns Hopkins, where he proceeded to found a School of Engineering and helped establish the school as a research university. He introduced many of the German laboratory techniques he had learned and wrote several important chemistry textbooks. In 1912 he stepped down as president and retired to Carmel, California.

In 1923 he was awarded the Priestley medal. He died on March 4, 1927.

After his death the new chemistry building was named after him at Johns Hopkins. His ashes are located behind a plaque in Remsen Hall; he is the only person buried on campus. According to legend, undergraduates who rub the plaque the night before their chemistry exam will do well.

His Baltimore house was added to the National Register of Historic Places and declared a National Historic Landmark in 1975.