Alfred was born in Toronto, Ontario, in 1898 to an English Quaker father and a Canadian mother. At age 9 he moved to Guelph and to Hamilton at age 14. He left school early at age 15 to work as an apprentice at a Hamilton lithography company. At the same time, he studied art at Central Technical School. Casson left Hamilton at age 17 and moved to Toronto. The first public exhibition of his work was at the Canadian National Exhibition, in 1917. He was hired by the commercial art firm Rous and Mann.

At Rous and Mann, Alfred was influenced and apprenticed by Group of Seven member Franklin Carmichael to sketch and paint on his own. Carmichael introduced Casson to The Arts and Letters Club of Toronto, where he met many well-known artists, including other Group of Seven members.

During the 1920s, Casson continued to paint during his spare time alone and with the Group of Seven. Alfred enjoyed watercolour and in 1925 along with Carmichael and F.H. Brigden, founded the Canadian Society of Painters in Watercolour.

After Frank Johnston, a Group of Seven member, left the group in 1921, Casson seemed like an appropriate replacement. Later in 1926, he was invited by Carmichael to become a Group of Seven member. Casson joined excitedly. He left Rous and Mann in 1927 and joined Sampson Matthews. He became their art director and later their vice-president.

In 1929 Casson married a woman named Margaret. His father died shortly after his marriage, and he had to take care of and support his widowed mother. After the ending of the Group of Seven, he co-founded the Canadian Group of Painters in 1933.

Casson developed a painting style with clear colours and background designs. Alfred "retired" in 1957 at age 60 but continued to paint full time. A.J. Casson died in 1992 at age 94 and is buried on the grounds of the McMichael Canadian Art Collection, along with six other Group of Seven members.