Joseph Alexander Mabry was born in Knox County, Tennessee, on January 26, 1825, to Joseph Alexander and Alice Hare Scott Mabry. Having made money through land and railroad speculation, he was a prominent businessman in Knoxville throughout his life and was influential in the city's development. In 1853, with his brother-in-law William G. Swann, he donated the initial land for Knoxville's Market Square. By the early 1860s, Mabry was one of the area's largest slaveholders and supported the state's secession leading up to the American Civil War, but would become more moderate as the war progressed.

In 1852, Mabry married Laura Evelyn Churchwell (1833-1906); in 1858, Mabry had a home built in East Knoxville (Mabry-Hazen House, listed on the National Register of Historic Places) where he lived for the remainder of his life. They had 14 children together. Following some lingering personal and business disputes, banker Thomas O'Conner shot and killed Mabry on the morning of October 19, 1882, on Gay Street in Knoxville. Mabry's son, Joseph Mabry (b. 1855), was present and shot O'Conner, who then fired one fatal shot back at the son before himself dying.