Joseph Alexander Mabry, Jr. was born in Knox County, Tennessee on January 26, 1826 to Joseph Alexander Mabry and Alice Hare Scott. Having made money through land and railroad speculation, he was a prominent businessman in Knoxville throughout his life and was influential in the city's development. In 1853, with his brother-in-law William G. Swann, he donated the initial land for Knoxville's Market Square. By the early 1860s, Mabry was one of the area's largest slaveholders and he supported the state's secession leading up the Civil War, but would become more moderate as the war progressed.

In 1852, Mabry married Laura Evelyn and a few years later, built her a home in East Knoxville (the Mabry-Hazen House) where he lived for the remainder of his life. They had 14 children together. Following some lingering personal and business disputes, banker Thomas O'Conner shot Mabry on the morning of October 19, 1882 on Gay Street in Knoxville. Mabry's son, Joseph Mabry III, was present and shot O'Conner, who fired one fatal shot back at the son before himself dying.