James Allen Smith was born in Elberton, Georgia in 1850 to Burgess and Harriet Wills Jordan Smith. In 1873, after attending Eastman Business College in Atlanta, Georgia, he moved to Knoxville, Tenn. and established J. Allen Smith Wholesale Grain Company. In 1881, he added a mill to his company. A year later, he started Knoxville City Mills with four other partners. In 1888, Smith merged Knoxville City Mills with his original company to establish J. Allen Smith & Company. A manufacturer of flour and cornmeal, the most popular product produced by the company was White Lily flour. By 1905, J. Allen Smith & Co. was one of the largest mills in the South. Smith ran the mill and oversaw the company until his death in 1920.

Smith was very active in the Knoxville business community as well as a contributor to many local charities. He married Lillie Powell in 1878; they only had one son to survive into adulthood, C. Powell Smith, who succeeded his father as president of J. Allen Smith and Company.