Joseph Edwin Johnson was born in Vernon, Alabama, on July 9, 1933. He graduated from Hueytown High School in Hueytown, Alabama in 1951. He received a Bachelor of Arts degree in History and Political Science in 1955 from Birmingham-Southern College and a certificate in Public Administration in 1956 from the Southern Regional Training Program. He received a Master of Arts degree in Political Science in 1960 and a Doctorate in Education in 1968 with a major in Educational Administration and Supervision from the University of Tennessee. He was a member of Phi Beta Kappa, Phi Kappa Phi, Golden Key Honor Society, and Omicron Delta Kappa. From 1956-1958 he served as a Sergeant in the United States Army.

Johnson worked as a research assistant for the University of Tennessee's Bureau of Public Administration from 1958 to 1960, during which time he published studies on property tax assessment and property tax collection in Knoxville and Knox County, Tennessee. He also began working at the University of Tennessee as an instructor and research associate in 1958. Since then, he has worked as Senior Budget Analyst and Executive Assistant to Governor Buford Ellington (1960-1963), Executive Assistant to the President of the University of Tennessee to President Andrew Holt (1963-1969), Chancellor of the University of Tennessee's Center for the Health Sciences in Memphis (1970-1973), and Vice President for Development at UT (1970, 1973-1991). Johnson served as the 19th President of the University of Tennessee from 1991 until his retirement in 1999. In 2003-2004 he returned for a brief period as interim president.