Brown's opposition did not want him to go unchallenged and so held an Anti-Brown convention at Milledgeville where they nominated Judge E. A. Nisbet for the governorship. Concerned about the prospect of Brown serving a third term, they argued that too frequent reelections would convert the governorship into a dictatorship and that Brown should willingly relinquish the office to Nisbet. Furthermore, a writer accused Brown of demagoguery and condemned him for nursing his "pet brigadiers," specifically mentioning Brown's conduct in keeping Brigadier General C. J. Phillips in camp at a cost of $100,000 to the state.[2]
In a letter to the people of Georgia, Brown attacked the Milledgeville caucus as being unnecessary, accusing Nisbet of being the candidate of the "politicians". He also accused his opposition of wishing to bring back political strife and oppose the will of the people. Supporters of Brown asserted that Nisbet had spent most of his life in the city and that his health was too delicate to undertake the heavy tasks of the governor's office. Additionally, they accused the Anti-Brown Democrats of being "neutral Yankees."[2]
Despite winning the governorship with 58.63% of the vote, Brown was met with a Nisbet legislature in November 1861, and it was prophesied that this body would "give him the devil." Throughout the duration of the war, this legislature proved to be a persistent source of difficulty for Brown, complicating his governance during the war years.[2]
On election day, October 2, 1861, Democratic nominee Joseph E. Brown won re-election by a margin of 13,691 votes against Anti-Brown Democratic nominee E. A. Nisbet, thereby becoming the first person to hold the office of Governor of Georgia for three terms.[3]
