The man who was ousted from Georgia's highest office after tackling the controversial issue of the Confederate battle emblem on the state flag says it's time for it to come down everywhere, in every state.

In 2001, Governor Roy Barnes led the effort to change Georgia's flag to remove the Rebel cross from the flag.  Today, he says that the symbol ought not be on anything that represents a modern Georgia--including some specialty license plates.

"I think it should come off," says Barnes, "and I think the governor and the leaders, as they walk cautiously through this, I think they'll come to the same conclusion."

About a half-hour after Governor Nathan Deal said on Tuesday that he would not support a redesign of the Sons of Confederate Veterans license plate, he called reporters back into his office to walk back that comment.  He said that he wants a redesign of the plate.

But should the group have a specialty tag at all? Former Governor Barnes was asked Wednesday.

"What are we going to say when ISIL wants a tag?" Barnes asked in reply.

Barnes also addressed the question of whether lawmakers tackling this issue 14 years after he did are walking through a minefield that he somewhat defused.  He replied that the political risks are not as great today--particularly in light of the church massacre in Charleston, SC.  A racist 21-year-old gunned down nine black people who were at a Wednesday night Bible study.  Dylann Roof had earlier posed for photographs with the Rebel flag.

"The Confederate flag is a racial symbol," says Barnes.  "Everybody's trying to say, 'Well, it's not.'  Well, it is.  I mean, look at the young fella over in Charleston.  Look at anyplace where the schools were integrated and there was a protest.  It was all Confederate flags.  It's a racial symbol.  And race is the third rail in southern politics--and it may be in national politics, also."

Barnes said yes, the flag is a part of our history, but a small part.  He revealed that after he left the governor's office, he said as much to a Confederate Memorial Day gathering at Oakland Cemetery.

"I spoke about how it was wrong for our forefathers to rebel, particularly over trying to preserve slavery," Barnes recalled.

For that reason, historical societies and museums and Confederate cemeteries, he says, are appropriate places for the Rebel flag, but setting aside a Confederate Memorial Day or Month is not appropriate.

"History should be balanced, and you should not have an over-emphasis on a time that we should really regret, when the South rebelled for the wrong reasons," he says.  "Yes, we treasure the valor of those who fought, like my great-grandfather.  But they were fighting for the wrong reasons."

Barnes has a great-grandfather who fought for the Confederacy in the Civil War.  Corbin Barnes was one of many captured at Vicksburg, and who was freed after swearing an oath of loyalty to the Union.  Another great-grandfather from the same north Georgia community, he says, fought for the Union.  As a kid, he says, he, too, had affection for the Rebel flag.

"Listen.  I was in high school before I knew the South had lost the Civil War," he chuckles.  "I was just as big [into] all that.  Southern and Confederate.  Education and life experience, he said, changed that.  He studied the reason the war was fought and read all the secession ordinances.  Not one, he said, mentions "high tariffs," which some people try to argue was the cause of the war, but all of them mention slavery.

Barnes pointed out that the issue of the flag simply "came up on my watch" when he was governor, and he realized there was no putting it off any longer.  He saw how the similar debate was paralyzing the state of South Carolina then, too.  The massive Confederate battle emblem on Georgia's flag was placed there in 1956, as an angry response to the Supreme Court's order to integrate schools.

Barnes says there is "no doubt" that changing the flag cost him his job as the state's top elected official.

"And if I had it to do over again, I would do exactly the same thing, even if I knew I were going to be defeated," says Barnes.

As for Georgia's current flag, which is essentially like the first national flag of the Confederate States of America but with Georgia's state seal added, Barnes says he won't opine on whether it should also get a makeover.  He said that then-State Representative Tyrone Brooks, a Democrat who was president of the Georgia Association of Black Elected Officials, chose that flag.

"It doesn't have the reputation as much," says Barnes.  "So I think that in careful consideration, the flag's probably okay.  But, you know, whether it should is something that should be discussed."