Mitt Romney’s religion, a subject he prefers not to talk about so far in the 2012 presidential campaign, inevitably comes up in political discourse. It did again this week in a way that forced him to respond.

Discussing the likely Republican presidential candidate’s difficulty in attracting Latino voters, Montana's Democratic Governor Brian Schweitzer, pointed out that Romney’s family “came from a polygamy commune in Mexico.”

Mr. Romney is Mormon, descended from English converts to Mormonism who immigrated to the United States in the 19th century. The denomination (also known as The Church of Jesus Christ of Latter-day Saints) encouraged polygamy or the taking of “plural wives” by Mormon men until the practice was officially ended in 1890.

At a time when Mormonism was under attack in the United States, resulting in state and federal legislation against polygamy, some Mormon groups – including Romney’s ancestors – fled to Mexico where they were able to continue the practice until the Mexican Revolution forced them to return to the United States.

As detailed in “The Real Mitt Romney” by Michael Kranish and Scott Helman of The Boston Globe, the former Massachusetts governor’s early ancestors did practice polygamy in North America, but that did not include his grandfather’s family in the United States or Mexico.

Gov. Schweitzer told the Daily Beast it was “ironic” that Romney could not use the fact that his father (George Romney, who became governor of Michigan and ran for president) was born in Mexico to attract Latino voters.

“I am not alleging by any stretch that Romney is a polygamist and approves of [the] polygamy lifestyle, but his father was born into [a] polygamy commune in Mexico,” Schweitzer said, who also noted that even a distant connection with polygamy probably would not sit well with women voters – another group Romney is seeking to attract.

To those voters wary of Romney’s religion (including some evangelical Christians), it does not help the candidate that small offshoot groups disavowed by the church continue to practice – and sometimes be prosecuted for – polygamy in the Southwest.

Responding to Gov. Schweitzer’s comment, Romney told Fox News: "My dad's dad was not a polygamist. My dad grew up in a family with a mom and a dad and a few brothers and one sister."

"They lived in Mexico and lived a very nice life there from what I understand and then when [my father] was 5 or 6 years old there was a revolution in Mexico,” he said. “They escaped…. My dad had a very tough upbringing."

Conservatives have criticized Schweitzer for bringing up Romney’s religion in this way, pointing out that President Obama’s Kenyan father was married to more than one woman at a time.