The practice has been illegal under federal law in the U.S. since the 1860s.
The Browns are members of the Apostolic United Brethren and say they practice polygamy as part of their religious beliefs.
And like most polygamists, Brown only has a valid marriage license with his first wife, Meri. He married the other three in religious ceremonies. They consider themselves "spiritually married."
Fewer than a dozen Utah cases have challenged the law — none successfully. It has most commonly been used to prosecute polygamists typically when the defendants were also charged with other crimes, such as child abuse. The criminal investigation into Warren Jeffs, for instance, and his polygamous Fundamentalist Church of Jesus Christ of Latter Day Saints group, began as a probe into child sex abuse allegations.
He is now serving a life sentence in Texas after convictions on child sex and bigamy charges.
Some experts liken the issue to gay marriage or sodomy bans, but note neither carries the historical stigma that comes with polygamy.
Polygamy "opens up a whole can of legal worms that is not really opened with the same sex marriage issue," said Stanford University constitutional law expert Jane Schacter. She noted that polygamy throughout history has been associated with underage marriages and child sex abuse.
Regardless of whether there are polygamous relationships among consenting adults where no other crimes are occurring, that historical stigma persists, she said.
"It's not an issue I think will be laughed out of court, but I don't think it is poised for a victory at this point," Schacter added. "Contemporary law under the First Amendment regarding the exercise of religion is just not very friendly to the claims of polygamists."
Copyright 2012 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.