I'd certainly have something against not allowing it, that would be called the First Amendment. It seems the US views any religion other than Christianity (and to an extend Judaism) with immense cynicism to the point that they seem to implicitly establish a religion. The debate over a Mosque at Ground Zero typifies this cynicism, and thus I would be critical of the efforts to oppose a mosque being constructed there.
The case to block the construction seems to just be an argument against A religion, not ALL religion. For a start to block the mosque on the basis that Muslims were responsible for 9/11 is ridiculous in itself. The perpetrators of 9/11 no better represent Islam than the KKK represents Christianity. Therefore there is no basis in that argument of those who oppose the mosque.