A prominent group of American Muslim leaders called on Monday for more interfaith dialogue and voiced their support for a controversial Islamic center set to be built two blocks from the former World Trade Center.

Some 20 Muslim organizations including the Council on American-Islamic Relations (CAIR) and the Muslim American Society gave their remarks following a special summit on Sunday held to address the anti-Muslim backlash that has occurred over the proposed center.

"We stand for the constitutional right of Muslims, and Americans of all faiths, to build houses of worship anywhere in our nation as allowed by local laws and regulations," the leaders said in a statement delivered at the Park51 site.

"Ground Zero belongs to all Americans," said Nihad Awad, national executive director of the Council on American-Islamic Relations.

Awad and other Muslim leaders have planned a "Week of Dialogue" for October 22-24 where they will hold meetings at their mosques in a bid to rally further support for the project.

Park51 has been the focus of a heated debate that has lasted for months now about whether or not the religious freedom rights of the center's developers should take precedence over sensitivity issues brought up by 9/11 family members.

Others see the center as part of a larger Islamic plan to establish Sharia law in the United States – an accusation that supporters of the center have labeled as bigotry.

Acts of vandalism and violence against Muslims have followed since the controversy erupted, and several large protests, including dueling demonstrations on this year's 9/11 anniversary, were held.

In a recent interview with ABC News, Park51 developer Imam Feisal Abdul Rauf said that he would have never proposed the center so close to its current site if he would have known of the controversy it has created.

"I would never have done it. I'm a man of peace," Rauf said. "The whole objective of peace work is not to do something that would provoke controversy."

Rauf also noted that he and other developers are seriously considering moving the center, although he remains concerned over the reaction in Muslim countries overseas.

"My major concern with moving it is that the headline in the Muslim world will be Islam is under attack in America," he said. "This will strengthen the radicals in the Muslim world, help their recruitment, this will put our people -- our soldiers, our troops, our embassies, our citizens -- under attack in the Muslim world."

Rauf is currently being protected by the New York Police Department and is avoiding working in the city due to security concerns.