Hecklers have called Australian Prime Minister Anthony Albanese "cheap and late" during his visit to a synagogue destroyed in a suspected terrorist firebombing, as he promised support for its rebuild.

The Adass Israel Synagogue was set alight in a pre-dawn attack with people inside on Friday.

Four days later, Albanese made a short visit, greeted by an irate crowd.

As he left one woman heckled "your words are cheap and late" and "you are late. You let this happen, buddy".

Another asked if he was "going to the Australian Open", referring to Albanese playing tennis in the days after the arson.

Rae Anaf, a Jewish community member, said the government did not care about Jewish people, labelling his visit as "too little, too late".

"It's a photo op and nothing more and I think his words are both anti-Israel, anti-Jewish, speak for themselves," she told AAP.

Another community member, who did not want to be named, criticised the government's decision to vote for a United Nations resolution to call on Israel to end its presence in Occupied Palestinian Territory.

Albanese spoke to Jewish community leaders at the front of the synagogue.

"This arson attack is an act of terrorism that was fuelled by anti-Semitism and it was stoked by hatred," Albanese said.

He urged unity within the wider community.

"People have come to Australia because we are a country that is peaceful," he said.

"We are a country that respects people of different faith and are enriched by our diversity here."

He committed his government to provide support to rebuild.

His visit comes a day after Opposition Leader Peter Dutton visited the site and Israel's ambassador to Australia Amir Maimon went to the synagogue on Tuesday morning.

Three people are on the run and terrorism investigators from Victoria Police, Australian Federal Police and spy agency ASIO are now in charge of the probe.

Albanese met with Australia's special envoy to combat anti-Semitism Jillian Segal, who said the arson attack was the final straw in a terrible year for the Jewish community.

Jews are being told they are not safe in Australia and to "exercise extreme caution" if coming to the country after Friday's arson.

The Simon Wiesenthal Center, a Jewish human rights organisation with consultative status at the United Nations, issued the warning on Tuesday morning Australian time.

The travel advisory is "a result of the failure of Australian authorities to stand up against persistent demonisation, harassment and violence against Jews and Jewish institutions in Australia".

Rabbi Abraham Cooper sent a letter to Australia's US ambassador Kevin Rudd to tell him of its decision and outrage Jewish Australians were being targeted by hate crimes.

"We are not convinced that Jews are safe as the authorities have failed to take necessary measures to protect Jewish communities from increasingly belligerent and violent targeting by lslamists and other extremists," he said.

He noted the synagogue attack came within days of Australia voting for a UN resolution against Israel earlier in December and said it "only emboldens those who seek Israel's demise".

Israel itself has a level two threat on Australia advising travellers to increase precautionary measures since the fire.

Premier Jacinta Allan hit out at the threat warning, saying "Melbourne, and Victoria continues to be a good, strong, safe place to live and to visit".

In Queensland, a man is facing charges for allegedly making verbal threats in a social media post to a Victorian Jewish community centre on Saturday.

Initial investigations indicate he was not involved with the synagogue arson.