What kept him away was the tough anti-migrant stance of Prime Minister Viktor Orban, in contrast to the 86-year-old pontiff's compassion for all refugees.
What brings him here now, besides his support for Catholics, is the war in Ukraine.
Hungary and Ukraine share a 134km (85-mile) border.
Unlike other EU leaders, Mr Orban has refused to back military aid for Kyiv and maintained relations with Russia's Vladimir Putin.
The Pope arrived in a small white Fiat, in a convoy of black limousines, to a spectacular parade by Hungarian Hussars on horseback outside the presidential palace in the castle district, overlooking the city.
He walked cautiously, with a stick, after a knee injury last year, but his voice was strong.
"We seem to be witnessing the sorry sunset of that choral dream of peace, as the soloists of war now take over," he told assembled dignitaries.
"World peace cannot be ensured except by creative efforts, proportional to the dangers threatening it", he quoted the Schuman Declaration of May 1950 which set out the founding principles of the forerunner to the European Union.
"At the present time, those dangers are many indeed, but I ask myself, thinking not least of war-torn Ukraine, where are creative efforts for peace?"
The Pope criticised what he called the rise of nationalism. And made a plea for more compassion from his hosts towards asylum seekers.
"When we think of Christ present in so many of our brothers and sisters who flee in desperation from conflicts, poverty and climate change, we feel bound to confront the problem without excuses and delay," he said.
Europe should work for "secure and legal corridors" to help refugees to safety, he continued.
Government ministers listened stony-faced.
In 2015, the Hungarian government built a razor-wire fence the whole 170km (105 miles) length of its border with Serbia, and has since added double and triple layers.
Last year, there were 158,000 push backs, or "escorts" as the Hungarian police describe them, of migrants who managed to enter Hungary and were expelled into Serbia.
The prime minister told the Pope, according to his spokesman, that Hungary only had a future if it stayed on the Christian road, and the Christian road was today the road of peace.
There was also praise in the Pope's speech, for the Hungarian government's attempts to defend the traditional family.
"I think of a Europe that is not hostage to its parts, neither falling prey to self-referential forms of populism, nor resorting to a fluid…'supranationalism' that loses sight of the life of its peoples," he said.
As an example, he cited what he called "so-called gender theory" and vaunting as progress a senseless "right to abortion" which is always a tragic defeat.
In the afternoon, the Pope visited St Stephen's Basilica, the main church or cathedral in downtown Budapest, for a meeting with bishops and clergy.
Hungarians and tourists flocked to the square as the many bells of the basilica peeled across the city.
Security was tight, with the anti-terrorist TEK police strongly in evidence, but on their best behaviour to manage the crowds.
The Orban government has given large sums to the main churches of Hungary in its 13 consecutive years in power, but in exchange, Church leaders rarely criticise government policies.
One exception is András Hodász, a former Roman Catholic priest who resigned from the priesthood because of differences of opinion with the church, and the pressure he came under to remain silent.
"The devil hides in the details," Hodász told the BBC, referring to the narrative on the war in Ukraine.
"The Hungarian government is calling for peace at any cost. An immediate ceasefire which could confirm existing front lines. That contrasts with the words of the Holy Father, that Russia should pull back to the old borders. The Pope recognises Ukraine's legitimate right to self-defence."
The visit continues on Saturday with meetings with homeless people and refugees.
They include Abouzar Soltani, an Iranian convert to Christianity, who spent 18 months in detention in a Hungarian "transit zone".
The Pope's visit ends on Sunday, with the celebration of mass in Kossuth square, in front of the Hungarian parliament.
Hundreds of thousands are expected to attend, including Hungarians from neighbouring countries.