Francis, 84, will spend only seven hours in the Hungarian capital Budapest to close an international Roman Catholic meeting before moving on to Slovakia, where he will stay much longer, visiting four cities before leaving on Wednesday.
The Vatican's schedule says Francis is due to meet Orban. But the extreme brevity of his stay in Budapest has prompted diplomats and Catholic media to suggest that the pope, making his first trip since surgery in July, is giving priority to Slovakia and in effect snubbing Hungary.
Francis has often denounced what he sees as a resurgence of nationalist and populist movements, called for European unity, and criticised countries that try to solve the migration crisis with unilateral or isolationist actions.
By contrast, Orban told the Bled Strategic Forum in Slovenia last week that the only solution to migration was for the European Union to "give all rights back to the nation state".
The pope has called for migrants to be welcomed and integrated to tackle what he has called Europe's "demographic winter". Orban said in Slovenia that today's migrants "are all Muslims" and that only "the traditional Christian family policy can help us out of that demographic crisis."
"The pope's decision to spend more time in Slovakia than Hungary must also be read in the context of his fierce criticism of nationalist-populist leaders such as Hungary's prime minister," the British Catholic newspaper The Tablet said.
Speaking to reporters on the plane returning from Iraq in March, Francis said his stop in Budapest was "not a visit to a country but for a Mass." About 75,000 people are expected.
The U.S. conservative National Catholic Register newspaper reported that Hungarian officials, including some bishops, had failed to convince the pope to stay longer in Hungary and perhaps make a full state visit.
The Register said some regarded the decision as "a gigantic slap in the face" of Orban.
Francis is also due to meet Hungarian President Janos Ader before saying the Mass to conclude a Church congress that began last Sunday.
Asked by Spanish Radio network COPE last week what he would say to Orban about closing borders, Francis said: "When I am in front of a person, I look him in the eyes and let things come out".
The pope is leaving Rome unusually early on Sunday - at 6 a.m. - so that he can say the Mass in Budapest and reach Slovakia in the afternoon, without spending the night in Hungary.
When a reporter asked at a briefing on Thursday why the pope appeared to be "running away from Hungary", Vatican spokesman Matteo Bruni said the stop there was a "spiritual pilgrimage" that should be seen in a religious context.
Orban’s office said in an email to Reuters on Thursday that the pope’s stop for a religious event was "an enormous honour" and that comparisons with the Slovakia leg would be "misleading".
The trip will be the first test of the pope's strength since surgery in which 33 cm (13 inches) of his intestine were removed because of a narrowing of the colon.
In Slovakia he will meet the president, prime minister and parliamentarians. Slovakia is against uncontrolled immigration but its leaders have been much less strident in their opposition than Hungarian leaders.
The pope will also meet Slovakia's Jewish community. About 105,000 Slovak Jews were killed in the Holocaust and today the community numbers around 3,000.
In Kosice, he will meet with the Roma population in the dilapidated Lunik IX district, one of the largest concentrations in the country. Slovakia has a Roma population about 440,000, many in settlements on outskirts of towns.
Francis will say two open-air Masses in Slovakia - which is about 65% Catholic - including one service in the lengthy Byzantine rite.