Cucumbers, lemons and oranges are among the 30 varieties banned from being wrapped in plastic.
Larger packs as well as chopped or processed fruit will be exempt.
President Emmanuel Macron called the ban "a real revolution" and said it showed the country's commitment to phase out single use plastics by 2040.
More than a third of fruit and vegetable products in France are thought to be sold in plastic wrapping, and government officials believe that the ban could prevent a billion items of single use plastics being used every year.
In a statement announcing the new law, the Environment Ministry said that France uses an "outrageous amount" of single use plastics and that the new ban "aims at cutting back the use of throwaway plastic and boost its substitution by other materials or reusable and recyclable packaging".
The ban forms part of a multi-year programme introduced by Mr Macron's government that will see plastics slowly eased out in many industries.
From 2021, the country banned plastic straws, cups and cutlery, as well as polystyrene takeaway boxes.
And later in 2022 public spaces will be forced to provide water fountains to reduce the use of plastic bottles, publications will have to be shipped without plastic wrapping, and fast-food restaurants will no longer be able offer free plastic toys.
However, industry figures have expressed concerns over the speed at which the new ban is being introduced.
Philippe Binard, from the European Fresh Produce Association, said the "removal of plastic packaging from most fruit and vegetables at such short notice does not allow alternatives to be tested and introduced in a timely manner and stocks of existing packaging to be cleared".
Several other European countries have announced similar bans in recent months as they pursue commitments made at the recent COP26 conference in Glasgow.
Earlier this month Spain announced that it will introduce a ban on the sale of fruit and vegetables in plastic packaging from 2023 to allow business to find alternative solutions.
Mr Macron's government also announced several other new environmental regulations, including rules calling on car adverts to promote greener alternatives such as walking and cycling.