The Cessna, registered in Vienna, had been due to land in Cologne, Germany, but instead headed out into the Baltic.
German businessman Peter Griesemann died - it was his private plane. German media say the other three victims were his wife, daughter and her boyfriend.
Griesemann was prominent among organisers of the Cologne Carnival.
The Carnival's Facebook page has a mourning tribute to him. He was president of Blaue Funken (blue sparks) - one of the oldest carnival associations.
When the Cessna 551 failed to respond to controllers' calls German and Danish fighter jets were sent to follow it, and later a Swedish helicopter searched for it.
Latvia now has three small ships and a helicopter in the area of Sunday's crash.
"We found three parts of the plane, we think - experts will have to say," the Latvian maritime search and rescue service (MRCC) told the BBC.
Spokesperson Liva Veita said there was no sign of those on board the Cessna. She also confirmed earlier reports that oil had been seen there.
She said two small Latvian naval ships and a coastguard vessel were at the crash site, north-west of Ventspils, as well as a Latvian border guards helicopter.
A Latvian minesweeper could send down a robot submersible to locate the plane, she added.
Quick Air, an air charter company based in Cologne, told Reuters news agency that the Cessna was Peter Griesemann's private plane. It confirmed his death and said three others were on board too, but did not identify them.
The Griesemann family, based in Cologne, own a holiday cottage in the Cádiz area of south-west Spain, German media report.
On Sunday a Danish ferry and two helicopters - from Lithuania and Sweden - conducted a search in vain in the Baltic.
Earlier, Nato pilots and Swedish officials tracking the plane saw no-one in the cockpit.
"The aircraft was flying from Spain to Cologne, but during the flight the aircraft changed its flight route," the Latvian Civil Aviation Authority said in a statement.
"Air traffic controllers were unable to communicate with the aircraft's crew," it said.
The FlightRadar24 data tracking website says the plane took off from the Spanish city of Jerez de la Frontera at 12:56 GMT.
At 17:37 it was listed on FlightRadar24 as losing speed and altitude.
German newspaper Bild says the plane reported cabin pressure problems after take-off and contact was lost after it cleared the Iberian peninsula.
The aircraft crashed "when it ran out of fuel", Sweden's search and rescue operation leader, Lars Antonsson, told AFP news agency.
Mr Antonsson said rescuers had "no explanation at all" and could "only speculate" about what had happened. "But they [the people on board] were clearly incapacitated," he said.