Combine milk, white sugar, salt, orange zest, lemon zest, cinnamon, cloves, and vanilla in a saucepan, whisk, and bring to a simmer over medium heat. Reduce heat to medium-low and simmer gently for 5 minutes.

Strain milk through a mesh strainer and transfer 2 cups into a shallow dish in which to soak stale bread; set aside.

Transfer remaining 2 cups infused milk into a saucepan to make the custard sauce; add yolks and whisk thoroughly. Return saucepan to medium-low heat, and cook, stirring with a spatula and scraping the bottom constantly, until the mixture gets hot and thickens enough to coat the back of a spoon. Remove from heat; immediately pour through a fine mesh strainer. Let cool and refrigerate until needed.

Soak stale bread in milk, turning a few times, until most of milk is absorbed and bread is saturated. Remove to a pan or plate and refrigerate until needed.

Beat whole eggs in a wide shallow bowl. Coat bread on both sides in the egg.

Pour olive oil, 1/4 to 1/2 inch deep, into a nonstick skillet, and heat over medium heat. Fry soaked bread in the hot oil until browned and puffed, about 4 minutes per side. When fully cooked, the bread will spring back when touched in the center.