To make the custard, heat the milk and cream in a heavy-bottomed saucepan. Meanwhile, whisk the egg yolks and sugar together in a large bowl until pale and thick. When the milk and cream are just below boiling point, remove from the heat and slowly pour into the yolk mixture, whisking constantly. Return the custard to the pan and place over a low heat, stirring, until thickened.
To make the pudding, pour the custard into a baking tin and add the brioche slices. Push the slices down with your fingers so the custard soaks in. Cover and set aside.
To make the plums, preheat the oven to 200C/180C Fan/Gas 6. On a baking tray lined with baking paper, lay out the plums, cut-side up, and sprinkle each one with the sugar. Bake for 15 minutes, or until the sugar caramelises and the fruit softens. Take the plums out but leave the oven on.
Meanwhile, to make the caramel, place the sugar in a saucepan with a tablespoon of water. Cook over a medium heat, gently swirling the pan, until the sugar dissolves to a light brown caramel colour. Turn off the heat and stir in the butter and cream to create a smooth sauce. Set aside.
Return to making the pudding. Heat the oil in a frying pan. Take each slice of brioche out of the custard, allowing the excess to drip off. Fry each slice on both sides until golden brown, then transfer to a baking tray. Sprinkle sugar over each slice and bake for 10–12 minutes. The slices should be caramelised on the outside but remain soft in the middle.
Serve the brioche slices with the roasted plums, clotted cream and a drizzle of caramel sauce on top.