You can assemble the soaked lady fingers and pour the prepared the custard over them and allow the mixture to set in the fridge overnight, the day before you intend to serve the Trifle. Before you serve it, add the whipped cream and berries and remaining toppings.
I’ve also seen some Trifles similar to the one described above, where sliced bananas are added in. If that sounds good to you, try it out!