Latency problems!

good evening everyone !

I am working on a new setup for live performance using tidal and routing audio and midi to ableton.
I'm also using a prophet 6 external modul and controlling it by midi from tidal .
my connections are these at the moment :
prophet is connected by the midi port on my focusrite
tidal is routing audio to ableton using loopback
I set up ableton with a sample rate of 48000 and a buffer size of 64 sample.
the device are in sync but sometimes (randomly) the prophet starting to have latency problems .

At the moment the latecy on boottidal.hs is set to 0.05 (default ) and i set the latency of the mididevice to 0

I really don't understand where is the problem and how I can solve it

if someone have some idea I'll be very happy to hear some suggestion.

PS : I have a new maxbook pro with M2 and 32 giga of RAM

have a good evening