With a shared external frequency reference, in streaming mode only, there should be zero phase drift. However, every time you change center frequency, there will be a new phase offset.
If there is phase drift from floating point frequency correction rounding errors at some frequencies, I would think it would be very small and identical across devices and therefore cancel, but I have not tested this (maybe 1 degree per minute?).
At even multiples of 20 MHz (e.g. 200.00 MHz), this correction should be zero, so you should get a true zero phase drift. If you test this, let me know. If there is phase drift, there should be a way to easily get rid of it, possibly by using streaming IF instead of I/Q…