How to speed up UDP throughput

I'm using fread of the UDP object to read out the whole UDP buffer every 50ms.

When first starting my code, it keeps up, but as time goes on, it starts to drift behind. I still am able to clear the UDP object buffer with no problem or limit on execution time. However, it lags so far behind, that if I stop sending UDP packets(from a seperate computer), my code continues to receive data for a significant amount of time after. The data recieved after the packets stop are all sequential and I don't loose any. So it appears the limitation is getting the data from the windows buffer to the UDP buffer. How can I speed that up?

I have verified on wireshark that the packets do indeed stop immediately.

to clear the buffer:

bytesAvailable = UDP.BytesAvailable;

data = fread(UCP, bytesAvailable);

My UDP object has DatagramTerminateMode = 'off' so I can read the whole buffer out.