Real-Time Audio Processing: Varying Delay Depending on Initial Under/Overruns
18 views (last 30 days)
Show older comments
JFr on 6 Sep 2017
Commented: Jimmy Lapierre on 10 Jun 2022
i am using the Audio System Toolbox to do real-time processing. When running my program multiple times I experienced that the delay between input and output was not always the same though i did not change the code between the runs. The only difference i could observe was a different amount of underruns when starting the program. After a few frames the system was then in a 'stable loop' and running without any underruns - however, with different delays.
A simple test set-up: Put a microphone to the soundcard (Fireface UC in my case). Use the audioPlayerRecorder from the Audio System Toolbox. Within the processing-loop map the input vector to the output vector (so basically just a talkthrough). Force some underruns within the first frames (i observed this often before the system then runs smoothly). When system is stable, check the delay between input and output. Repeat - but with different amount of underruns.
Observation: With different amounts of initial underruns, the delay when the system is stable is different.
Expected Behaviour: To my understanding i would expect the following framing steps: 1. Frame A is collected at the soundcard. 2. When done, frame B is collected at the soundcard and frame A is processed within Matlab. 3. After that, frame C is collected, frame B is processed and frame A is read to the output of the soundcard. 4. Frame D is collected, frame C is processed, frame B is read to the output, and so on...
This should result in a delay of 2 frames. When overruns occur, i would expect input frames to be dropped. When underruns occur, the processing was too slow and probably the old output buffer content is read out again - the processing should be interrupted and start processing the following frame. However, in both cases the framing should remain the same.
Summary/Question: Since the processing after the initial under/overruns is stable, the complexity of the processing is not too high and must therefore be processable within one frame. However, the initial under/overruns seem to influence the resulting framing and can cause output frames to be (constantly) read out later than expected - which in turn yields a (constant) higher delay.
What is happening here? Is my understanding of the framing wrong (or too much influenced from Digital Signal Processors?)? Is this behaviour a bug or in the end even desired?
Best regards and thank you very much for helping out!
plewanig on 28 May 2019
Hello. this is not an answer. I just want to add here that I experience the exact same problem. I get dropouts in the beginning of an audio loop from time to time (underruns), which cause a permanent change in the round trip latency. In my case I use a buffer size of 2048 samples, an asio driver with the "dante virtual soundcard" and there is no processing involved. I play back and record (multi channel) in order to measure transfer functions. So the conditions should be optimal in order to have no dropouts. Are there any new regarding this subject? Is there any documentation about how to properly "setup()" an audio system object, especially the audioPlayerRecoder? wbr
Zhao Wang on 8 Sep 2017
As a first step, you can run the Measure Audio Latency example to verify that your audio I/O system is configured correctly: https://www.mathworks.com/help/audio/examples/measure-audio-latency.html
For profiling your code: https://www.mathworks.com/help/audio/examples/measure-performance-of-streaming-real-time-audio-algorithms.html
Output signal silence (underrun) occurs when the device buffer is empty and it is time for digital-to-analog conversion. This results when the processing loop in MATLAB does not supply samples at the rate the sound card demands. The number of samples underrun is returned when you call your audioPlayerRecorder or audioDeviceWriter object.
Input signal drops (overrun) occur when the processing stage does not keep pace with the acquisition of samples. The number of samples overrun is returned when you call your audioPlayerRecorder or audioDeviceReader object.
Generally, since you encounter overrun or underrun, I recommend you to try improving your I/O system:
1. Identify when the overrun or underrun occurs. If it occurs in the first few iterations, consider calling setup on your System objects before the loop where real-time processing is required. You can also run the I/O system with dummy data for a few frames before starting the real processing. For more information, see Measure Performance of Streaming Real-Time Audio Algorithms.
2. If you are using a DirectSound driver on a Windows platform, consider switching to a WASAPI or ASIO driver. ASIO drivers have the least overhead. If you are using an ASIO driver, make sure to match the frame size in MATLAB to the ASIO buffer size.
3. If you can afford to add more latency to your application, consider increasing the buffer size of your object. By default, the buffer size is the frame size of the data processed by the audio object.
4. If you can afford to decrease signal resolution, consider decreasing the sample rate.
5. Close all nonessential processes on your machine, such as mail checkers and file sync utilities. These processes can asynchronously ask for CPU time through interrupts and disturb the audio processing loop.
6. To maximize performance, remove all plotting and visualization from your real-time loop. If you require a visualization to update in your processing loop, use a DSP System ToolboxT scope such as dsp.TimeScope, , dsp.SpectrumAnalyzer, , or dsp.ArrayPlot, . Follow the recommendations listed in point 1 to setup and pre-run your scopes. If you require custom graphics or are processing callbacks in the loop, use the drawnow command and specify a limited update rate to optimize your event queue.
7. If the processing loop is algorithm-heavy, try profiling your loop to locate the bottlenecks, and then apply appropriate measures: o Replace handwritten code with MATLAB features that have been optimized for speed. o Follow best practices for performance: Techniques to Improve Performance. o Generating MATLAB executables (MEX files) using MATLAB CoderT may result in faster execution. See Desktop Real-Time Audio Acceleration with MATLAB Coder for an example. You can also generate standalone executables (EXE files). See Generate Standalone Executable for Parametric Audio Equalizer for an example. o If you are considering turning your algorithm into a VST plugin, then try running it as a VST plugin within MATLAB. VST plugin generation uses C code generation technology under the hood, and running the generated VST plugin within MATLAB may result in faster execution than with your original MATLAB code. See Design an Audio Plugin and Host External Audio Plugins to learn how to design, generate, and then host a VST plugin.
About Latency, there are output latency and input latency. If properties and frame size remain consistent, the ratio of input latency (between when audio enters the sound card to when the frame is output by the processing stage) to output latency (between the generation of an audio frame in MATLAB to the time that audio is heard through the speaker) is consistent between calls to an audioPlayerRecorder object. To minimize latency, you can:
1. Optimize the processing stage. If your processing stage has reached a peak algorithmically, compiling your MATLAB code into C code using MATLAB Coder may result in faster execution.
2. Decrease the sample rate.
3. Decrease the frame size.
James Rex on 6 May 2022
I have experienced the same issue using the audioPlayerRecorder object in MATLAB R2017a, and in my latest version MATLAB R2020b (both under Windows). Observed behaviour is that live audio round-trip latency, with loopback from audio device output to input, is not constant, but tends to increase whenever audioPlayerRecorder detects multiple under/overruns within a short time. I believe this is caused by MATLAB software, as round-trip latency does NOT vary, even after under/overruns, when using the same audio interface hardware (Focusrite Scarlett 8i6) with Max 8 software performing the same function (simultaneous live audio input + output).
I believe this is an undocumented feature of the MATLAB audio interface driver software, and I would like MathWorks staff to confirm this, and tell me how I can achieve constant low-latency live audio processing in MATLAB, even after occasional under/overruns (heard as audio dropouts). Ideally, the MATLAB audio interface driver software would be updated to allow users to choose either (a) current behaviour, where latency may increase automatically, or (b) guaranteed constant latency, even after under/overruns.
Note that I have already applied all the solutions suggested by Zhao Wang above in this thread, where they are possible in my application. I can achieve the low latency I need with MATLAB for a short time after starting audio processing, but after a few tens or hundreds of milliseconds latency usually increases significantly. Thereafter, latency remains constant, but is now too high for my application.
Hoping for an answer and a fix :-)
Jimmy Lapierre on 25 May 2022
Some updates since these issues were raised:
- Starting 21b, Audio Toolbox I/O has a new buffering mechanism to prevent the latency from increasing when there is a frame underrun.
- The Impulse Response Measurer application is another alternative for measuring loopback latency.
Jimmy Lapierre on 10 Jun 2022
Thank you very much for checking it out, this is exactly the pain point we wanted to eliminate after getting customer feedback.
Find more on Audio Processing Algorithm Design in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!Start Hunting!