Latency for NN deployment information using DL HDL toolbox

Hello,

I'm using Deep Learning HDL Toolbox Support Package For Intel FPGA And SoC Devices to implement a Convolutional Neural Network (CNN) on an Intel FPGA. The CNN consists of convolutional layers, activation layers, and fully connected layers. After compiling the entire model with all its layers, I observed that the latency report generated using the command [prediction, speed] = hW.predict(single(inputImg),'Profile','on') only provides information for convolutional and fully connected layers and there is no latency metric provided for activation layers.

I am interested in understanding how the latency is computed in this context and why activation layers do not have a latency metric in the report. Could you please provide more information on this matter?

--------------------------

Software version:

Matlab R2023b

Ubuntu 20.04

---------------------------

Answer 1

[Bharathi Yogaraj]

When the activation layers are placed after the convolution or fully connected layers, they are fused together to avoid any extra latency. For further details, please refer to the additional information provided: Supported Networks, Layers, Boards, and Tools - MATLAB & Simulink (mathworks.com)

[Wang Chen]

Hi Ruben,

Also, to answer your question on "how the latency is computed": the DL Processor is saving the events like the start and end of the each layer into an event memory, and these event data are pulled from FPGA to MATLAB when you turn 'profile', 'on'. These event data contains the actual clock cycle numbers, so the profiler can calculate out the exact cycle time each layer used in the FPGA.

As Bharathi pointed out, activation layers are merged into the previous layer during execution, so they don't cost extra clock cycles, that's why you don't see them in the profiler report.

Thanks,

Wang