Simulink code generation for a deep reinforcement learning agent's policy network for a Hardware-in-Loop application using a dSpace MicroAutoBox2 with an rti1401 hardware

Dear Community!

I have Simulink code generation related issues in MATLAB 2023a Update 5 regarding a project involving integrating a deep reinforcement learning agent's policy network for a Hardware-in-Loop application using a dSpace MicroAutoBox2 with an rti1401 hardware. I followed the related documentation and avaible materials on the internet until the point where I have a Simulink model with a dSpace rti1401 as target for code generation, a trained SAC agent, and an 'evaluatePolicy.m' function with the 'agentData.mat' which were generated by the 'generatePolicyFunction' function.

As the first approach I tried to use a 'policy' block to load the agentData to simulate the policy, then integrated it appropriately into my Simulink model. During building however, I received an error that for deep learning libraries, little endian settings for prodEndianess and targetEndianess must be set, but after changing these parameters accordingly, another error occured that the dSpace hardware's processor unit can only process big endian coded data, so I must change the settings back. The two exact error messages are the following:

"Error:Code generation of deep learning models only supports 'LittleEndian' option for properties 'ProdEndianess' and 'TargetEndianess'.

Function 'Policy/PolicyWrapper' (#341.361.397), line 11, column 12:

"coder.loadRLPolicy("agentData.mat")""

and

"Error:You have specified options for Embedded hardware on the Hardware Implementation page of the Configuration Parameters dialog which are not suitable for RTI1401:

Option: "Byte ordering"

Current value: LittleEndian

Requested value: BigEndian"

In addition, I tried to use both the 'predict' block, but I received the same errors as above.

As a workaround I also experimented with the method in which case I use the evaluatePolicy function to generate general plain C code (so without third-party compilers), then integrate it into Simulink with a C Caller block or an S Function block to simulate the C code, based on the following: https://www.mathworks.com/matlabcentral/answers/757154-how-to-implement-reinforcement-learning-using-code-generation. After successfully generating the C code and building the S function the whole MATLAB crasher upon starting a simulation or building the model.

The building of the Simulink model was tested without the deep learning agents and the model builds properly.

[Nathan Malimban]

Hi Szilard,

Thanks for your detailed explanation. The bottom line is that deep learning code generation only supports little endian today, as you surmised from the error message. However, I'd like to get some info about your use case so we can craft an enhancement request for codegen with different endianness options.

One thing about supporting alternate endianness options is that other features depend on certain byte orderings. I'd like to know whether you'd theoretically want to use these features along with generating big endian code for your use case (this will inform the enhancement request):

1. Would you want to be able to update your deep learning network weights (either online or offline) after generating code?
2. Would you want to compress your deep learning network weights into a more compact format to reduce memory and increase performance? (bfloat16 codegen)

Lastly, do you know if your hardware can be configured to work with little endian? If so, that could be something to try. I know some hardware can be configured in this way but I am not too familiar with rti1401.

Nathan

[Szilard Hunor Toth]

Dear Nathan!

Thank you for your response. I'd looked into the hardware configuration options and it seems that the real-time processor only supports big endian code. As for answering the questions:

1. Yes, training on the HIL system would be important, and I would also like to use the networks without updating the weights.
2. This would definitely be useful for huge network architectures, so I would also say yes.

Also, may I ask when can I expect the enhancement to be finished and released?

Thank you for your support!

Szilard

[Nathan Malimban]

Hi Szilard,

Great, thank you for this info. The details are definitely helpful. I can't predict timelines, but we will definitely take this enhancement under consideration.