Getting Started with Deep Learning for Radar Systems

Hello. Thank you for attending this webinar on Getting Started with Deep Learning for Radar Systems. My name is Babak Memarzadeh. I'm a product manager at MathWorks, and I joined MathWorks about a year ago.

Before that, I was working for a ground penetrating radar company for about seven years where I gained a lot of experience working with radar systems. And prior to that, I was doing research on metal materials and metal surfaces for light manipulation as part of my PhD program at Northeastern University.

I will start today's presentation with some introduction to show you the workflows that you can use for using deep learning for radar systems. And then I will explain some of the challenges that you might face during this workflow. In the second chapter, we'll see some classification examples that we can use deep learning for, and in the third chapter, we'll see some regression examples that we can use deep learning for.

Now through these examples, I will show you the tools and capabilities that you can take advantage of to overcome the challenges that we discuss in the introduction. And then at the end, we will summarize the presentation.

Radar systems have many different applications. We can see multifunction radar systems that perform different functions. Searching and tracking, for example. We can also find radar systems in other industries-- automotive industry is an example of it-- where we use the radar systems for applications such as adaptive cruise control, or more generally, for sensing modality in the autonomous driving.

Another application that comes to mind is synthetic aperture radar for generating images. So the advancement in the technology that has enabled all these applications have also generated some challenges. One of these challenges is the system design complexity that is increasing rapidly. As a result, the requirements of these complex systems cannot be satisfied with traditional methods.

That's why we need newer methods-- for example, artificial intelligence to provide more flexibility to the system architecture and to also improve the performance of systems or maybe generate better results out of these systems.

Machine learning can be categorized into two sections, unsupervised learning and supervised learning. Unsupervised learning work with unlabeled data, and the output is usually a cluster or a group of data that have the same properties and characteristics. On the other hand, supervised learning work with labeled data. You provide an input to the network that you believe has an impact on the response. You also provide a response to train the network. And then after you are done with training, you want to provide an input and expect a response from the network.

Now based on the nature of the response, if the response is discreet in nature, you are expecting a classification output from your network. And if the output from your-- if the response is continuous in nature, you're expecting a regression from your network. As you can see in this chart, deep learning is a subset of machine learning. While machine learning works on extracted features that you provide to the network, in machine learning, the features are automatically extracted from the input data.

Now moving forward, I will show you two examples that are using deep learning for classification. Those are classifying hand gesture movement and also classifying synthetic aperture radar target. And then another two examples that we will be using deep learning for regression to synthesize a beam pattern and also to apply digital pre-distortion on the input of the-- on the input signal of the power amplifier.

The workflow for using deep learning for radar systems start with creating or accessing prerecorded data. Now you want to make sure-- if you are synthesizing data, you want to make sure you are modeling and simulating with good fidelity that the results are a good representative of real-life situations.

On the other hand, you might be dealing with large amounts of data, and that can be a challenge to import the data and also label the imported or synthesized data. The next step is to apply pre-processing and transform the data to prepare it for training the network. Here is where you can take advantage of tools that can automate the pre-processing and transformation of the data to prepare it for network training.

Next step is to develop predictive models. You might want to model a network from scratch, or you want to import your reference models. There are tools that I will show you how you can easily do either modeling from scratch or importing your reference models. Here is where you want to also train the network. And having tools to accelerate the training and also optimize the parameters for the training is going to be very useful.

And then after you are satisfied with the performance of your trained network, you might want to deploy this network to targeted hardware. For example, a desktop app or an embedded device or maybe an enterprise system. Where you might find it challenging to generate the required code, either it is HDL code or CUDA code or C++ code. So there are tools within MATLAB and Simulink that you can take advantage of for other generating these types of codes from your network.

Now let's go through the examples and see what are the tools and capabilities that you can use to overcome the challenges that we discussed. We will start with the first example with the classifying hand gesture movement with radar signals. This is a publicly available data set that has been recorded by three different ultra wideband impulse radars.

These radars are located around the hand as shown in the graph here. And within the data, there are 12 different hand gestures that have been captured by these three sensors. And eight different volunteers have helped to collect their hand gesture movements with these three sensors.

So you can already see, there is a lot of data that you need to not only label, but also pre-process and prepare for training to be able to train your network and ultimately expect a classification for the input of the network. Let's go to MATLAB so I can show you how you can take advantage of the tools to deal with this large data set.

OK. Now let's go to MATLAB script where I can show you some of the tools that you can use to manage and also work with large recorded data sets that contains signal data. I explain the publicly available data set for the hand gesture movements. After obtaining the data, we will explore the data with Signal Data Store, which is a tool that enable you to manage large data sets that contain signals.

You can preview your data and see the size of the data that you have from different sensors in your radar queue, and you can also select a specific radar for a specific hand gestures movement and look at its sample signal to understand what pre-processing you need to apply on this data to make the data more suitable for training the network.

In this example, you can think of pulse canceller to remove the clutter from the signal, which is the reflections from the environment and objects around the radar. And as you can see, you're verifying here that by applying the pulse canceller, the signal is more representative of some sort of movement here, which would be more beneficial for training the network.

Then we want to prepare the data for training, and there are other tools available to you for that. For example, you can use the array data store to generate the labels for different files and then combine that array data store with the signal data store to have a label for each signal data that you have from all different sensors.

Another tool that is useful is transform, which defines the preprocessing that will be applied on all the data within your data set. And then read all is the function that you call to import and label and also transform the data. Here if the training data set fits the available memory, you can take advantage of Parallel Computing Toolbox to speed up this process.

There are other tools like Count Labels and Split Labels which lets you analyze the distribution of the data, and also prepare the portions of the data for training, validation, and also testing. In this example, we are using 70% of the data for training, 15% for validation, and 15% for testing.

Then we want to prepare the network for training. So remember, we have three different sensors and we want to get one output out of these three sensors. So for that, we define a network that has three different inputs to three different convolutional neural network branches and combine those into one single output through the main branch as shown in this graph here.

So here, you saw that we use the scripting to generate this network. Right after this example, I will show you an application that you can use, Deep Network Designer app that you can use to generate models from scratch or modify network models that you have. You can also analyze the performance of the network by classifying the testing data and looking at the confusion matrix of the output of the network.

There are also tools that you can use to explore the network predictions further by looking at the score of the output of each CNN branch for each of the movement labels and understand how the prediction-- the final prediction is related to different sensors.

OK, let's go to MATLAB and go to the Apps tab and find a Deep Network Designer. You can also type the name in the command line. You can see that there are different options to start with, from Pretrained Network and Predefined Network to blank to start with a-- to build up a network from scratch. Or you can import from your workspace.

So let's go ahead and import our network from workspace. As you can see here, we can bring in the repeat branch, which was the 17-layer convolutional neural network that we defined for each sensor, or the main branch, or maybe the-- it's better to import the complete network. So here, you can zoom in to different sections, and by selecting each layer, you can look at the parameters of that layer.

So you can see, we have three different inputs or three different CNNs for each sensor. And they all combine into the main branch by the additional layer and a fully connected layer, which has the output size of 12, one for each of the hand movement categories that we have. And finally, the class output classification layer for the output of the network.

And to learn more about this app, its capabilities and examples of different workflows, you can click on this Help button on the top-right to go to the documentation page where you can find the description of the app and some related examples and documentation. OK. The next example that I want to show you is to classify synthetic aperture radar targets using deep learning.

Here, we have another publicly available data set that is published by Air Force Research Laboratory that includes more than 8,600 synthetic aperture radar images from seven ground vehicles and one calibration target that has been captured with an X-band radar in spotlight mode at two different depression angles.

On top you can see the images of these different targets as well as the calibration target with their synthetic aperture radar images. So here is another example where you can see, you are dealing with a large amount of recorded data set. So let's go to MATLAB so I can show you other tools that you can take advantage of to deal with this large data set that contains image data.

OK. And now let's go through the examples script so I can show you the tools that you can use to manage and work with large image data sets. So I explained the data that we are going to use for this example, which is a part of the MSTAR data published by the Air Force Research Laboratory. After obtaining the data, we want to load and analyze the image data.

And for that, we can use the Image Data Store which helps us to work with large data sets. Again, similar to the Signal Data Store. Each individual file should be able to fit in the available memory, but not necessarily the whole data set. And the Image Data Store also enables you to automatically label the data based on, for example, the folder name.

You can use this data store to take a look at some of the sample training images and get an intuitive feeling of how different or how similar different image-- different target images are to each other. There are other tools, like the Count Each Label, which is helpful to analyze the distribution of different targets that we have in the data set. As well as Split Each Label, which is helpful for dividing the data between training, validation, and testing.

So in this case, for each of them, we are defining a data store. OK. Because these images are not necessarily all the same size and they might not match the size of the network's input layer, it is important to be able to automatically resize all these images and maybe perform other transformations. For example, you might want to rotate the images or scale the images and so on.

So with augmented image data store, you can perform these transformations, as well as making sure that the size of all the images in your data set are matching the network's input layer.

After you have all the data available for training, validation, and testing of the network, you define the network here. We define a convolutional neural network. And then you want to define the training options to start training your network. For training options, you have many different options to specify. For example, the solver that you want to use for this training, the maximum number of epochs where epoch is one complete training cycle with all the available training data.

You can also define other parameters, such as validation data and validation frequency. By the way, you want to shuffle the data every often, the training data. And in this example, we are defining the periodicity of that to be every epoch.

Another important property that you want to define for your training option is the execution environment. You can perform this training on GPU or on CPU or on multiple CPUs. If you leave that as auto and there is a compatible GPU, then the code automatically runs the training on GPU. Otherwise, it runs on the CPU or the option that you have defined.

And then you can run the training by trained network and look at the progress of the accuracy of both training data that is shown with the blue line here, and also the validation data that is shown with the black dashed line here on the top chart.

You can also see some other details about this training, including the validation accuracy at the end of the training, as well as the time that it takes and other information like the hardware resource that was used for this training. And of course, after done-- after you are done with your training, you want to evaluate the accuracy of your trained network where you can use the classify function to perform the-- to input the test data to the network and get the predictions of the network and its output.

Here, we can see that the accuracy of the network has been more than 98%. And you can also the performance in the confusion matrix shown here, and see between which images there is more confusion to evaluate if you want to train the network with more data or maybe modify your network to match it better to the data sets that you have.

Now this example is available in our documentation page. And if you are interested to learn more about the network that we are using in this example, there are more information that you can find in the script. So I encourage you to take a look at the script in the documentation page to get yourself more familiar with the tools that you can use to work with large data sets like the MSTAR data set that we showed here.

OK. Now let's see the other two example that use deep learning for regression. The first example is to synthesize a beam pattern with deep learning. We have a circular array antenna, and we want to find the array element weights to have a beam pattern with maximum activity and suppressing the interference 30 dB below the main lobe.

We also want to keep the side lobes below minus 17 dB, between minus 20 and plus 20 degree in azimuth and elevation angles. And we also want to see which method between optimization and also deep learning can provide faster results. Optimization techniques can be used for beam pattern synthesis.

For example, one of the common requirements for beam pattern synthesis is to place a null towards certain directions. Now placing a null in a beam pattern synthesis is a problem that you can solve with closed form formulas. As you can see here, there are three different angles that the beam pattern has generated a null at to avoid the interferences.

But the requirements of the beam pattern might be more demanding. For example, here, we see a beam pattern requirement mask that shows the maximum activity at 0-degree azimuth angle while the side lobes need to be below minus 20 dB, and in a certain region, between minus 10-degree and minus 30-degree azimuth, you want to have the directivity below minus 40 dB.

So you can see that with this new requirement, we are now introducing inequality to the conditions of the problem. That's where optimization techniques, such as quadratic programming or second-order cone programming can become handy to solve these problems with inequality conditions. So now that we know we can use optimization for generating beam pattern synthesis, let's see how we can use deep learning for that.

To use deep learning, first we need to prepare the data for training, validation, and testing. First, we generate lots of patterns with random main lobes and different interference placements. Then we use optimization techniques to generate the array element weights that we'll synthesize the pattern that we have input to the optimization technique.

Then we need the Signal Data Store to manage this large data set. And not only label the data, but also split the data for training, validation, and testing. And then use the training data to train the neural network that we have. Here, we are using a convolutional neural network for solving this regression problem.

And then we want to test the performance of the trained network. So we use the portion of the data that we allocated for testing. And by giving that to the network, we expect a set of array element weights as the output of the network. And then we use those array element weights to generate the beam pattern.

So now let's go see what is the difference between the performance of the optimization techniques and deep learning technique. You can see the results of the optimization technique here where the directivity has been maximized, the interference suppression has been kept very close to the requirement, and the sidelobe levels are meeting the requirements.

Now how about the deep learning network? We can see that while the activity similar to the optimization software has been obtained, and the sidelobe levels are below the requirements, but the interference suppression performance is not good as in the case of the optimization software.

At the same time, we can see that the optimization software will generate this array element weights in more amount of time compared to the deep learning network. Now for improving the performance of your network, you might want to go back and generate more data, and again, train the network with more data. Or modify some of the processing that you apply on the data.

I would also like to mention another trade-off that you are facing here. This trained network is tied to the geometry of the array that we started with. So if you change the geometry of the array, then you need to go back and train the network with the new data sets that you are generating.

And now let's go to the second example that we use the planning for regression, which is digital distortion. Nonlinear behavior of power amplifiers can result in distortion of the signal that can cause challenges in the receiving of these signals. Now when the input of the power amplifier exceeds a certain threshold, we see compression, which is a nonlinear effect, which shows that the output of the power amplifier does not increase with the input in a linear fashion.

The problem gets more complicated when we look at the memory properties of the power amplifier. In other words, the power amplifier output not only depends on the current input of the power amplifier, but also on the previous samples that have been input to the power amplifier.

Traditionally, the nonlinearity of the power amplifier was modeled with memory polynomial models that take into account not only the current samples, but also the previous samples with their amplitude power. You can use deep learning and generate a neural network to apply digital pre-distortion to offset the effects of the power amplifier on the signal.

Now, you want to train the network, and this diagram shows the workflow for training the network. You obtain the measurements from the power amplifier by providing it some OFDM signal after oversampling, and then measuring the output of the power amplifier. Now you want to use these two data sets to train the network.

The input for your network training is the output of the power amplifier, or in other words, the distorted signal. And the desired output from your network that you use for the-- that you use for training the network is the input of the power amplifier. The signal before it gets distorted by the power amplifier.

And then after training the network, you want to measure the performance of the network and see how the network is behaving. So you want to deploy this network. You generate the OFDM signal again, and after oversampling it, now you provide that as an input to the trained network. And the output of the network has been pre-distorted to go to the power amplifier and compensate for the nonlinear effects of the power amplifier. So at the output shown by z here, you get a linear signal.

For the network, we are using an augmented real value time delay network that consists of several fully connected layers. To train the network, the inputs are the current and delayed complex baseband IQ samples. And also, the current and delayed baseband IQ sample amplitude up to k-th power.

Now while the current and delayed samples are representing the memory behavior of the power amplifier, the amplitude powers are representing the nonlinear compression behavior of the power amplifier. OK. After testing the network, you want to be able to quantify the performance of the network. And here, you can see three different measurements that you can perform to obtain the adjacent channel power ratio or the normalized mean square error by comparing the output of the power amplifier and the input of the neural network digital distortion. And you can also measure the RMS error vector magnitude.

And looking at the metrics without digital pre-distortion and with memory polynomial DPT and also neural network DPT, you can see that the neural network digital pre-distortion outperforms the other two. And we can see that in the spectrum analyzer here, that the output of the neural network digital pre-distortion has the least amount of adjacent channel power ratio shown by the red trace here.

OK. And now let's summarize the presentation. We looked at the workflow that you can use for applying deep learning for your radar system design. Through two sets of examples for classification problems and also regression problems, I showed you different tools that you can overcome the challenges that you might face during this workflow, including the tools that you might find useful for working with large data sets, labeling the data sets, and also transforming these data sets, either if they are signal data or image data.

I also showed you the Deep Network Designer app that you can use to design a network from scratch or modify your reference models. We also saw the tools that you can take advantage of for training your network and accelerate the training process, and also optimize the parameters of the training.

And there are other resources for you to learn more about deep learning for radars. In the page that I'm showing you here, there are examples webinars, white papers, and more resources that you can find. And if you are interested to get started with deep learning, in MATLAB, there is a great on-ramp which is a self-paced training module that you can take advantage of.

With that, I would like to thank you a lot for taking the time and attending this webinar. I hope you find it useful. Thank you.