Image Segmentation and Analysis | Making Vehicles and Robots See: Getting Started with Perception for Students
From the series: Making Vehicles and Robots See: Getting Started with Perception for Students
This video introduces the concept of performing image segmentation and analyzing the regions of interest using interactive apps. Starting with the most used segmentation technique—color-based segmentation—discover how you can use the Color Thresholder app to segment regions specific to colors in an image. Then use the results from the Color Thresholder app to refine the mask by using the Image Segmenter app. You can analyze the image regions using the Image Region Analyzer app. Also, learn how to export the masks from the apps. Follow the exercise at the end of this video to use the exported masks from the MATLAB® apps in Simulink®.
Published: 27 Jul 2021
We are well on our way to equip our vehicles and robots to help it move autonomously. Hello, everyone. In today's video, you will see how you can mock image segments that make it easy for us to work with our objects of interest.
We will see how you can segment an image based on colors. We find these detections and analyze the resulting image regions, all using interactive apps. Let's take a scenario where you want to maneuver your vehicle to mock cones autonomously.
For this, you need to differentiate between these distinct cones and the rest of the image. This process is called image segmentation, which helps you understand the location of segments of interest in the entire environment. For image segmentation, you want to find a method that can help you get something like this.
This is called a mask, and it's a matrix of zeros and ones. The white area is indicative of the area of interest. And multiplying it with the original image, you get the areas that are of interest to you.
There are various methods to perform segmentation, threshold based, color based, geometry based. But we'll begin with one of the most used techniques in segmentation, which is color-based segmentation. Before we go into color thresholding, we will need to pre-process that image. You need just a view of the road, and not the car and the rest of the environment.
So let us crop the images using the imcrop function. You will crop the image interactively by selecting the image part that we need. And the resulting cropped image will be present in this variable, along with size and position of the cropped rectangle.
The first task is to use color-based thresholding to distinguish the red color from the rest of the background. To do this, you can try an interactive method using the Color Thresholder app. You will find this in the MATLAB tool strips, under Apps, under Image Processing and Computer Vision, and you can click on Color Thresholder.
First thing's first. We will lower the image of interest. For us, it is the cropped image, which we will load from the workspace. We will use the cropped image.
You can select the color space which you want to work at, which are represented by point clouds here. We will just go with the RGB color space. You will then get these interactive sliders, so you can move these around to find the exact color that you need to detect these cones. You can view the mask superimposed on the image, or you can just view the binary.
And now if you need to use it later for a different task, this image can be exported as an image or as a MATLAB function. We will export this image. We will export the Binary Mask, the Masked Binary Image. But we don't need the Input RGB image because it's already on our workspace. Say OK, and we have the variables, which have been created from the app.
One thing to keep in mind here, is that color-based thresholding is sensitive to changes in environment and the lighting. For example, a color in bright sunlight would look different from the one in a dim light. A solution to this is to take multiple sample photos from the environment, apply the thresholds and test the results. You can find the threshold values that suits the test images the best, and use it in the rest of your algorithm.
Great, let's see the mask. But it does not look like the correct representation of what was intended, right? There are black pixels in areas that should have been completely white, probably due to light reflected from the object. This is why an image needs to be refined.
You can do it interactively using the Image Segmenter app. The app is again located under Apps. Under Image Processing and Computer Vision, you can click on Image Segmenter.
Let's first load the image from Workspace. Our image is called Cropped Image. We will then load the mask. You can create a new mask, too. But since we want to segment based on colors, we will use the Color Thresholder mask that we had exported just a while back.
There are various methods, but what we will use today, is the Local Graph Cut method. You can select the area of interest, click on Mark Foreground, scribble in the area where you think it's the foreground and just click on Apply. You can do the same thing to mark the background, as well.
This may work well for some objects which have a very prominent foreground object, but may not work well for some others. We can then proceed to refining the mask. We can say Fill Holes, which will fill this hole due to the reflected light. Or we can go to Morphological Operations. Some of the morphological operations are Dilate, Erode, Open and Close.
You can select the one depending on whichever is the best performing algorithm for you and just click on Apply. We can see the mask using Show Binary, which we can eventually export as an image or as a MATLAB function. We'll just export it as an image. After getting the satisfied masked image, you can find the position of the connected components of the segmented cones in the masked image and annotate the original image to mock the segmented objects.
Segmenting an image is an art. You need to try different methods to see what works best for you and your data. Once you have the mask, you want to probably reject regions with a small area and want only the cones at the very front of the car. You can do this interactively using the Image Region Analyzer app, which is located under the MATLAB tool strip, Apps and then Image Processing and Computer Vision.
Here you must select the mask that we're interested in, which is the mask we exported from the Image Segmenter app. Select the filters that you need. Let's say, we want to reject any area that is lesser than 75 pixels. And now you see that this area, which was wrongly detected as a cone, has been rejected.
For all of you who are using Simulink, you can use the Export Function feature from the Apps. Select the part of the code that you need, and plug it into the MATLAB Function plug in your Simulink model. As an exercise, we will count the number of circular yellow objects in this image using Simulink. We will first read the image, detect the yellow colored objects in the image using the Color Thresholder app.
We will then refine the mask and filter the result by eccentricity to open the number of circular objects. To sum up, today we saw how we can perform color-based segmentation using the Color Thresholder app. We then refined the mask using the Image Segmenter app.
We then, eventually, saw how we can analyze regions in the image using the Region Analyzer app. The graphic interfaces of the apps helps reduce difficulties and the time that you spend on trying different image segmentation methods tuning those parameters and testing your performance interactively. And you can also export these results to use them in other various tasks.
If you have any questions while you're working through this video or through the exercises, please feel free to reach out to us at roboticsarena@mathworks.com Thank you, and see you in the next video.