How to stabilize dynamic background in a video using Computer Vision Toolbox?

My video consists of a constant FOV with an object in motion in dynamic background. My goal is to make the background static i.e. only the foreground object should be moving with a constant background. Any ideas on how should I proceed with this?

[Florian Morsch]

So to be clear, you have a object in motion and something else also is moving in the background? Like you want to follow a red ball which is shown in front and in the back there are kids playing or something like that?

[Kartik Bharadwaj]

Precisely. Though, instead of kids, it is trees swaying in the background. And I want to make this dynamic movement of trees, static. I don't want to track/follow the ball instead make the background static.

[Florian Morsch]

Sounds nearly impossible.

A idea would be to get a background image without the moving object and then try to find some fix points. After that you could try to allign all following images from the stream to those points, so the background would look nearly stationary. Problem is, that the foreground object would "move" more, because it would also be transformed to those points.

You could try if you can re-use the video stabilisation for your project https://de.mathworks.com/help/vision/examples/video-stabilization-using-point-feature-matching.html

Answer 1

[Gaurav Ahuja]

Let me clarify what I understand based on the description of the issue.

Your application has a constant field of view. I am assuming that the background also remains same (say video of a garden). But there are some local random movements in the background (say movement in tree branches or leaves). Now wish to detect an object in the foreground. You do not wish to track it, but with respect to this object, you wish to make the background static. And by this I understand that you wish to superimpose the foreground object on the static background With this understanding, I feel the bellow approach may be of help.

I feel you should perform 'background learning'. on a very high level, it would mean to average out the frames of the background over a long period of time. This would become your reference frame. the new frame can be compared against this reference. the region of maximum difference would provide you with the new object which is not the part of the reference. if you wish,k you can overlap the maximum difference pixels over the reference frame (this would mean that the object on the foreground is seen over a static reference.)

Obviously, you would need to optimize this approach according to your use case, for example, you would need to learn the background again after some time to account for the change in sunlight. there might be other factors, that may be very specific to your use case and would require you to optimize your solution in that direction.