How can I move my mouse according to a centroid value?

I have been trying to complete a gesture recognition mouse control program in MATLAB and want to do this without using color detection. So far I have been able to structure the code how I best see fit but I am unable to get the mouse to move with the centroid of my hand.

The gestures I have are number-based:

1 - the mouse will move according to the centroid of my hand.

2 - left click

3 - right click

4 - double click

5- scroll

I have gesture 1 as the start of a while loop with gestures 2-5 inside the loop. Is there any way I can move the mouse according to my centroid value?

I have tried using color detection, which works well, but it isnt an option for my final project. So far, the finger detection has worked for gestures 2-5 but not for 1. Any help is greatly appreciated!

close all
%% Import Java Robot
import java.awt.Robot;
import java.awt.event.*
mouse = Robot;
 
%% Create video input object. 
cam = videoinput('macvideo');   %OR ('winvideo');
cam.Name = 'FaceTime HD Camera'; %change your camera name if needed
cam.TriggerRepeat = 10000;  %change to Inf when needed
cam.FrameGrabInterval = 1;
 
picture = getsnapshot(cam);
frameSize = size(picture);
videoplayer = vision.VideoPlayer('Name', 'Hand Gesture Recognition', 'Position', [100 100 [frameSize(2), frameSize(1)]]);
set(cam,'ReturnedColorspace','rgb')
 
%% Count Number of Fingers
runloop = true;
while runloop
    img1 = getsnapshot(cam);
    img1 = rgb2gray(img1);
    img2 = imcomplement(imbinarize(img1));
    img3 = imfill(img2, 'holes');               %filling all holes
    img4 = bwareaopen(img3, 10000);             %removing objects lesser than 10K size
    
    SE = strel('disk', 10);                     %defining structural elements
    SE1 = strel('disk', 50);
    SE2 = strel('disk', 60);
    img4e = imerode(img4, SE1);                 %image erosion
    img4d = imdilate(img4e, SE2);               %image dilation
    imgfo = img4-img4d;                         %getting fingers
    imgfo(imgfo==1)=0;
    imgfo = logical(imgfo);
    imgfo = bwareaopen(imgfo,5000);
    CC = bwconncomp(imgfo);                     %connected component analysis
    nof = CC.NumObjects;                        %getting number of fingers
    imgfog = uint8(255.*imgfo);
    nofs = num2str(nof);
    imgfogrgb = insertText(imgfog, [0,0], nofs, 'FontSize', 30, 'BoxColor', 'white', 'BoxOpacity', 1, 'TextColor', 'black');
    step(videoplayer, imgfogrgb);
    runloop = isOpen(videoplayer);              %checking if videoplayer is on or off
    set(gcf, 'Position',  [100, 100, 500, 400])
    movegui('northeast')
    pause(2);
  
%     preview(cam);
%     movegui('northeast');
    
%% Begin Move Movements
while nof==1 %move
     st = regionprops(imgfo, 'BoundingBox', 'Centroid');
     imshow(imgfo)
     hold on
     for k = 1 : length(st) 
     bbox = st(k).BoundingBox;
     cbox = st(k).Centroid;
     rectangle('Position',bbox,'EdgeColor','r','LineWidth',1 )
     plot(cbox(1),cbox(2), '-m+')
     a = text(cbox(1)+15, cbox(2), strcat('X:  ', num2str(round(cbox(1))), '    Y: ', num2str(round(cbox(2)))));
     set(a, 'FontSize', 12, 'Color', 'white');
     %hold on;
     end
     X = cbox(1);
     Y = cbox(2);
     set(0,'PointerLocation',[X Y]);
     centroids = cat(1,st.Centroid);
     %mouse.mouseMove(1600-(centroids(1,1)*(5/2)),(centroids(1,2)*(5/2)-180)); 

[Walter Roberson]

set figure or axes CurrentPoint rather than root PointerLocation

https://www.mathworks.com/matlabcentral/answers/429274-get-axes-currentpoint-stack-on-not-updated-current-point#answer_347137

Answer 1

[Avni Agrawal]

Hi !@Priyana Kumar,

I understand that you are tryinh to move the mouse cursor based on the centroid of your hand, ensure you map the centroid coordinates from the image space to your screen resolution and use the Java `Robot` class effectively. Here's a concise guide:

1. Calculate the Screen Position: Scale the hand's centroid position from the image coordinates to your screen's coordinates.

2. Use Java `Robot` to Move Mouse:

% Assuming 'cbox' is the centroid [X, Y] of your hand in the image
screenSize = get(0, 'screensize'); % [left, bottom, width, height]
screenWidth = screenSize(3);
screenHeight = screenSize(4);

imageWidth = size(img1, 2); % 'img1' is your image matrix
imageHeight = size(img1, 1);

% Scale centroid to screen resolution
scaledX = round((cbox(1) / imageWidth) * screenWidth);
scaledY = round((cbox(2) / imageHeight) * screenHeight);

% Move mouse
mouse.mouseMove(scaledX, scaledY);

Ensure you correctly detect the centroid `cbox` of your hand for this to work. Adjust the scaling as necessary to match your screen's resolution and the camera's field of view.

I hope this helps.