Freehand ROI for MEDSAM automated algorithm in medical image labeller

I am trying to implement the segmentation labelling based on MEDSAM from this link into the medical image labeller:

Interactively Segment and Analyze ROI Using MedSAM and Radiomics - MATLAB & Simulink - MathWorks América Latina

Automate Labeling in Medical Image Labeler - MATLAB & Simulink - MathWorks América Latina

However, I the unique way to create do this is opening a figure different than the interface of the medical image labeller, who takes times and is inefficient. Is there a way at least to create the ROI inside the medical image labeller (using freehand), and then passing this ROI to the automated function? I created my function below from the image. Also I was trying to create a listener without success for tuning the ROI while doing the segmentation on the medical image labeller.

function MASK = medsam_p(I,MASK)

%Medical Image Processing Function with MEDSAM

%

% Replace the sample below with your code----------------------------------

medsam = medicalSegmentAnythingModel;

embeddings = extractEmbeddings(medsam,I);

%roi = drawrectangle;

f = figure;

ax = axes(f);

dispIm = imshow(I,[],Parent=ax);

f.WindowState="maximized";

roi = drawrectangle(ax,'Label','ROI');

%wait(roi); % Wait for the user to finish drawing

% Get the position of the drawn rectangle

MASK = segmentObjectsFromEmbeddings(medsam,embeddings,size(I),BoundingBox=roi.Position);

%--------------------------------------------------------------------------

end

Answer 1

[UDAYA PEDDIRAJU]

Hi Emilio.

When I am working on my previous project, I came across the following links, you should have a look at these. Let me know if your issue resolved: https://www.mathworks.com/help/medical-imaging/ug/get-started-with-medical-image-labeler.html and additionally: https://www.mathworks.com/help/medical-imaging/ug/Label-3-d-medical-image-using-medical-image-labeler.html.

Answer 2

[Sanjana]

Hi,

I understand that you want to create an ROI in MedicalImageLabeler using freehand and then pass this ROI to the automated algorithm created to perform segmentation labelling using MEDSAM model.

After creating an freehand ROI in MedicalImageLabeler, it creates a MASK of the ROI, and the Image and MASK are passed as inputs to the automated algorithm created in the "Automate" section.

Inorder to extract the ROI created using freehand and pass this ROI as input to the MEDSAM model, you would need to do some processing on the MASK Data, to determine the ROI positions which can be passed as input to the "segmentObjectsFromEmbeddings" function.

I have taken a sample image and labelled the image using freehand in MedicalImageLabeler, and was able to extract the ROI position from the MASK using the following post processing code,

%find the row and col indexes of white pixels
[row,col] = find(MASK==1);
%Determine the ROI
min_row = min(row);
max_row = max(row);
min_col = min(col);
max_col = max(col);

%code to display the ROI on the Image,
figure
imshow(img)
hold on
rectangle('Position', [min_col, min_row, max_col - min_col, max_row - min_row], ...
          'EdgeColor', 'r', 'LineWidth', 2);
% Annotate the corners
plot(min_col, min_row, 'go', 'MarkerSize', 10, 'LineWidth', 2); % Top-left
plot(max_col, max_row, 'go', 'MarkerSize', 10, 'LineWidth', 2); % Bottom-right
hold off

Now, from the determine ROI position from the MASK data passed to the automated algorithm, we can use this to extract the bounding box coordinates of the freehand ROI and pass it to MEDSAM model as shown in the below code,

function MASK = medsam_p(I,MASK)
%Medical Image Processing Function with MEDSAM
%
% Replace the sample below with your code----------------------------------
medsam = medicalSegmentAnythingModel;
embeddings = extractEmbeddings(medsam,I);
%roi = drawrectangle;
%find the row and col indexes of white pixels
[row,col] = find(MASK==1);
%Determine the ROI
min_row = min(row);
max_row = max(row);
min_col = min(col);
max_col = max(col);

%calculate bounding box coordinates in x,y,w,h format.
boundingBoxValues = [min_row,min_col,max_col-min_col,max_row-min_row];


%wait(roi);  % Wait for the user to finish drawing
% Get the position of the drawn rectangle
MASK = segmentObjectsFromEmbeddings(medsam,embeddings,size(I),BoundingBox=boundingBoxValues);
%--------------------------------------------------------------------------
end

Hope this helps!