How to recognize 6 colors of a face rubik's cube at the same time?

[DGM]

Will all the images have the object consistently positioned within the frame?

Will it always be the same cube and lighting?

You might be able to try sampling regions within each square, finding their average colors, and then trying to find the closest match between the observed colors and a list of known colors.

If the object position or lighting changes a lot, things might get complicated.

[Sang]

I understand your problem but I don't know what the command is

Answer 1

[DGM]

Let's start with an example that doesn't solve the problem. Herein, I extract the mean color in each block. Much of this example (the figures) is purely for illustration and wouldn't be necessary in practice. All you need is a correctly ordered color table.

% start with the image
Argb = imread('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1157503/image.jpeg');

% create a mask that samples the centers of the blocks
% stdfilt() is used to find regions of contrast (boundaries of blocks)
% this image is thresholded to create a rough low-contrast region mask
% the image is trimmed and padded with zero to detatch blobs from the border
se = ones(100,1);
blockmask = max(stdfilt(Argb,ones(21)),[],3) < 10;
blockmask = padarray(blockmask(2:end-1,2:end-1),[1 1],0,'both');

% imopen() is used with a long strel to cut the blocks apart
% doing it this way is sensitive to image rotation
% erosion shrinks the mask to exclude regions near block edges
% bwareafilt() gets rid of any remaining small blobs
blockmask = imopen(imopen(blockmask,se),se');
blockmask = imerode(blockmask,ones(25));
blockmask = bwareafilt(blockmask,9);

% this is the mask
imshow(blockmask)

% segment the mask and the source image
% CT is a color table containing the mean color in each mask blob
CC = bwconncomp(blockmask);
nblocks = CC.NumObjects;
CT = zeros(nblocks,3);
Act = reshape(Argb,[],3);
for b = 1:nblocks
    pixidx = CC.PixelIdxList{b};
    sampledpixels = Act(pixidx,:);
    CT(b,:) = mean(sampledpixels,1)/255;
end

% get the centroids of all the blobs
S = regionprops(CC,'centroid');
C = vertcat(S(:).Centroid);

% display indicators on the mask
% note that blobs aren't ordered as expected
clf
imshow(blockmask)
for b = 1:nblocks
    text(C(b,1),C(b,2),sprintf('%d',b),'color','k','fontsize',20)
end

% try to snap the centers to a grid
% again, this assumes the image isn't severely rotated/distorted
climits = [min(C,[],1); max(C,[],1)];
Cnom = round((C-climits(1,:))./range(climits,1)*2 + 1);

% reorder color samples and centroids
idx = sub2ind([3 3],Cnom(:,2),Cnom(:,1));
CT(idx,:) = CT;
C(idx,:) = C;

% display indicators on the mask after sorting
% now the block information is in column-major order
clf
imshow(blockmask)
for b = 1:nblocks
    text(C(b,1),C(b,2),sprintf('%d',b),'color','k','fontsize',20)
end

% display colored patches on the original image
% this shows that the sample colors correspond to the blocks
clf
imshow(Argb); hold on
hs = scatter(C(:,1),C(:,2),5000,CT,'filled');
hs.MarkerEdgeColor = 'k';

% so this is the color table for this image
CT

CT = 9×3

0.0490 0.7965 0.1848 0.1900 0.4245 0.7063 0.8080 0.7477 0.1320 0.9248 0.3623 0.4214 0.8225 0.8080 0.8325 0.9897 0.5775 0.3256 0.9105 0.3310 0.3839 0.0045 0.7953 0.1136 0.9107 0.8509 0.1033

Okay, so now you have a color table. Now what? At this point, you'd have to try to classify the colors. Since I only have one image, I really don't have anything to test against. You might take a few pictures and build a reference table from the average of similar-class samples. Then you can do distance minimization to find which reference color matches a given image under test.

This answer includes a similar process. Similarly, rgb2ind() could be used as well.

If you can provide other images, i might be able to expand on this, otherwise it's up to you to work with the images you're receiving.

[DGM]

Here's one example based on the linked thread:

% let's pretend we have a new image to test
% i'm going to reuse this same image and modify it
Atest = imread('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1157503/image.jpeg');
Atest = rot90(imnoise(Atest,'gaussian',0,0.001),1);

imshow(Atest)

% ------THIS PART IS BASICALLY THE SAME--------

% create a mask that samples the centers of the blocks
% stdfilt() is used to find regions of contrast (boundaries of blocks)
% this image is thresholded to create a rough low-contrast region mask
% the image is trimmed and padded with zero to detatch blobs from the border
se = ones(100,1);
blockmask = max(stdfilt(Atest,ones(21)),[],3) < 10;
blockmask = padarray(blockmask(2:end-1,2:end-1),[1 1],0,'both');

% imopen() is used with a long strel to cut the blocks apart
% erosion shrinks the mask to exclude regions near block edges
% bwareafilt() gets rid of any remaining small blobs
blockmask = imopen(imopen(blockmask,se),se');
blockmask = imerode(blockmask,ones(25));
blockmask = bwareafilt(blockmask,9);

% segment the mask and the source image
CC = bwconncomp(blockmask);
nblocks = CC.NumObjects;
CT = zeros(nblocks,3);
Act = reshape(Atest,[],3);
for b = 1:nblocks
    pixidx = CC.PixelIdxList{b};
    sampledpixels = Act(pixidx,:);
    CT(b,:) = mean(sampledpixels,1)/255;
end

% get the centroids of all the blobs
S = regionprops(CC,'centroid');
C = vertcat(S(:).Centroid);

% try to snap the centers to a grid
% again, this assumes the image isn't severely rotated/distorted
climits = [min(C,[],1); max(C,[],1)];
Cnom = round((C-climits(1,:))./range(climits,1)*2 + 1);

% reorder color samples and centroids
idx = sub2ind([3 3],Cnom(:,2),Cnom(:,1));
CT(idx,:) = CT;
C(idx,:) = C;

% ------THIS PART IS NEW--------

% specify a correlated list of colors and color names
% these are drawn from the reference image(s)
colornames = {'w','r','g','b','y','o'};
colorrefs = [0.8245 0.8105 0.8363; 0.9179 0.3463 0.4028; ...
    0.02596 0.7964 0.148; 0.1896 0.4244 0.7066; ...
    0.8615 0.8011 0.1177; 0.9916 0.58 0.3277];

% find color distances in RGB
D = CT - permute(colorrefs,[3 2 1]);
D = squeeze(sum(D.^2,2)) ;

% find index of closest match for each block
[~,idx] = min(D,[],2);

% look up color names
blocknames = reshape(colornames(idx),3,3)

blocknames = 3×3 cell array
    {'r'}    {'g'}    {'y'}
    {'r'}    {'w'}    {'o'}
    {'g'}    {'b'}    {'y'}

Bear in mind that things like thresholds and whatnot may need to be adjusted. I only made this work with the one image.

[Sang]

ah i understood what you mean. Initially I will make a program like in the above example to find its color series. When I find that string, I will write a program like below that I will get the desired color result, right?

[Sang]

I send you an example like this, can you help me do it like the above example?

[DGM]

Now that there are two examples, I'm going to modify the reference color table by finding the average of the color samples across both images. This isn't something that needs to be done every time, but the reference table should be built from more than one image.

% so this is the CT for the first image
CT = [0.04898 0.7965 0.1848;0.19 0.4245 0.7063;0.808 0.7477 0.132;0.9248 0.3623 0.4214;0.8225 0.808 0.8325;0.9897 0.5775 0.3256;0.9105 0.331 0.3839;0.004511 0.7953 0.1136;0.9107 0.8509 0.1033];
% this is the CT for the second image
CT2 = [0.854 0.4329 0.2038;0.6642 0.6453 0.6144;0.7724 0.25 0.2505;0.1537 0.348 0.5887;0.8225 0.2716 0.2594;0.003548 0.7385 0.002046;0.7865 0.7929 0.731;0.882 0.8471 0.001494;0.8792 0.8484 0.001481];

% find averages to build reference table from CT1,CT2
%   CT1		CT2
% w 5		[2 7]
% r [4 7]	[3 5]
% g [1 8]	6
% b 2		4
% y [3 9]	[8 9]
% o 6		1
% in order: 'w','r','g','b','y','o';
colorrefs = [mean([CT(5,:); CT2([2 7],:)],1);
             mean([CT([4 7],:); CT2([3 5],:)],1); 
             mean([CT([1 8],:); CT2(6,:)],1);
             mean([CT(2,:); CT2(4,:)],1);
             mean([CT([3 9],:); CT2([8 9],:)],1); 
             mean([CT(6,:); CT2(1,:)],1)];

% this is the table to be used later as a reference
mat2str(colorrefs,4)

ans = '[0.7577 0.7487 0.726;0.8576 0.3037 0.3288;0.01901 0.7768 0.1001;0.1719 0.3862 0.6475;0.87 0.8235 0.05957;0.9219 0.5052 0.2647]'

With the new reference table, the prior code should work to identify the colors:

% the new test image
Atest = imread('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1158338/image.jpeg');

% create a mask that samples the centers of the blocks
% stdfilt() is used to find regions of contrast (boundaries of blocks)
% this image is thresholded to create a rough low-contrast region mask
% the image is trimmed and padded with zero to detatch blobs from the border
se = ones(100,1);
blockmask = max(stdfilt(Atest,ones(21)),[],3) < 10;
blockmask = padarray(blockmask(2:end-1,2:end-1),[1 1],0,'both');

% imopen() is used with a long strel to cut the blocks apart
% erosion shrinks the mask to exclude regions near block edges
% bwareafilt() gets rid of any remaining small blobs
blockmask = imopen(imopen(blockmask,se),se');
blockmask = imerode(blockmask,ones(25));
blockmask = bwareafilt(blockmask,9);

% segment the mask and the source image
CC = bwconncomp(blockmask);
nblocks = CC.NumObjects;
CT = zeros(nblocks,3);
Act = reshape(Atest,[],3);
for b = 1:nblocks
    pixidx = CC.PixelIdxList{b};
    sampledpixels = Act(pixidx,:);
    CT(b,:) = mean(sampledpixels,1)/255;
end

% get the centroids of all the blobs
S = regionprops(CC,'centroid');
C = vertcat(S(:).Centroid);

% try to snap the centers to a grid
% again, this assumes the image isn't severely rotated/distorted
climits = [min(C,[],1); max(C,[],1)];
Cnom = round((C-climits(1,:))./range(climits,1)*2 + 1);

% reorder color samples and centroids
idx = sub2ind([3 3],Cnom(:,2),Cnom(:,1));
CT(idx,:) = CT;
C(idx,:) = C;


% specify a correlated list of colors and color names
% these are drawn from the reference image(s)
colornames = {'w','r','g','b','y','o'};
% this is the old reference from image 1 alone
% colorrefs = [0.8245 0.8105 0.8363; 0.9179 0.3463 0.4028; ...
% 			 0.02596 0.7964 0.148; 0.1896 0.4244 0.7066; ...
% 			 0.8615 0.8011 0.1177; 0.9916 0.58 0.3277];
% this is the updated reference from both images (calculated above)
colorrefs = [0.7577 0.7487 0.726;0.8576 0.3037 0.3288; ...
             0.01901 0.7768 0.1001;0.1719 0.3862 0.6475;
             0.87 0.8235 0.05957;0.9219 0.5052 0.2647];

% find color distances in RGB
D = CT - permute(colorrefs,[3 2 1]);
D = squeeze(sum(D.^2,2)) ;

% find index of closest match for each block
[~,idx] = min(D,[],2);

% look up color names
blocknames = reshape(colornames(idx),3,3)

blocknames = 3×3 cell array
    {'o'}    {'b'}    {'w'}
    {'w'}    {'r'}    {'y'}
    {'r'}    {'g'}    {'y'}

As the comments note, the method I used for blob isolation and ordering is sensitive to how well the cube is framed. If the lighting becomes inconsistent, it may also make it difficult to use an approach based on comparison to fixed reference colors like this.

[Sang]

Thank you. can this program be applied to any face of a rubik's cube?

[DGM]

This has been tailored only to two photos of this one particular cube. If the framing/lighting is vastly different, or if the cube has branding printed on one of the faces, then there will be problems.

If it's just another face on this same cube in a similar presentation, it might work. Like I said, the reference CT is still only based on two images. It would still probably be worth building that from a broader average.

[Sang]

if the picture will look like this, will the program change much? can you help me to solve it?

[Sang]

[Image Analyst]

It really depends on your image capture situation. Do you have control over that? Before you had the cube cropped so that only it was there. Now you have it smaller in the image over things that are not very vividly colored. Could the next one be the cube over a very brightly colored and vivid background, like a pile of crayons, or children's toys? If the background could be absolutely anything, then you're going to have to have a more sophisticated algorithm, like perhaps deep learning, to find the cube, than if it's cropped or on similar backgrounds.

[Sang]

If I leave the background white, will the recognition of colors work?

[Image Analyst]

Yes, as long as you don't have the edge faces all white, because then it might be difficult to find the bounding box of the whole cube.

[Sang]

ok thank you

[Sang]

why I still can't receive png images

[Image Analyst]

Why not? Who were you going to receive them from? A JPG should be fine since we're just looking at big differences and don't really care if the color or spatial information is corrupted a little.

[Sang]

when i import png image it gives such error. Is there any other way to recover png and jpg images?

[Image Analyst]

So are you saying that you DID receive the PNG format images but when you try to read them in with imread() it throws an error? The screenshot you gave has nothing to do with whether or not your PNG images can be read in, or are corrupted. Please attach your xylyanh2.m file and your PNG file.

[DGM]

As I mentioned, if the position/framing changes significantly, changes will need to be made. The uncropped photo is a significant change. Likewise, the part of the code that's throwing an error is attempting to re-grid the blobs based on the assumption that the object is close-cropped, centered, and aligned with the frame. The reason it's throwing the error is probably not due to the image being uncentered, but rather because C (the list of centroids) is empty because the segmentation failed for some reason. Since I don't know what PNG file you're testing, I don't know why.

As to what background would work best, choose something that contrasts with everything on the target object. Black would probably be best, since part of the target is nominally white.

Even if it's easy to distinguish the object from the background based on color, it's not simple to universally deal with the perspective. The problem is trying to exclude the sides of the cube and only deal with the top face. Trying to rely on the shadow would make the cropping routine heavily dependent on orientation and lighting.

Unless you need to find and process the cube in any presentation, simplify the task by making the presentations consistent and favorable for the processing task.

This is a naive attempt to crop the top cube face from the given image based on color alone. This will be sensitive to changes in the image, and should be considered not as an example of a robust segmentation, but rather as an example of how easy it is to make robust segmentation difficult.

Auncropped = imread('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1158898/IMG_20221013_201913.jpg');

% assume we can find the cube based on color alone
[H S V] = imsplit(rgb2hsv(Auncropped));
objmask = V.*S > 0.5; % threshold (dependent on lighting/background)
objmask = imclose(objmask,ones(30)); % try to make sure blocks are connected
objmask = bwareaopen(objmask,70E3); % get rid of small blobs (dependent on image size)

% assuming the only thing left is the cube, crop it
% this will break if the cube hasn't been isolated
mkrows = any(objmask,2);
mkcols = any(objmask,1);
Acropped = Auncropped(mkrows,mkcols,:);

% this is the cropped result
imshow(Acropped)

At this point, the prior code should work.

[Sang]

Thank you so much for your help

[Sang]

Can you help me combine these 2 programs to make it work in common?

1.image cropping program

Auncropped = imread('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1158898/IMG_20221013_201913.jpg');
% assume we can find the cube based on color alone
[H S V] = imsplit(rgb2hsv(Auncropped));
objmask = V.*S > 0.5; % threshold (dependent on lighting/background)
objmask = imclose(objmask,ones(30)); % try to make sure blocks are connected
objmask = bwareaopen(objmask,70E3); % get rid of small blobs (dependent on image size)
% assuming the only thing left is the cube, crop it
% this will break if the cube hasn't been isolated
mkrows = any(objmask,2);
mkcols = any(objmask,1);
Acropped = Auncropped(mkrows,mkcols,:);
% this is the cropped result
imshow(Acropped)

2. color recognition program

% the new test image
Atest = imread('rubik8.jpg');
imshow(Atest)
% create a mask that samples the centers of the blocks
% stdfilt() is used to find regions of contrast (boundaries of blocks)
% this image is thresholded to create a rough low-contrast region mask
% the image is trimmed and padded with zero to detatch blobs from the border
se = ones(100,1);
blockmask = max(stdfilt(Atest,ones(21)),[],3) < 10;
blockmask = padarray(blockmask(2:end-1,2:end-1),[1 1],0,'both');
% imopen() is used with a long strel to cut the blocks apart
% erosion shrinks the mask to exclude regions near block edges
% bwareafilt() gets rid of any remaining small blobs
blockmask = imopen(imopen(blockmask,se),se');
blockmask = imerode(blockmask,ones(25));
blockmask = bwareafilt(blockmask,9);
% segment the mask and the source image
CC = bwconncomp(blockmask);
nblocks = CC.NumObjects;
CT = zeros(nblocks,3);
Act = reshape(Atest,[],3);
for b = 1:nblocks
    pixidx = CC.PixelIdxList{b};
    sampledpixels = Act(pixidx,:);
    CT(b,:) = mean(sampledpixels,1)/255;
end
% get the centroids of all the blobs
S = regionprops(CC,'centroid');
C = vertcat(S(:).Centroid);
% try to snap the centers to a grid
% again, this assumes the image isn't severely rotated/distorted
climits = [min(C,[],1); max(C,[],1)];
Cnom = round((C-climits(1,:))./range(climits,1)*2 + 1);
% reorder color samples and centroids
idx = sub2ind([3 3],Cnom(:,2),Cnom(:,1));
CT(idx,:) = CT;
C(idx,:) = C;
% specify a correlated list of colors and color names
% these are drawn from the reference image(s)
colornames = {'w','r','g','b','y','o'};
% this is the old reference from image 1 alone
% colorrefs = [0.8245 0.8105 0.8363; 0.9179 0.3463 0.4028; ...
% 			 0.02596 0.7964 0.148; 0.1896 0.4244 0.7066; ...
% 			 0.8615 0.8011 0.1177; 0.9916 0.58 0.3277];
% this is the updated reference from both images (calculated above)
colorrefs = [0.7577 0.7487 0.726;0.8576 0.3037 0.3288; ...
             0.01901 0.7768 0.1001;0.1719 0.3862 0.6475;
             0.87 0.8235 0.05957;0.9219 0.5052 0.2647];
% find color distances in RGB
D = CT - permute(colorrefs,[3 2 1]);
D = squeeze(sum(D.^2,2)) ;
% find index of closest match for each block
[~,idx] = min(D,[],2);
% look up color names
blocknames = reshape(colornames(idx),3,3)

with the following image as input

[DGM]

The code I gave for cropping and processing should still work on that image. Take Acropped and process it with the second script. In this case, I'm just going to rename Atest to Acropped. Nothing else has been changed.

Auncropped = imread('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1160693/image.jpeg');

% assume we can find the cube based on color
[H S V] = imsplit(rgb2hsv(Auncropped));
objmask = V.*S > 0.5; % threshold (dependent on lighting/background)
objmask = imclose(objmask,ones(30)); % try to make sure blocks are connected
objmask = bwareaopen(objmask,70E3); % get rid of small blobs (dependent on image size)

% assuming the only thing left is the cube, crop it
% this will break if the cube hasn't been isolated
mkrows = any(objmask,2);
mkcols = any(objmask,1);
Acropped = Auncropped(mkrows,mkcols,:);

% this is the cropped result
imshow(Acropped)

% create a mask that samples the centers of the blocks
% stdfilt() is used to find regions of contrast (boundaries of blocks)
% this image is thresholded to create a rough low-contrast region mask
% the image is trimmed and padded with zero to detatch blobs from the border
se = ones(100,1);
blockmask = max(stdfilt(Acropped,ones(21)),[],3) < 10;
blockmask = padarray(blockmask(2:end-1,2:end-1),[1 1],0,'both');

% imopen() is used with a long strel to cut the blocks apart
% erosion shrinks the mask to exclude regions near block edges
% bwareafilt() gets rid of any remaining small blobs
blockmask = imopen(imopen(blockmask,se),se');
blockmask = imerode(blockmask,ones(25));
blockmask = bwareafilt(blockmask,9);

% segment the mask and the source image
CC = bwconncomp(blockmask);
nblocks = CC.NumObjects;
CT = zeros(nblocks,3);
Act = reshape(Acropped,[],3);
for b = 1:nblocks
	pixidx = CC.PixelIdxList{b};
	sampledpixels = Act(pixidx,:);
	CT(b,:) = mean(sampledpixels,1)/255;
end

% get the centroids of all the blobs
S = regionprops(CC,'centroid');
C = vertcat(S(:).Centroid);

% try to snap the centers to a grid
% again, this assumes the image isn't severely rotated/distorted
climits = [min(C,[],1); max(C,[],1)];
Cnom = round((C-climits(1,:))./range(climits,1)*2 + 1);

% reorder color samples and centroids
idx = sub2ind([3 3],Cnom(:,2),Cnom(:,1));
CT(idx,:) = CT;
C(idx,:) = C;

% specify a correlated list of colors and color names
% these are drawn from the reference image(s)
colornames = {'w','r','g','b','y','o'};
colorrefs = [0.7577 0.7487 0.726;0.8576 0.3037 0.3288; ...
			 0.01901 0.7768 0.1001;0.1719 0.3862 0.6475;
			 0.87 0.8235 0.05957;0.9219 0.5052 0.2647];

% find color distances in RGB
D = CT - permute(colorrefs,[3 2 1]);
D = squeeze(sum(D.^2,2)) ;

% find index of closest match for each block
[~,idx] = min(D,[],2);

% look up color names
blocknames = reshape(colornames(idx),3,3)

blocknames = 3×3 cell array
    {'g'}    {'b'}    {'o'}
    {'y'}    {'o'}    {'o'}
    {'r'}    {'y'}    {'w'}

Bear in mind if the cropping procedure needs to become more complicated, it might be convenient to modularize things and break this down into two separate functions or something.

[Sang]

really thank you very much, you help a lot in my project

[Sang]

Can you please tell me how to fix this error

[Image Analyst]

You must have a version older than r2018b, when imsplit was introduced. So you can do

hsvImage = rgb2hsv(Auncropped);
H = hsvImage(:, :, 1);
S = hsvImage(:, :, 2);
V = hsvImage(:, :, 3);

[DGM]

Do you have an older version (R2018a or older), or do you not have IPT? If you don't have IPT, there will be other issues. If you're running something significantly older than R2018x, there may be other version-dependent issues that may show up.

If the only issue is the lack of imsplit(), you can use the attached file to replace it. MIMT splitchans() can do everything IPT imsplit() does, and should not be dependent on anything. You'll obviously have to change that line to call splitchans() instead of imsplit().

Answer 2

[Image Analyst]

There are lots of ways to "solve" this but it depends on what you want to know? Do you want to know only how many colors there are? Do you want to know the predicted reference color for every pixel in the image? Do you want to find each cube face and find the dominant color for that cube face?

What I'd to is to classify every pixel in the image into one of your six known reference colors. For example you could use the attached demo to use discriminant analysis (see attached demo) or K nearest neighbors (see attached demo), or you could convert to LAB color space and find the lowest delta E color difference like I did in this link https://www.mathworks.com/matlabcentral/answers/1819405-how-to-change-the-color-of-each-pixel-in-a-masked-image#comment_2410068 for a poster who was comparing a melanoma spot to 6 reference colors.

Answer 3

[Image Analyst]

I think you need to do it this way. It holds the Guinness Book of World's record for the fastest Rubik solution ever:

https://www.youtube.com/watch?v=by1yz7Toick