For a shape like that, it's pretty easy to get smooth results once you move to polar coordinates.
A = imread('https://www.mathworks.com/matlabcentral/answers/uploaded_files/912650/image.png');
s = size(A);
cn = round(s(1:2)/2); % [ycenter xcenter]
% find locations of outline pixels
[idxy idxx] = find(A>128); % in rect coord
[idxth idxr] = cart2pol(idxx-cn(2),idxy-cn(1)); % in polar coord
% sort pixel locations by angular position wrt image center
[idxth sortmap] = sort(idxth,'ascend');
idxr = idxr(sortmap);
% one approach would be to fit a spline to both interpolate and denoise
newth = linspace(-pi,pi,1000).';
pp = fit(idxth,idxr,'smoothingspline','smoothingparam',1-1E-5);
newr = pp(newth);
% show the two profiles
plot(idxth,idxr,'.'); hold on
plot(newth,newr,'.')
% construct output image
[newx newy] = pol2cart(newth,newr);
B = false(s);
B(sub2ind(s,round(newy+cn(1)),round(newx+cn(2)))) = true;
figure
imshow(B)
Of course, that would require CFT. If you don't have that toolbox, you can get perfectly reasonable results with regular 1D interpolation:
figure
% otherwise, you could use any 1-D interpolation or fitting method
newth = linspace(-pi,pi,1000).';
newr = interp1(idxth,idxr,newth,'makima');
nanmk = ~isnan(newr);
newr = newr(nanmk);
newth = newth(nanmk);
% show the two profiles
plot(idxth,idxr,'.'); hold on
plot(newth,newr,'.')
% construct output image
[newx newy] = pol2cart(newth,newr);
B = false(s);
B(sub2ind(s,round(newy+cn(1)),round(newx+cn(2)))) = true;
figure
imshow(B)
Note that I assumed that the center of the image corresponds to the center of the object. If that's not the case, you'd obviously want to find the center of the object's bounding box.
