Annotations are a pain to use unless the axes are normalized to 0:1. I see no reason why Matlab cannot allow users to specify the units in annotations which would make the tool a lot more useful.
"Im just looking for a way to place 'X's at specific xy coordinates to signify that those points are no longer being considered for my calculations"
That's much easier than anything annotation can do. Simply use the xy coordinates to plot "X"'s
If you wanted rectangles for whatever reason,
widthHeight = [.1,.1];
rectangle('Position', [x-widthHeight(1)/2, y-widthHeight(2)/2, widthHeight])