I have no idea what Omega is, or what most of that mathematical set theory mumbo jumbo is. Just state it in words. If you want to take the median filter, use medfilt2().
medianFilteredImage = medfilt2(grayImage, [a, b]);
As far as determining what Omega should be, I have no idea since I don't know what Omega is.