You could try to do it in RGB colorspace. That would be faster, though perhaps not as precise, however if it's a quite distinct color with no others like it, then it should work.
See attached demo for my demo of HSV tracking in a video. But like I said, you could just do it in RGB color space and then you wouldn't be doing tons of time consuming math.
And perhaps just reuse BW instead of creating a new variable largestBlob.