The easiest way to see what’s going on with it is to use fvtool to look at it, and to give it a short random signal to look at the output:
signal = rand(1, 1E+4);
[y,df] = bandstop(signal, [4500 950000], 5e9);
It appears to be a well-designed filter, and works with a random signal. I suspect that something about your ‘signal’ vector could be part of the problem, possibly its amplitude, since the output of the random signal (with an amplitude between 0 and 1) produces an output on the order of . The normalised frequency of the stopband center frequency is about . (In my experience, this is a bit extreme.)
In situations like this, it’s sometimes worthwhile to design both a lowpass filter and a highpass filter, and then add the output vectors. (There is no way to cascade them, since the output of one filter will not overlap with the output of the other filter.)
The problem does not appear to be with the filter itself, since the fvtool analysis suggests that it’s stable and works well.