Super-complicated? Seriously? :) NOT. But I suppose complexity is in the eyes of the beholder.
Start with this:
That strongly suggests that theta, clearly an angle of some sort, is in degrees. (You don't need the brackets to define a scalar. As well, you should learn to terminate lines with a semi-colon.)
Why do I say this is clearly degrees? Because if it were in terms of radians, 15 radians is equivalent to
This is way larger than 360 degrees. So it seems clear that you are thinking in terms of degrees. It also suggests, that if Theta were really 859.4367... degrees, then the solution might be complex valued. It might be. I have not checked to see if your messy code is actually correct for the not that complex equation you showed.
But to work in degrees, you need to use the functions tand, cosd, sind, etc. Nothing you have done here will make sense, because the functions sin, cos, tan, etc., all use RADIANS. You really need to read the getting started tutorials. Also, get used to the idea that radians are actually a better way to do mathematics, at least as you progress further in mathematics, engineering or the sciences. It will make your life easier in the long term. That is your choice though.
You want to solve the equation for beta. How would use use the symbolic TB to do that?
Eqn = 2*cotd(Beta)*(M_1^2*sind(Beta).^2 - 1)/(M_1^2*(Gamma + cosd(2*Beta)) + 2);
When you break it down into so many sub-terms, this merely makes it MUCH more likely you will make a mistake in one of those terms. Instead, just learn to write code in MATLAB. Then display the result, to make sure it is as you intended.
As you should see, the symbolic toolbox converted the cotd, sind and cosd terms into sines and cosines that are defined in terms of radians. Beta and Theta are still in degrees though.
But now we can solve for a result.
Betasol = solve(Eqn,Beta)
That finds 3 reasonable solutions. And the fact that they are reasonable numbers one would expect to see in the context of degrees makes me more positive that you are thinking in terms of degrees.
Of course, while it found exactly 3 solutions, since this involves periodic functions, we can probably add integer multiples of 180 (degrees) to get new solutions. Only multiples of 180 happens because you have a cos(2*Beta) in your equation. We can find the complete family of solutions as:
Betasol = solve(Eqn,Beta,'returnconditions',true)
So for integer values of k, we can add any integer multiple of 180 degrees to the base solutions.
Regardless, I think this is what you were looking to find. In the end, it all came down to your assumption that functions like sin, cos, and cot are all defined in terms of degrees. They use radians.