Missing from your equation, is the exact definition of G. You're currently using the sobel operator to compute the gradient and as documented in the algorithms section of imgradient whichever operator you use is going to give you vastly different results for the magnitude.
I suspect that your equation use a completely different definition than 'sobel' for G.
As asked by Raunak, for us to be able to tell you what is wrong, we need the actual source image and whatever you compare it to.
As per the formula you have mentioned above, the syntax for Calculating ‘AG’ is mistaken. For implementing the formula, you can use below commands.
%Here X is the array for which Gradient is to be calculated
[Gmag,Gdir] = imgradient(X,'sobel');
% Here since Gmag represent Magnitude of Gradient so no need to take absolute value
AG = sum(sum(Gmag))./(sqrt(2)*(size(X,1)-1)*(size(X,2)-1));
If you want to try other method for calculating gradient you may investigate ’method’ parameter in imgradient. Please keep in mind that imgradientworks only for grayscale images and would not provide required results for RGB images. Here is the link to the documentation of imgradient.