It looks like you're trying to add a boxplot to the y-historgram axes. Recall that the y-histogram in scatterhist is reversed. Also, you're rotating the scatter axes so that x and y are switched but the histograms aren't switched. Why not just switch x and y in the scatterhist function?
h = scatterhist(y,x,___);
To add a boxplot to the y-histogram axis,
Use h(2) for the x-histogram or h(1) for the scatter axes.