The most important mistake is adding i to hit in the loop instead of just one, i is a counter that goes from 1 to 10000 so it gets really big.
Also, you need to do the error checking with the for loop to allow for preliminary exit after the required precision has been reached.
Finally, be careful with correct naming of your variables. It matters if you write hit or hits and it is not clear were you define hits and it should definitely not be used in calculating the estimated pi as there you want to use the actual number stored in hit
The code below should work as you wanted
maxIt = 10000;
for i = 1:maxIt
x = rand();
y = rand();
if x^2 + y^2 <= 1
hit = hit + 1 ;
piest = 4 * (hit/i);
error = abs(pi-(piest));
if error < .001
sprintf('Number of iterations = %1.0f',i)