It is not entirely clear what you need help with. Are you trying to dynamically create 2D or 3D arrays of variable size? I.e., are you trying to create arrays on the fly with variable size, but still use the [ ][ ][ ] indexing syntax? If so, the answer is you can't do this directly in C (although you could easily do this in that "ancient" language called Fortran). My advice is to abandon the [ ][ ][ ] indexing syntax in your function. There is a way to do it, but you have to allocate pointer arrays off to the side to facilitate the syntax. Instead, just do the indexing manually. E.g., to create a "2D" matrix dynamically of size [A][B], one could do this:
int A = whatever;
int B = whatever;
// double x[A][B] substitute:
x = (double *) mxMalloc( A * B * sizeof(*x) );
Then whenever you want to reference x[i][j] in your code, you would write this instead:
Another little trick that some programmers use is to create a macro that facilitates the indexing. E.g.,
#define X(i,j) x[(j)+(i)*B]
Then whenever you would normally have written x[i][j], you can just write X(i,j) instead.
I will make another observation as well. In your text you state "... Z is output which is array of variable size ...". Are you aware that none of the X, I, and z variables in the function argument list are arrays? They are all pointers. The first level of [ ] is always interpreted as "pointer to" when used in a function argument list.