What is this:
table = dataset('xlsfile','xyz.xlsx');
X = table.x;
a = 1;
for k = 1:n
a = a * X(k)*i;
Is that all in one single m-file? If so, it's a script followed by a function. And the function program would need to have an extra "end" at the end of it. Those are the rules for a function inside a script.
Also, that is that m-file called? I don't think you can call it program.m because program() is a nested function inside the m-file. You could name it program.m if those first two lines weren't there. Otherwise you'll need to pick a different name, like testProgram.m or something.
But since you seem to want to also call program() from the command line, you'd be best off making an m-file with only these lines of code in them:
function a = program(factorToMultiplyBy)
a = 1;
for k = 1 : n
a = a * X(k) * factorToMultiplyBy;