To define a function, create a new m-file on the command line
>> edit myfun.m
write your function in this file, first non-command line must contain the function header using the "function" keyword, like function y = myfun(x). Save it, and call it from the command line or within another function or within a script
myfun(123)
You've probably created a file with some matlab commands, i.e., a script, and somewhere below in this script you defined a function. That's not allowed. Either move your function to a separate file, or convert your script to a function by adding a first line like
function mycommands
Now it's a function, and you can define a function within this function. However, such a nested function is visible only within this function, not at the command line and not within other functions. And functions can be nested only one level, but there can be multiple nested functions.
