That syntax merely takes a random permutation of your data.
So the question becomes, how does the randperm function work?
You're in luck because it's explaind in the randperm documentation page. See the "Tips" section.
You'll see that randperm uses a uniform pseudorandom number generator that can have different internal settings and states. There are several types of random number generators. See the table:
To determine which one you're using, run
struct with fields:
State: [625×1 uint32]
For example, "twister" is defined by the mt19937ar generator which is fully described in the link above and contains citations to primary literature, depending on how far down the rabbit hole you want to go.
To summarize, the randsample(n,k) syntax merely subsamples your data using randperm without replacement. The random selection within randperm is controlled by the random number generator you're using which can be determined by running rng().