Word processing: How can I get token numbers from a document?

5 views (last 30 days)
I'm trying to tokenize a huge document (wikipedia) (so that I can convert the document to word vectors). I want to convert the giant char array into a numeric array of token IDs (indexing into a dictionary I have) in word order. I was able to write code for this using for loops of regexp()'s, but it's taking days and days to run. I see that tokenizedDocument() might be a good alternative, except that I can't figure out how to get the document back as a list of numeric token IDs.
Has anyone successfully tokenized a document in this way? If so, how?
Thanks!
  1 Comment
E
E on 17 Dec 2021
For example, 'a cat ran ...' should be converted to [1,49,34,...] (where cat is the 49th word in the dictionary, etc).

Sign in to comment.

Answers (1)

Rishabh Singh
Rishabh Singh on 5 Jan 2022
Hi,
You can use "tokenzedDocument" to tokenize your document. The actual performance will be impacted when you will assign rank number to each token. I would suggest you to use map containter for the purpose.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!