Main Content

textanalytics.unicode.nfd

Unicode decomposed normalized form (NFD)

Since R2021a

    Description

    example

    newStr = textanalytics.unicode.nfd(str) normalizes the string str to the Unicode canonical decomposition form (NFD).

    Examples

    collapse all

    Strings that look identical can have different underlying representations. The Unicode canonical decomposition form (NFD) ensures that equivalent strings have a unique binary representation.

    Consider the string "jalapeño" which contains 8 letters.

    str = "jalapeño";
    strlength(str)
    ans = 8
    

    Normalize the string using the textanalytics.unicode.nfd function. On some systems, the output string appears to be identical to the input string.

    newStr = textanalytics.unicode.nfd(str)
    newStr = 
    "jalapeño"
    

    View the number of code points in the new string. The normalized representation includes one extra code point. In this case, the function splits the accented letter "ñ" into two separate code points.

    strlength(newStr)
    ans = 9
    

    Extract the seventh and eighth code points in the normalized string. On some systems, the output appears to be a single character.

    extractBetween(newStr,7,8)
    ans = 
    "ñ"
    

    Check whether the strings str and newStr are equal using the == operator. The operator returns 0 because the strings have different underlying representations.

    tf = str == newStr
    tf = logical
       0
    
    

    Input Arguments

    collapse all

    Input text, specified as a string array, character vector, or cell array of character vectors.

    Example: ["An example of a short sentence."; "A second short sentence."]

    Data Types: string | char | cell

    Output Arguments

    collapse all

    Output text, returned as a string array, character vector, or cell array of character vectors. str and newStr have the same data type.

    Algorithms

    collapse all

    Unicode Normalization Forms

    For more information about Unicode normalization forms, see Unicode Standard Annex #15 Unicode Normalization Forms.

    References

    [1] Whistler, Ken, ed. "Unicode Standard Annex #15: Unicode Normalization Forms." Unicode Technical Reports, August 27, 2021. https://unicode.org/reports/tr15/.

    Version History

    Introduced in R2021a