Mojibake (Mojibake v0.8.0)

Copy Markdown View Source

Unicode text processing for Elixir, backed by the Mojibake C library.

Mojibake adds what String and :unicode leave out, with Unicode data that does not depend on the OTP release:

  • normalization to NFC, NFD, NFKC, NFKD, and NFKC_Casefold, with quick checks (UAX #15)
  • locale-aware upper, lower, and titlecase, full and simple case folding
  • caseless matching (Unicode Core Specification D144-D147)
  • grapheme, word, and sentence segmentation (UAX #29) and line-break opportunities (UAX #14)
  • terminal width (UAX #11) and truncation to a number of terminal columns

All functions take and return UTF-8 binaries.

Malformed input

Transforms such as normalize/3 and upcase/2 take a :malformed option that says what to do with input that is not well-formed UTF-8:

  • :stop (the default) - return {:error, %Mojibake.Error{reason: :malformed_input}} with the byte offset and length of the first ill-formed subsequence
  • :replace - replace each maximal ill-formed subsequence with U+FFFD
  • :skip - drop each maximal ill-formed subsequence

Queries such as graphemes/1, grapheme_count/1, and terminal_width/2 never fail: they treat each maximal ill-formed subsequence as U+FFFD.

Scheduling

Inputs larger than 16 KiB are processed on a dirty CPU scheduler, so that long strings do not block the normal schedulers.

Summary

Types

A caseless matching relation from the Unicode Core Specification

A normalization form.

A line-break opportunity and the byte offset it is at.

A casing locale: nil for locale-independent casing, or a BCP 47 language tag such as "tr" or :lt. Only the locales that SpecialCasing.txt tailors (Turkish, Azerbaijani, and Lithuanian) change the result; any other well-formed tag behaves like nil.

What to do with input that is not well-formed UTF-8.

How East Asian Width Ambiguous characters are measured (UAX #11): one terminal cell (:narrow) or two, as in CJK terminals (:east_asian).

Functions

Case-fold string for caseless comparison.

Like casefold/2, but returns the string or raises Mojibake.Error.

Return true if first and second are equal without regard to case, under the caseless matching relation mode (see caseless_mode/0). Defaults to :canonical.

The number of codepoints in string. Each maximal ill-formed subsequence counts as one.

Convert string to lowercase with the full case mappings of Unicode, including the Greek final sigma rule.

Like downcase/2, but returns the string or raises Mojibake.Error.

The optional features the C library was compiled with: any of :names, :collation, :idna, and :security.

The number of extended grapheme clusters in string.

The first count extended grapheme clusters of string, or all of string if it has fewer.

Split string into extended grapheme clusters (UAX #29).

The line-break opportunities of string (UAX #14), as {kind, byte_offset} tuples in order.

Normalize string to form (UAX #15).

Return true if string is in normalization form.

The quick check of UAX #15 for string and form: :yes, :no, or :maybe.

The number of sentences in string.

Split string into sentences (UAX #29). Each sentence keeps its trailing spaces.

The number of terminal columns that string takes, for printable single-line text.

Titlecase each word of string: the first cased character of each UAX #29 word is titlecased and the rest are lowercased.

Like titlecase/2, but returns the string or raises Mojibake.Error.

The longest prefix of string that fits in columns terminal columns, without splitting a grapheme.

The version of Unicode that the data in this library follows.

Convert string to uppercase with the full case mappings of Unicode, which can change its length.

Like upcase/2, but returns the string or raises Mojibake.Error.

The version of the Mojibake C library.

The number of words in string: the word-break segments that contain a letter or a digit.

The first count segments that words/1 returns, including spaces and punctuation, or all of string if it has fewer.

Split string at the word boundaries of UAX #29.

Types

caseless_mode()

@type caseless_mode() :: :canonical | :default | :compatibility | :identifier

A caseless matching relation from the Unicode Core Specification:

  • :canonical - D145, NFD(toCasefold(NFD(X)))
  • :default - D144, toCasefold(X), without normalization
  • :compatibility - D146, compatibility caseless match
  • :identifier - D147, toNFKC_Casefold(NFD(X))

form()

@type form() :: :nfc | :nfd | :nfkc | :nfkd | :nfkc_casefold

A normalization form.

line_break()

@type line_break() :: {:mandatory | :allowed, non_neg_integer()}

A line-break opportunity and the byte offset it is at.

locale()

@type locale() :: String.t() | atom() | nil

A casing locale: nil for locale-independent casing, or a BCP 47 language tag such as "tr" or :lt. Only the locales that SpecialCasing.txt tailors (Turkish, Azerbaijani, and Lithuanian) change the result; any other well-formed tag behaves like nil.

malformed()

@type malformed() :: :stop | :replace | :skip

What to do with input that is not well-formed UTF-8.

width_profile()

@type width_profile() :: :narrow | :east_asian

How East Asian Width Ambiguous characters are measured (UAX #11): one terminal cell (:narrow) or two, as in CJK terminals (:east_asian).

Functions

casefold(string, opts \\ [])

@spec casefold(String.t(), keyword()) ::
  {:ok, String.t()} | {:error, Mojibake.Error.t()}

Case-fold string for caseless comparison.

Options

  • :mode - :full (the default) applies the C and F mappings of CaseFolding.txt, which can make the string longer. :simple applies the C and S mappings, which map each character to one character.
  • :malformed - see the module documentation. Defaults to :stop.

To compare two strings without regard to case, caseless_equal?/3 is usually what you want.

Examples

iex> Mojibake.casefold("Straße")
{:ok, "strasse"}

iex> Mojibake.casefold("Straße", mode: :simple)
{:ok, "straße"}

casefold!(string, opts \\ [])

@spec casefold!(String.t(), keyword()) :: String.t()

Like casefold/2, but returns the string or raises Mojibake.Error.

caseless_equal?(first, second, mode \\ :canonical)

@spec caseless_equal?(String.t(), String.t(), caseless_mode()) :: boolean()

Return true if first and second are equal without regard to case, under the caseless matching relation mode (see caseless_mode/0). Defaults to :canonical.

Ill-formed UTF-8 is compared as if each maximal ill-formed subsequence were U+FFFD.

Examples

iex> Mojibake.caseless_equal?("Straße", "STRASSE")
true

iex> Mojibake.caseless_equal?("Å", "a\u030A")
true

iex> Mojibake.caseless_equal?("Å", "a\u030A", :default)
false

iex> Mojibake.caseless_equal?("fi", "FI", :compatibility)
true

codepoint_count(string)

@spec codepoint_count(String.t()) :: non_neg_integer()

The number of codepoints in string. Each maximal ill-formed subsequence counts as one.

Examples

iex> Mojibake.codepoint_count("e\u0301")
2

downcase(string, opts \\ [])

@spec downcase(String.t(), keyword()) ::
  {:ok, String.t()} | {:error, Mojibake.Error.t()}

Convert string to lowercase with the full case mappings of Unicode, including the Greek final sigma rule.

Takes the same options as upcase/2.

Examples

iex> Mojibake.downcase("ΟΔΟΣ")
{:ok, "οδος"}

iex> Mojibake.downcase("DİYARBAKIR", locale: "tr")
{:ok, "diyarbakır"}

downcase!(string, opts \\ [])

@spec downcase!(String.t(), keyword()) :: String.t()

Like downcase/2, but returns the string or raises Mojibake.Error.

features()

@spec features() :: [:names | :collation | :idna | :security]

The optional features the C library was compiled with: any of :names, :collation, :idna, and :security.

They are all on by default. See the README to turn them off.

grapheme_count(string)

@spec grapheme_count(String.t()) :: non_neg_integer()

The number of extended grapheme clusters in string.

Examples

iex> Mojibake.grapheme_count("e\u0301🇮🇹")
2

grapheme_prefix(string, count)

@spec grapheme_prefix(String.t(), non_neg_integer()) :: String.t()

The first count extended grapheme clusters of string, or all of string if it has fewer.

Examples

iex> Mojibake.grapheme_prefix("e\u0301🇮🇹abc", 2)
"e\u0301🇮🇹"

graphemes(string)

@spec graphemes(String.t()) :: [String.t()]

Split string into extended grapheme clusters (UAX #29).

Each grapheme is a sub-binary of string; nothing is copied.

Examples

iex> Mojibake.graphemes("e\u0301🇮🇹👨‍👩‍👧")
["e\u0301", "🇮🇹", "👨‍👩‍👧"]

line_breaks(string)

@spec line_breaks(String.t()) :: [line_break()]

The line-break opportunities of string (UAX #14), as {kind, byte_offset} tuples in order.

kind is :mandatory where a line must break, for example after a newline, and :allowed where it may. A line can break before the byte at byte_offset. The end of the text is always the last opportunity.

Examples

iex> Mojibake.line_breaks("Hello world")
[allowed: 6, mandatory: 11]

iex> Mojibake.line_breaks("a\nb")
[mandatory: 2, mandatory: 3]

normalize(string, form \\ :nfc, opts \\ [])

@spec normalize(String.t(), form(), keyword()) ::
  {:ok, String.t()} | {:error, Mojibake.Error.t()}

Normalize string to form (UAX #15).

form is :nfc (the default), :nfd, :nfkc, :nfkd, or :nfkc_casefold. When string is already normalized, it is returned as is, without a copy.

Options

  • :malformed - see the module documentation. Defaults to :stop.

Examples

iex> Mojibake.normalize("e\u0301")
{:ok, "é"}

iex> Mojibake.normalize("fi", :nfkc)
{:ok, "fi"}

iex> Mojibake.normalize("Straße", :nfkc_casefold)
{:ok, "strasse"}

iex> {:error, error} = Mojibake.normalize(<<"a", 0xFF>>)
iex> {error.reason, error.offset, error.length}
{:malformed_input, 1, 1}

normalize!(string, form \\ :nfc, opts \\ [])

@spec normalize!(String.t(), form(), keyword()) :: String.t()

Like normalize/3, but returns the string or raises Mojibake.Error.

Examples

iex> Mojibake.normalize!("Å", :nfd)
"A\u030A"

normalized?(string, form \\ :nfc)

@spec normalized?(String.t(), form()) :: boolean()

Return true if string is in normalization form.

It runs the quick check of UAX #15 and normalizes only when the quick check cannot decide. Input that is not well-formed UTF-8 is never normalized.

Examples

iex> Mojibake.normalized?("é")
true

iex> Mojibake.normalized?("e\u0301")
false

iex> Mojibake.normalized?("e\u0301", :nfd)
true

quick_check(string, form)

@spec quick_check(String.t(), :nfc | :nfd | :nfkc | :nfkd) ::
  {:ok, :yes | :no | :maybe} | {:error, Mojibake.Error.t()}

The quick check of UAX #15 for string and form: :yes, :no, or :maybe.

form is :nfc, :nfd, :nfkc, or :nfkd. A :maybe answer needs a full normalization to decide; normalized?/2 does that.

Examples

iex> Mojibake.quick_check("abc", :nfc)
{:ok, :yes}

iex> Mojibake.quick_check("e\u0301", :nfc)
{:ok, :maybe}

sentence_count(string)

@spec sentence_count(String.t()) :: non_neg_integer()

The number of sentences in string.

Examples

iex> Mojibake.sentence_count("One. Two? Three!")
3

sentences(string)

@spec sentences(String.t()) :: [String.t()]

Split string into sentences (UAX #29). Each sentence keeps its trailing spaces.

Examples

iex> Mojibake.sentences("Hello. How are you? Fine!")
["Hello. ", "How are you? ", "Fine!"]

terminal_width(string, opts \\ [])

@spec terminal_width(String.t(), keyword()) :: non_neg_integer()

The number of terminal columns that string takes, for printable single-line text.

Wide and fullwidth characters (UAX #11) and most emoji take two columns, combining marks and other zero-width characters take none.

Options

Examples

iex> Mojibake.terminal_width("abc")
3

iex> Mojibake.terminal_width("日本語")
6

iex> Mojibake.terminal_width("e\u0301")
1

iex> Mojibake.terminal_width("α", profile: :east_asian)
2

titlecase(string, opts \\ [])

@spec titlecase(String.t(), keyword()) ::
  {:ok, String.t()} | {:error, Mojibake.Error.t()}

Titlecase each word of string: the first cased character of each UAX #29 word is titlecased and the rest are lowercased.

Takes the same options as upcase/2.

Examples

iex> Mojibake.titlecase("hello WORLD")
{:ok, "Hello World"}

iex> Mojibake.titlecase("džungla")
{:ok, "Džungla"}

titlecase!(string, opts \\ [])

@spec titlecase!(String.t(), keyword()) :: String.t()

Like titlecase/2, but returns the string or raises Mojibake.Error.

truncate_to_width(string, columns, opts \\ [])

@spec truncate_to_width(String.t(), non_neg_integer(), keyword()) :: String.t()

The longest prefix of string that fits in columns terminal columns, without splitting a grapheme.

Options

  • :profile - :narrow (the default) or :east_asian. See width_profile/0.
  • :break - :grapheme (the default) cuts between graphemes. :word cuts only between the segments that words/1 returns, so it does not split words.

Examples

iex> Mojibake.truncate_to_width("日本語", 5)
"日本"

iex> Mojibake.truncate_to_width("Hello world", 8)
"Hello wo"

iex> Mojibake.truncate_to_width("Hello world", 8, break: :word)
"Hello "

unicode_version()

@spec unicode_version() :: String.t()

The version of Unicode that the data in this library follows.

Examples

iex> Mojibake.unicode_version()
"18.0.0"

upcase(string, opts \\ [])

@spec upcase(String.t(), keyword()) ::
  {:ok, String.t()} | {:error, Mojibake.Error.t()}

Convert string to uppercase with the full case mappings of Unicode, which can change its length.

Options

  • :locale - a BCP 47 language tag that tailors the mapping, such as "tr". Defaults to nil, locale-independent casing. See locale/0.
  • :malformed - see the module documentation. Defaults to :stop.

Examples

iex> Mojibake.upcase("straße")
{:ok, "STRASSE"}

iex> Mojibake.upcase("istanbul", locale: "tr")
{:ok, "İSTANBUL"}

upcase!(string, opts \\ [])

@spec upcase!(String.t(), keyword()) :: String.t()

Like upcase/2, but returns the string or raises Mojibake.Error.

version()

@spec version() :: String.t()

The version of the Mojibake C library.

Examples

iex> Mojibake.version()
"0.8.0"

word_count(string)

@spec word_count(String.t()) :: non_neg_integer()

The number of words in string: the word-break segments that contain a letter or a digit.

Scripts that are usually segmented with a dictionary, such as Chinese, Japanese, and Thai, count about one word per character.

Examples

iex> Mojibake.word_count("Hello, world! 42")
3

word_prefix(string, count)

@spec word_prefix(String.t(), non_neg_integer()) :: String.t()

The first count segments that words/1 returns, including spaces and punctuation, or all of string if it has fewer.

Examples

iex> Mojibake.word_prefix("Hello, world!", 4)
"Hello, world"

words(string)

@spec words(String.t()) :: [String.t()]

Split string at the word boundaries of UAX #29.

The result includes the spaces and punctuation between words, so the segments always join back into string.

Examples

iex> Mojibake.words("Hello, world!")
["Hello", ",", " ", "world", "!"]

iex> Mojibake.words("can't stop 3.14")
["can't", " ", "stop", " ", "3.14"]