# `Mojibake`
[🔗](https://github.com/zaerl/mojibake/blob/main/src/elixir/lib/mojibake.ex#L5)

Unicode text processing for Elixir, backed by the Mojibake C library.

Mojibake adds what `String` and `:unicode` leave out, with Unicode data that does not
depend on the OTP release:

  * normalization to NFC, NFD, NFKC, NFKD, and NFKC_Casefold, with quick checks (UAX #15)
  * locale-aware upper, lower, and titlecase, full and simple case folding
  * caseless matching (Unicode Core Specification D144-D147)
  * grapheme, word, and sentence segmentation (UAX #29) and line-break opportunities (UAX #14)
  * terminal width (UAX #11) and truncation to a number of terminal columns

All functions take and return UTF-8 binaries.

## Malformed input

Transforms such as `normalize/3` and `upcase/2` take a `:malformed` option that says what to do
with input that is not well-formed UTF-8:

  * `:stop` (the default) - return `{:error, %Mojibake.Error{reason: :malformed_input}}` with
    the byte offset and length of the first ill-formed subsequence
  * `:replace` - replace each maximal ill-formed subsequence with U+FFFD
  * `:skip` - drop each maximal ill-formed subsequence

Queries such as `graphemes/1`, `grapheme_count/1`, and `terminal_width/2` never fail: they
treat each maximal ill-formed subsequence as U+FFFD.

## Scheduling

Inputs larger than 16 KiB are processed on a dirty CPU scheduler, so that long strings do not
block the normal schedulers.

# `caseless_mode`

```elixir
@type caseless_mode() :: :canonical | :default | :compatibility | :identifier
```

A caseless matching relation from the Unicode Core Specification:

  * `:canonical` - D145, `NFD(toCasefold(NFD(X)))`
  * `:default` - D144, `toCasefold(X)`, without normalization
  * `:compatibility` - D146, compatibility caseless match
  * `:identifier` - D147, `toNFKC_Casefold(NFD(X))`

# `form`

```elixir
@type form() :: :nfc | :nfd | :nfkc | :nfkd | :nfkc_casefold
```

A normalization form.

# `line_break`

```elixir
@type line_break() :: {:mandatory | :allowed, non_neg_integer()}
```

A line-break opportunity and the byte offset it is at.

# `locale`

```elixir
@type locale() :: String.t() | atom() | nil
```

A casing locale: `nil` for locale-independent casing, or a BCP 47 language tag such as `"tr"`
or `:lt`. Only the locales that `SpecialCasing.txt` tailors (Turkish, Azerbaijani, and
Lithuanian) change the result; any other well-formed tag behaves like `nil`.

# `malformed`

```elixir
@type malformed() :: :stop | :replace | :skip
```

What to do with input that is not well-formed UTF-8.

# `width_profile`

```elixir
@type width_profile() :: :narrow | :east_asian
```

How East Asian Width Ambiguous characters are measured (UAX #11): one terminal cell
(`:narrow`) or two, as in CJK terminals (`:east_asian`).

# `casefold`

```elixir
@spec casefold(String.t(), keyword()) ::
  {:ok, String.t()} | {:error, Mojibake.Error.t()}
```

Case-fold `string` for caseless comparison.

## Options

  * `:mode` - `:full` (the default) applies the `C` and `F` mappings of `CaseFolding.txt`, which
    can make the string longer. `:simple` applies the `C` and `S` mappings, which map each
    character to one character.
  * `:malformed` - see the module documentation. Defaults to `:stop`.

To compare two strings without regard to case, `caseless_equal?/3` is usually what you want.

## Examples

    iex> Mojibake.casefold("Straße")
    {:ok, "strasse"}

    iex> Mojibake.casefold("Straße", mode: :simple)
    {:ok, "straße"}

# `casefold!`

```elixir
@spec casefold!(String.t(), keyword()) :: String.t()
```

Like `casefold/2`, but returns the string or raises `Mojibake.Error`.

# `caseless_equal?`

```elixir
@spec caseless_equal?(String.t(), String.t(), caseless_mode()) :: boolean()
```

Return `true` if `first` and `second` are equal without regard to case, under the caseless
matching relation `mode` (see `t:caseless_mode/0`). Defaults to `:canonical`.

Ill-formed UTF-8 is compared as if each maximal ill-formed subsequence were U+FFFD.

## Examples

    iex> Mojibake.caseless_equal?("Straße", "STRASSE")
    true

    iex> Mojibake.caseless_equal?("Å", "a\u030A")
    true

    iex> Mojibake.caseless_equal?("Å", "a\u030A", :default)
    false

    iex> Mojibake.caseless_equal?("ﬁ", "FI", :compatibility)
    true

# `codepoint_count`

```elixir
@spec codepoint_count(String.t()) :: non_neg_integer()
```

The number of codepoints in `string`. Each maximal ill-formed subsequence counts as one.

## Examples

    iex> Mojibake.codepoint_count("e\u0301")
    2

# `downcase`

```elixir
@spec downcase(String.t(), keyword()) ::
  {:ok, String.t()} | {:error, Mojibake.Error.t()}
```

Convert `string` to lowercase with the full case mappings of Unicode, including the Greek
final sigma rule.

Takes the same options as `upcase/2`.

## Examples

    iex> Mojibake.downcase("ΟΔΟΣ")
    {:ok, "οδος"}

    iex> Mojibake.downcase("DİYARBAKIR", locale: "tr")
    {:ok, "diyarbakır"}

# `downcase!`

```elixir
@spec downcase!(String.t(), keyword()) :: String.t()
```

Like `downcase/2`, but returns the string or raises `Mojibake.Error`.

# `features`

```elixir
@spec features() :: [:names | :collation | :idna | :security]
```

The optional features the C library was compiled with: any of `:names`, `:collation`, `:idna`,
and `:security`.

They are all on by default. See the README to turn them off.

# `grapheme_count`

```elixir
@spec grapheme_count(String.t()) :: non_neg_integer()
```

The number of extended grapheme clusters in `string`.

## Examples

    iex> Mojibake.grapheme_count("e\u0301🇮🇹")
    2

# `grapheme_prefix`

```elixir
@spec grapheme_prefix(String.t(), non_neg_integer()) :: String.t()
```

The first `count` extended grapheme clusters of `string`, or all of `string` if it has fewer.

## Examples

    iex> Mojibake.grapheme_prefix("e\u0301🇮🇹abc", 2)
    "e\u0301🇮🇹"

# `graphemes`

```elixir
@spec graphemes(String.t()) :: [String.t()]
```

Split `string` into extended grapheme clusters (UAX #29).

Each grapheme is a sub-binary of `string`; nothing is copied.

## Examples

    iex> Mojibake.graphemes("e\u0301🇮🇹👨‍👩‍👧")
    ["e\u0301", "🇮🇹", "👨‍👩‍👧"]

# `line_breaks`

```elixir
@spec line_breaks(String.t()) :: [line_break()]
```

The line-break opportunities of `string` (UAX #14), as `{kind, byte_offset}` tuples in order.

`kind` is `:mandatory` where a line must break, for example after a newline, and `:allowed`
where it may. A line can break before the byte at `byte_offset`. The end of the text is always
the last opportunity.

## Examples

    iex> Mojibake.line_breaks("Hello world")
    [allowed: 6, mandatory: 11]

    iex> Mojibake.line_breaks("a\nb")
    [mandatory: 2, mandatory: 3]

# `normalize`

```elixir
@spec normalize(String.t(), form(), keyword()) ::
  {:ok, String.t()} | {:error, Mojibake.Error.t()}
```

Normalize `string` to `form` (UAX #15).

`form` is `:nfc` (the default), `:nfd`, `:nfkc`, `:nfkd`, or `:nfkc_casefold`. When `string` is
already normalized, it is returned as is, without a copy.

## Options

  * `:malformed` - see the module documentation. Defaults to `:stop`.

## Examples

    iex> Mojibake.normalize("e\u0301")
    {:ok, "é"}

    iex> Mojibake.normalize("ﬁ", :nfkc)
    {:ok, "fi"}

    iex> Mojibake.normalize("Straße", :nfkc_casefold)
    {:ok, "strasse"}

    iex> {:error, error} = Mojibake.normalize(<<"a", 0xFF>>)
    iex> {error.reason, error.offset, error.length}
    {:malformed_input, 1, 1}

# `normalize!`

```elixir
@spec normalize!(String.t(), form(), keyword()) :: String.t()
```

Like `normalize/3`, but returns the string or raises `Mojibake.Error`.

## Examples

    iex> Mojibake.normalize!("Å", :nfd)
    "A\u030A"

# `normalized?`

```elixir
@spec normalized?(String.t(), form()) :: boolean()
```

Return `true` if `string` is in normalization `form`.

It runs the quick check of UAX #15 and normalizes only when the quick check cannot decide.
Input that is not well-formed UTF-8 is never normalized.

## Examples

    iex> Mojibake.normalized?("é")
    true

    iex> Mojibake.normalized?("e\u0301")
    false

    iex> Mojibake.normalized?("e\u0301", :nfd)
    true

# `quick_check`

```elixir
@spec quick_check(String.t(), :nfc | :nfd | :nfkc | :nfkd) ::
  {:ok, :yes | :no | :maybe} | {:error, Mojibake.Error.t()}
```

The quick check of UAX #15 for `string` and `form`: `:yes`, `:no`, or `:maybe`.

`form` is `:nfc`, `:nfd`, `:nfkc`, or `:nfkd`. A `:maybe` answer needs a full normalization to
decide; `normalized?/2` does that.

## Examples

    iex> Mojibake.quick_check("abc", :nfc)
    {:ok, :yes}

    iex> Mojibake.quick_check("e\u0301", :nfc)
    {:ok, :maybe}

# `sentence_count`

```elixir
@spec sentence_count(String.t()) :: non_neg_integer()
```

The number of sentences in `string`.

## Examples

    iex> Mojibake.sentence_count("One. Two? Three!")
    3

# `sentences`

```elixir
@spec sentences(String.t()) :: [String.t()]
```

Split `string` into sentences (UAX #29). Each sentence keeps its trailing spaces.

## Examples

    iex> Mojibake.sentences("Hello. How are you? Fine!")
    ["Hello. ", "How are you? ", "Fine!"]

# `terminal_width`

```elixir
@spec terminal_width(String.t(), keyword()) :: non_neg_integer()
```

The number of terminal columns that `string` takes, for printable single-line text.

Wide and fullwidth characters (UAX #11) and most emoji take two columns, combining marks and
other zero-width characters take none.

## Options

  * `:profile` - `:narrow` (the default) or `:east_asian`. See `t:width_profile/0`.

## Examples

    iex> Mojibake.terminal_width("abc")
    3

    iex> Mojibake.terminal_width("日本語")
    6

    iex> Mojibake.terminal_width("e\u0301")
    1

    iex> Mojibake.terminal_width("α", profile: :east_asian)
    2

# `titlecase`

```elixir
@spec titlecase(String.t(), keyword()) ::
  {:ok, String.t()} | {:error, Mojibake.Error.t()}
```

Titlecase each word of `string`: the first cased character of each UAX #29 word is
titlecased and the rest are lowercased.

Takes the same options as `upcase/2`.

## Examples

    iex> Mojibake.titlecase("hello WORLD")
    {:ok, "Hello World"}

    iex> Mojibake.titlecase("ǆungla")
    {:ok, "ǅungla"}

# `titlecase!`

```elixir
@spec titlecase!(String.t(), keyword()) :: String.t()
```

Like `titlecase/2`, but returns the string or raises `Mojibake.Error`.

# `truncate_to_width`

```elixir
@spec truncate_to_width(String.t(), non_neg_integer(), keyword()) :: String.t()
```

The longest prefix of `string` that fits in `columns` terminal columns, without splitting a
grapheme.

## Options

  * `:profile` - `:narrow` (the default) or `:east_asian`. See `t:width_profile/0`.
  * `:break` - `:grapheme` (the default) cuts between graphemes. `:word` cuts only between the
    segments that `words/1` returns, so it does not split words.

## Examples

    iex> Mojibake.truncate_to_width("日本語", 5)
    "日本"

    iex> Mojibake.truncate_to_width("Hello world", 8)
    "Hello wo"

    iex> Mojibake.truncate_to_width("Hello world", 8, break: :word)
    "Hello "

# `unicode_version`

```elixir
@spec unicode_version() :: String.t()
```

The version of Unicode that the data in this library follows.

## Examples

    iex> Mojibake.unicode_version()
    "18.0.0"

# `upcase`

```elixir
@spec upcase(String.t(), keyword()) ::
  {:ok, String.t()} | {:error, Mojibake.Error.t()}
```

Convert `string` to uppercase with the full case mappings of Unicode, which can change its
length.

## Options

  * `:locale` - a BCP 47 language tag that tailors the mapping, such as `"tr"`. Defaults to
    `nil`, locale-independent casing. See `t:locale/0`.
  * `:malformed` - see the module documentation. Defaults to `:stop`.

## Examples

    iex> Mojibake.upcase("straße")
    {:ok, "STRASSE"}

    iex> Mojibake.upcase("istanbul", locale: "tr")
    {:ok, "İSTANBUL"}

# `upcase!`

```elixir
@spec upcase!(String.t(), keyword()) :: String.t()
```

Like `upcase/2`, but returns the string or raises `Mojibake.Error`.

# `version`

```elixir
@spec version() :: String.t()
```

The version of the Mojibake C library.

## Examples

    iex> Mojibake.version()
    "0.8.0"

# `word_count`

```elixir
@spec word_count(String.t()) :: non_neg_integer()
```

The number of words in `string`: the word-break segments that contain a letter or a digit.

Scripts that are usually segmented with a dictionary, such as Chinese, Japanese, and Thai,
count about one word per character.

## Examples

    iex> Mojibake.word_count("Hello, world! 42")
    3

# `word_prefix`

```elixir
@spec word_prefix(String.t(), non_neg_integer()) :: String.t()
```

The first `count` segments that `words/1` returns, including spaces and punctuation, or all of
`string` if it has fewer.

## Examples

    iex> Mojibake.word_prefix("Hello, world!", 4)
    "Hello, world"

# `words`

```elixir
@spec words(String.t()) :: [String.t()]
```

Split `string` at the word boundaries of UAX #29.

The result includes the spaces and punctuation between words, so the segments always join
back into `string`.

## Examples

    iex> Mojibake.words("Hello, world!")
    ["Hello", ",", " ", "world", "!"]

    iex> Mojibake.words("can't stop 3.14")
    ["can't", " ", "stop", " ", "3.14"]

---

*Consult [api-reference.md](api-reference.md) for complete listing*
