Mojibake.Bidi (Mojibake v0.8.0)

Copy Markdown View Source

The Unicode Bidirectional Algorithm (UAX #9), which lays out text that mixes left-to-right scripts, such as Latin, with right-to-left ones, such as Hebrew and Arabic.

Text is stored in logical order, the order it is typed and read in. resolve/2 computes the embedding level of every character of a paragraph: even levels are left to right, odd levels right to left. A line of the paragraph is then reordered for display with reorder/2, split into directional runs with runs/2, or rendered in visual order with visual/2:

iex> paragraph = Mojibake.Bidi.resolve!("abc אבג")
iex> paragraph.direction
:ltr
iex> Mojibake.Bidi.levels(paragraph)
[0, 0, 0, 0, 1, 1, 1]
iex> Mojibake.Bidi.visual(paragraph)
"abc גבא"

Positions and lines

Positions are indices into the characters of the paragraph, 0 to paragraph.length - 1, not codepoint indices of the text: rule X9 removes the embedding and override controls and the Boundary Neutrals, such as U+200D ZERO WIDTH JOINER, from the paragraph. chars/1 gives the byte offset of each character in the text.

The functions that take a line take a range of positions with step 1, such as 0..9, and default to the whole paragraph. Break a paragraph into lines yourself, with Mojibake.line_breaks/1 and the width of your output, and lay out each line separately.

A paragraph is one paragraph: split text at paragraph separators, such as "\n", first. Resolving it is the expensive step; reordering a line of it is cheap, and the resolved paragraph can be shared between processes.

Summary

Types

A character of a resolved paragraph

The base direction of a paragraph. :auto takes it from the first strong character (rules P2 and P3), and is left to right if there is none.

A run of characters with the same embedding level on a reordered line

Functions

Return the characters of paragraph, in logical order. See char_info/0.

Return the paragraph direction of text: the direction of its first strong character, or :ltr if it has none (rules P2 and P3). Raises Mojibake.Error if text is not well-formed UTF-8.

Return the resolved embedding level of each character of paragraph, in logical order.

Reorder a line of paragraph for display (rules L1 and L2). Returns the positions of the line in visual order, from left to right.

Resolve the embedding levels of text, one paragraph, with base direction direction.

Split a reordered line of paragraph into runs of the same embedding level, from left to right. See run/0.

Return a line in visual order, with mirrored characters, such as parentheses in right-to-left text, replaced by their mirroring glyphs (rule L4). The controls that rule X9 removes are left out.

Types

char_info()

@type char_info() :: %{
  codepoint: non_neg_integer(),
  offset: non_neg_integer(),
  level: non_neg_integer(),
  class: Mojibake.Char.bidi_class(),
  mirror: non_neg_integer() | nil
}

A character of a resolved paragraph:

  • codepoint - the character
  • offset - its byte offset in the paragraph text
  • level - its resolved embedding level; odd levels are right to left
  • class - its class after the weak and neutral rules (W1-W7, N1-N2) resolved it, an atom of Mojibake.Char.bidi_class/0. Mojibake.Char.bidi_class/1 gives its original class.
  • mirror - the Bidi_Mirroring_Glyph to display instead of the character (rule L4), when it is mirrored and its level is odd, else nil

direction()

@type direction() :: :ltr | :rtl | :auto

The base direction of a paragraph. :auto takes it from the first strong character (rules P2 and P3), and is left to right if there is none.

run()

@type run() :: %{
  visual: Range.t(),
  logical: Range.t(),
  level: non_neg_integer(),
  direction: :ltr | :rtl
}

A run of characters with the same embedding level on a reordered line:

  • visual - the positions of the run in the visual order, indices into the list that reorder/2 returns for the same line
  • logical - the paragraph positions the run covers, in logical order
  • level - the embedding level of the run
  • direction - :rtl for odd levels, :ltr otherwise

Functions

chars(paragraph)

@spec chars(Mojibake.Bidi.Paragraph.t()) :: [char_info()]

Return the characters of paragraph, in logical order. See char_info/0.

Examples

iex> "(א)" |> Mojibake.Bidi.resolve!() |> Mojibake.Bidi.chars() |> hd()
%{codepoint: ?(, offset: 0, level: 1, class: :right_to_left, mirror: ?)}

direction(text)

@spec direction(String.t()) :: :ltr | :rtl

Return the paragraph direction of text: the direction of its first strong character, or :ltr if it has none (rules P2 and P3). Raises Mojibake.Error if text is not well-formed UTF-8.

Examples

iex> Mojibake.Bidi.direction("Hello")
:ltr

iex> Mojibake.Bidi.direction("123 שלום")
:rtl

iex> Mojibake.Bidi.direction("123")
:ltr

levels(paragraph)

Return the resolved embedding level of each character of paragraph, in logical order.

Examples

iex> "ab אב 12" |> Mojibake.Bidi.resolve!() |> Mojibake.Bidi.levels()
[0, 0, 0, 1, 1, 1, 2, 2]

reorder(paragraph, line \\ nil)

@spec reorder(Mojibake.Bidi.Paragraph.t(), Range.t() | nil) :: [non_neg_integer()]

Reorder a line of paragraph for display (rules L1 and L2). Returns the positions of the line in visual order, from left to right.

Raises ArgumentError if line is not a range of positions of the paragraph.

Examples

iex> paragraph = Mojibake.Bidi.resolve!("abc אבג")
iex> Mojibake.Bidi.reorder(paragraph)
[0, 1, 2, 3, 6, 5, 4]
iex> Mojibake.Bidi.reorder(paragraph, 4..6)
[6, 5, 4]

resolve(text, direction \\ :auto)

@spec resolve(String.t(), direction()) ::
  {:ok, Mojibake.Bidi.Paragraph.t()} | {:error, Mojibake.Error.t()}

Resolve the embedding levels of text, one paragraph, with base direction direction.

Returns {:error, %Mojibake.Error{}} if text is not well-formed UTF-8.

Examples

iex> {:ok, paragraph} = Mojibake.Bidi.resolve("שלום world")
iex> {paragraph.direction, paragraph.level, paragraph.length}
{:rtl, 1, 10}

iex> {:ok, paragraph} = Mojibake.Bidi.resolve("שלום world", :ltr)
iex> paragraph.direction
:ltr

iex> Mojibake.Bidi.resolve(<<?a, 0xFF>>)
{:error, %Mojibake.Error{reason: :malformed_input, offset: 1, length: 1, detail: :invalid_leading_byte}}

resolve!(text, direction \\ :auto)

@spec resolve!(String.t(), direction()) :: Mojibake.Bidi.Paragraph.t()

Like resolve/2, but raises Mojibake.Error.

Examples

iex> Mojibake.Bidi.resolve!("مرحبا").direction
:rtl

runs(paragraph, line \\ nil)

@spec runs(Mojibake.Bidi.Paragraph.t(), Range.t() | nil) :: [run()]

Split a reordered line of paragraph into runs of the same embedding level, from left to right. See run/0.

Raises ArgumentError if line is not a range of positions of the paragraph.

Examples

iex> "abc אבג" |> Mojibake.Bidi.resolve!() |> Mojibake.Bidi.runs()
[
  %{visual: 0..3, logical: 0..3, level: 0, direction: :ltr},
  %{visual: 4..6, logical: 4..6, level: 1, direction: :rtl}
]

visual(paragraph_or_text, line_or_direction \\ nil)

@spec visual(Mojibake.Bidi.Paragraph.t(), Range.t() | nil) :: String.t()
@spec visual(String.t(), direction()) :: String.t()

Return a line in visual order, with mirrored characters, such as parentheses in right-to-left text, replaced by their mirroring glyphs (rule L4). The controls that rule X9 removes are left out.

Takes a resolved paragraph and an optional line, or a text and an optional base direction. Visual order is for display on a device that does not do bidirectional layout itself, such as a terminal; do not store or process text in it.

Raises ArgumentError if line is not a range of positions of the paragraph, and Mojibake.Error if text is not well-formed UTF-8.

Examples

iex> Mojibake.Bidi.visual("abc אבג")
"abc גבא"

iex> Mojibake.Bidi.visual("abc (אבג)", :rtl)
"(גבא) abc"

iex> paragraph = Mojibake.Bidi.resolve!("abc אבג")
iex> Mojibake.Bidi.visual(paragraph, 2..5)
"c בא"