Mojibake.Encoding (Mojibake v0.8.0)

Copy Markdown View Source

Unicode encodings: detection, validation, and conversion between UTF-8, UTF-16, and UTF-32.

An encoding is one of:

  • :ascii and :utf8
  • :utf16le, :utf16be, :utf32le, and :utf32be, with a fixed byte order. A leading U+FEFF is text, not a byte-order mark.
  • :utf16 and :utf32, input only: the binary starts with a byte-order mark (BOM), which sets the byte order and is not part of the text.

Errors report byte offsets into the input, like the rest of Mojibake.

Summary

Types

t()

An encoding. See the module documentation.

Functions

Convert binary from one encoding to another.

Guess the encoding of binary: the one its byte-order mark gives, else :ascii or :utf8 if it is well-formed in them. Returns nil otherwise, and for an empty binary.

Return true if binary is well-formed in encoding.

Check that binary is well-formed in encoding. Returns {:error, %Mojibake.Error{}} with the byte range of the first ill-formed subsequence otherwise.

Types

t()

@type t() ::
  :ascii | :utf8 | :utf16 | :utf16le | :utf16be | :utf32 | :utf32le | :utf32be

An encoding. See the module documentation.

Functions

convert(binary, from, to, opts \\ [])

@spec convert(binary(), t(), t(), keyword()) ::
  {:ok, binary()} | {:error, Mojibake.Error.t()}

Convert binary from one encoding to another.

The :malformed option says what to do with input that is not well-formed: :stop (the default) returns an error, :replace writes U+FFFD, and :skip drops it. Converting to :ascii returns {:error, %Mojibake.Error{reason: :unsupported}} for non-ASCII text.

Examples

iex> Mojibake.Encoding.convert("hé", :utf8, :utf16be)
{:ok, <<0, ?h, 0, 0xE9>>}

iex> Mojibake.Encoding.convert(<<0xFF, 0xFE, ?h, 0, 0xE9, 0>>, :utf16, :utf8)
{:ok, "hé"}

iex> Mojibake.Encoding.convert(<<?a, 0xFF>>, :utf8, :utf8, malformed: :replace)
{:ok, "a\uFFFD"}

convert!(binary, from, to, opts \\ [])

@spec convert!(binary(), t(), t(), keyword()) :: binary()

Like convert/4, but raises Mojibake.Error.

Examples

iex> Mojibake.Encoding.convert!(<<0, 0, 0, ?A>>, :utf32be, :utf8)
"A"

detect(binary)

@spec detect(binary()) :: t() | nil

Guess the encoding of binary: the one its byte-order mark gives, else :ascii or :utf8 if it is well-formed in them. Returns nil otherwise, and for an empty binary.

A BOM gives an explicit byte order, such as :utf16le; the BOM is still part of the binary. Convert it from :utf16 or :utf32 to drop the BOM.

Examples

iex> Mojibake.Encoding.detect("hello")
:ascii

iex> Mojibake.Encoding.detect("héllo")
:utf8

iex> Mojibake.Encoding.detect(<<0xFF, 0xFE, ?h, 0>>)
:utf16le

iex> Mojibake.Encoding.detect(<<0xFF>>)
nil

valid?(binary, encoding)

@spec valid?(binary(), t()) :: boolean()

Return true if binary is well-formed in encoding.

Examples

iex> Mojibake.Encoding.valid?(<<0xC3, 0xA9>>, :utf8)
true

iex> Mojibake.Encoding.valid?(<<0xC3, 0xA9>>, :ascii)
false

validate(binary, encoding)

@spec validate(binary(), t()) :: :ok | {:error, Mojibake.Error.t()}

Check that binary is well-formed in encoding. Returns {:error, %Mojibake.Error{}} with the byte range of the first ill-formed subsequence otherwise.

Examples

iex> Mojibake.Encoding.validate("héllo", :utf8)
:ok

iex> Mojibake.Encoding.validate(<<?a, 0, 0x00, 0xDC>>, :utf16le)
{:error, %Mojibake.Error{reason: :malformed_input, offset: 2, length: 2, detail: :unpaired_surrogate}}