Unicode encodings: detection, validation, and conversion between UTF-8, UTF-16, and UTF-32.
An encoding is one of:
:asciiand:utf8:utf16le,:utf16be,:utf32le, and:utf32be, with a fixed byte order. A leading U+FEFF is text, not a byte-order mark.:utf16and:utf32, input only: the binary starts with a byte-order mark (BOM), which sets the byte order and is not part of the text.
Errors report byte offsets into the input, like the rest of Mojibake.
Summary
Functions
Convert binary from one encoding to another.
Like convert/4, but raises Mojibake.Error.
Guess the encoding of binary: the one its byte-order mark gives, else :ascii or :utf8 if
it is well-formed in them. Returns nil otherwise, and for an empty binary.
Return true if binary is well-formed in encoding.
Check that binary is well-formed in encoding. Returns {:error, %Mojibake.Error{}} with
the byte range of the first ill-formed subsequence otherwise.
Types
Functions
Convert binary from one encoding to another.
The :malformed option says what to do with input that is not well-formed: :stop (the
default) returns an error, :replace writes U+FFFD, and :skip drops it. Converting to
:ascii returns {:error, %Mojibake.Error{reason: :unsupported}} for non-ASCII text.
Examples
iex> Mojibake.Encoding.convert("hé", :utf8, :utf16be)
{:ok, <<0, ?h, 0, 0xE9>>}
iex> Mojibake.Encoding.convert(<<0xFF, 0xFE, ?h, 0, 0xE9, 0>>, :utf16, :utf8)
{:ok, "hé"}
iex> Mojibake.Encoding.convert(<<?a, 0xFF>>, :utf8, :utf8, malformed: :replace)
{:ok, "a\uFFFD"}
Like convert/4, but raises Mojibake.Error.
Examples
iex> Mojibake.Encoding.convert!(<<0, 0, 0, ?A>>, :utf32be, :utf8)
"A"
Guess the encoding of binary: the one its byte-order mark gives, else :ascii or :utf8 if
it is well-formed in them. Returns nil otherwise, and for an empty binary.
A BOM gives an explicit byte order, such as :utf16le; the BOM is still part of the binary.
Convert it from :utf16 or :utf32 to drop the BOM.
Examples
iex> Mojibake.Encoding.detect("hello")
:ascii
iex> Mojibake.Encoding.detect("héllo")
:utf8
iex> Mojibake.Encoding.detect(<<0xFF, 0xFE, ?h, 0>>)
:utf16le
iex> Mojibake.Encoding.detect(<<0xFF>>)
nil
Return true if binary is well-formed in encoding.
Examples
iex> Mojibake.Encoding.valid?(<<0xC3, 0xA9>>, :utf8)
true
iex> Mojibake.Encoding.valid?(<<0xC3, 0xA9>>, :ascii)
false
@spec validate(binary(), t()) :: :ok | {:error, Mojibake.Error.t()}
Check that binary is well-formed in encoding. Returns {:error, %Mojibake.Error{}} with
the byte range of the first ill-formed subsequence otherwise.
Examples
iex> Mojibake.Encoding.validate("héllo", :utf8)
:ok
iex> Mojibake.Encoding.validate(<<?a, 0, 0x00, 0xDC>>, :utf16le)
{:error, %Mojibake.Error{reason: :malformed_input, offset: 2, length: 2, detail: :unpaired_surrogate}}