Data.Text.Utf8

A Unicode code point.

A UTF-8 code unit is a byte. A Unicode code point can be encoded as up to four code units.

An index into the raw UTF-8 data of a Text. This is not the code point index as conventionally accepted by Text, so we wrap it to avoid confusing the two. Incorrect index manipulation can lead to surrogate pairs being sliced, so manipulate indices with care. This type is also used for lengths.

A space efficient, packed, unboxed Unicode text type.

Return whether text has exactly one case variation, such that this function will not return true when Aho–Corasick would differentiate when doing case-insensitive matching.

The return value of this function is not really an index. However the signature is supposed to make it clear that the length is returned in terms of code units, not code points.

Lower-Case a UTF-8 codepoint. Uses toLowerAscii for ASCII and toLower otherwise.

Inverse of Char.toLower/Utf8.lowerCodePoint

Returns all the characters that have the given character as their lower case, for example:

unlowerCodePoint a == "aA" unlowerCodePoint A == "" unlowerCodePoint '1' == "1" unlowerCodePoint i == İiI unlowerCodePoint ß == ẞß

Lowercase a Text by applying lowerCodePoint to each Char.

Lower-case the ASCII code points A-Z and leave the rest of ASCII intact.

Convert a Unicode Code Point c into a list of UTF-8 code units (bytes).

Decode 2 UTF-8 code units into their code point. The given code units should have the following format:

┌───────────────┬───────────────┐
│1 1 0 x x x x x│1 0 x x x x x x│
└───────────────┴───────────────┘

Decode 3 UTF-8 code units into their code point. The given code units should have the following format:

┌───────────────┬───────────────┬───────────────┐
│1 1 1 0 x x x x│1 0 x x x x x x│1 0 x x x x x x│
└───────────────┴───────────────┴───────────────┘

Decode 4 UTF-8 code units into their code point. The given code units should have the following format:

┌───────────────┬───────────────┬───────────────┬───────────────┐
│1 1 1 1 0 x x x│1 0 x x x x x x│1 0 x x x x x x│1 0 x x x x x x│
└───────────────┴───────────────┴───────────────┴───────────────┘

Decode a list of UTF-8 code units into a list of code points.

Get the code unit at the given CodeUnitIndex. Performs bounds checking.

Does exactly the same thing as unsafeIndexCodePoint', but on Text values.

Scan backwards through the text until we've seen the specified number of codepoints. Assumes that the initial CodeUnitIndex is within a codepoint.

See Data.Primitive.byteArrayContents.

See Data.Primitive.isByteArrayPinned.

Decode a code point at the given CodeUnitIndex. Returns garbage if there is no valid code point at that position. Does not perform bounds checking. See decode2, decode3 and decode4 for the expected format of multi-byte code points.

Intermediate state when you're iterating backwards through a Utf8 text.

Similar to unsafeIndexCodePoint', but assumes that the given index is the end of a utf8 codepoint. It returns the decoded code point and the index _before_ the code point. The resulting index could be passed directly to unsafeIndexEndOfCodePoint' again to decode the _previous_ code point.

O(n) Concatenate a list of Texts.

O(n) dropWhile p t returns the suffix remaining after takeWhile p t.

O(n+m) The isInfixOf function takes two Texts and returns True if and only if the first is contained, wholly and intact, anywhere within the second.

In (unlikely) bad cases, this function's time complexity degrades towards O(n*m).

O(1) Tests whether a Text is empty or not.

O(n) Convert a String into a Text. Performs replacement on invalid scalar values, so unpack . pack is not id:

>>> Data.Text.unpack (pack "\55555")"\65533"

O(n*m) replicate n t is a Text consisting of the input t repeated n times.

O(n) Convert a Text into a String.

O(n+m) Find the offsets of all non-overlapping indices of needle within haystack.

In (unlikely) bad cases, this algorithm's complexity degrades towards O(n*m).