HORIZON HASKELLDocslts/ghc-9.14.x45dd5372026-10-11Search names, modules, packages, or :: a typeCtrl K

GHC 9.14.1 · lts/ghc-9.14.x · 45dd537 · 2026-10-11

Modulestreamly-core-0.3.1Haskell2010

Streamly.Internal.Unicode.Stream

  • 4 types
  • 44 values

Setup

0 declarations

To execute the code examples provided in this module in ghci, please run the following commands first.

Example1 expression
:m
Example3 expressions
import qualified Streamly.Data.Fold as Foldimport qualified Streamly.Data.Stream as Streamimport qualified Streamly.Unicode.Stream as Unicode

For APIs that have not been released yet.

Example2 expressions
:set -XMagicHashimport qualified Streamly.Internal.Unicode.Stream as Unicode

Construction (Decoding)

1 declaration

UTF-8 Byte Stream Decoding

valuedecodeUtf8 :: Monad m => Stream m Word8 -> Stream m Char
#

Decode a UTF-8 encoded bytestream to a stream of Unicode characters. Any invalid codepoint encountered is replaced with the unicode replacement character.

valuedecodeUtf8' :: Monad m => Stream m Word8 -> Stream m Char
#

Decode a UTF-8 encoded bytestream to a stream of Unicode characters. The function throws an error if an invalid codepoint is encountered.

UTF-16 Byte Stream Decoding

valuedecodeUtf16le :: Monad m => Stream m Word16 -> Stream m Char
#

Decode a UTF-16 encoded stream to a stream of Unicode characters. Any invalid codepoint encountered is replaced with the unicode replacement character.

The Word16s are expected to be in the little-endian byte order.

Resumable UTF-8 Byte Stream Decoding

UTF-8 Array Stream Decoding

Elimination (Encoding)

0 declarations

Latin1 Encoding to Byte Stream

valueencodeLatin1' :: Monad m => Stream m Char -> Stream m Word8
#

Encode a stream of Unicode characters to bytes by mapping each character to a byte in 0-255 range. Throws an error if the input stream contains characters beyond 255.

UTF-8 Encoding to Byte Stream

valueencodeUtf8 :: Monad m => Stream m Char -> Stream m Word8
#

Encode a stream of Unicode characters to a UTF-8 encoded bytestream. Any Invalid characters (U+D800-U+D8FF) in the input stream are replaced by the Unicode replacement character U+FFFD.

valueencodeUtf8' :: Monad m => Stream m Char -> Stream m Word8
#

Encode a stream of Unicode characters to a UTF-8 encoded bytestream. When any invalid character (U+D800-U+D8FF) is encountered in the input stream the function errors out.

valueencodeUtf8_ :: Monad m => Stream m Char -> Stream m Word8
#

Encode a stream of Unicode characters to a UTF-8 encoded bytestream. Any Invalid characters (U+D800-U+D8FF) in the input stream are dropped.

UTF-8 Encoding to Chunk Stream

UTF-16 Encoding to Byte Stream

valueencodeUtf16le :: Monad m => Stream m Char -> Stream m Word16
#

Encode a stream of Unicode characters to a UTF-16 encoded stream. Any invalid characters in the input stream are replaced by the Unicode replacement character U+FFFD.

The resulting Word16s are encoded in little-endian byte order.

Transformation

5 declarations
valuestripHead :: Monad m => Stream m Char -> Stream m Char
#

Remove leading whitespace from a string.

Example1 expression
stripHead = Stream.dropWhile Char.isSpace

Pre-release

valuelines :: Monad m => Fold m Char b -> Stream m Char -> Stream m b
#

Fold each line of the stream using the supplied Fold and stream the result.

Definition:

Example1 expression
lines f = Stream.foldMany (Fold.takeEndBy_ (== '\n') f)

Usage:

Example1 expression
Stream.toList $ Unicode.lines Fold.toList (Stream.fromList "line1\nline2\nline3\n\n\n")["line1","line2","line3","",""]

Pre-release

valuewords :: Monad m => Fold m Char b -> Stream m Char -> Stream m b
#

Fold each word of the stream using the supplied Fold.

Definition:

Example1 expression
words = Stream.wordsBy Char.isSpace

Usage:

Example1 expression
Stream.toList $ Unicode.words Fold.toList (Stream.fromList " ab  cd   ef ")["ab","cd","ef"]

Pre-release

valueunlines :: MonadIO m => Unfold m a Char -> Stream m a -> Stream m Char
#

Unfold a stream to character streams using the supplied Unfold and concat the results suffixing a newline character \n to each stream.

Definition:

Example2 expressions
unlines = Stream.unfoldEachEndBy '\n'unlines = Stream.unfoldEachEndBySeq "\n" Unfold.fromList

Pre-release

valueunwords :: MonadIO m => Unfold m a Char -> Stream m a -> Stream m Char
#

Unfold the elements of a stream to character streams using the supplied Unfold and concat the results with a whitespace character infixed between the streams.

Example2 expressions
unwords = Stream.unfoldEachSepBy ' 'unwords = Stream.unfoldEachSepBySeq " " Unfold.fromList

Pre-release

StreamD UTF8 Encoding / Decoding transformations.

8 declarations

Decoding String Literals

1 declaration
valuefromStr# :: MonadIO m => Addr# -> Stream m Char
#

Read UTF-8 encoded bytes as chars from an Addr# until a 0 byte is encountered, the 0 byte is not included in the stream.

Unsafe: The caller is responsible for safe addressing.

Note that this is completely safe when reading from Haskell string literals because they are guaranteed to be NULL terminated:

Example1 expression
Stream.fold Fold.toList (Unicode.fromStr# "Haskell"#)"Haskell"

Word16 Utilities

2 declarations
valuemkEvenW8Chunks
  1. :: Monad m
  2. => Stream m (Array Word8)
  3. -> Stream m (Array Word8)
#

Ensure chunks of even length. This can be used before casting the arrays to Word16. Use this API when interacting with external data.

The chunks are split and merged accordingly to create arrays of even length. If the sum of length of all the arrays in the stream is odd then the trailing byte of the last array is dropped.

valueswapByteOrder :: Word16 -> Word16
#

Swap the byte order of Word16

swapByteOrder 0xABCD == 0xCDAB
swapByteOrder . swapByteOrder == id

Deprecations

3 declarations