HORIZON HASKELLDocslts/ghc-10.0.x8fef28d2026-11-02Search packagesCtrl K
Theme: |

GHC 10.0.0.20260917 · lts/ghc-10.0.x · 8fef28d · 2026-11-02

Modulestreamly-core-0.3.1Haskell2010

Streamly.Internal.Unicode.Stream

Contents
  1. Setup
  2. Construction (Decoding)
  3. UTF-8 Byte Stream Decoding
  4. UTF-16 Byte Stream Decoding
  5. Resumable UTF-8 Byte Stream Decoding
  6. UTF-8 Array Stream Decoding
  7. Elimination (Encoding)
  8. Latin1 Encoding to Byte Stream
  9. UTF-8 Encoding to Byte Stream
  10. UTF-8 Encoding to Chunk Stream
  11. UTF-16 Encoding to Byte Stream
  12. Transformation
  13. StreamD UTF8 Encoding / Decoding transformations.
  14. Decoding String Literals
  15. Word16 Utilities
  16. Deprecations
Synopsis

Setup

To execute the code examples provided in this module in ghci, please run the following commands first.

Example1 expression
:m
Example3 expressions
import qualified Streamly.Data.Fold as Foldimport qualified Streamly.Data.Stream as Streamimport qualified Streamly.Unicode.Stream as Unicode

For APIs that have not been released yet.

Example2 expressions
:set -XMagicHashimport qualified Streamly.Internal.Unicode.Stream as Unicode

Construction (Decoding)

UTF-8 Byte Stream Decoding

valuedecodeUtf8 :: Monad m => Stream m Word8 -> Stream m Char
Source#

Decode a UTF-8 encoded bytestream to a stream of Unicode characters. Any invalid codepoint encountered is replaced with the unicode replacement character.

UTF-16 Byte Stream Decoding

valuedecodeUtf16le :: Monad m => Stream m Word16 -> Stream m Char
Source#

Decode a UTF-16 encoded stream to a stream of Unicode characters. Any invalid codepoint encountered is replaced with the unicode replacement character.

The Word16s are expected to be in the little-endian byte order.

Resumable UTF-8 Byte Stream Decoding

UTF-8 Array Stream Decoding

Elimination (Encoding)

Latin1 Encoding to Byte Stream

valueencodeLatin1' :: Monad m => Stream m Char -> Stream m Word8
Source#

Encode a stream of Unicode characters to bytes by mapping each character to a byte in 0-255 range. Throws an error if the input stream contains characters beyond 255.

UTF-8 Encoding to Byte Stream

valueencodeUtf8 :: Monad m => Stream m Char -> Stream m Word8
Source#

Encode a stream of Unicode characters to a UTF-8 encoded bytestream. Any Invalid characters (U+D800-U+D8FF) in the input stream are replaced by the Unicode replacement character U+FFFD.

valueencodeUtf8' :: Monad m => Stream m Char -> Stream m Word8
Source#

Encode a stream of Unicode characters to a UTF-8 encoded bytestream. When any invalid character (U+D800-U+D8FF) is encountered in the input stream the function errors out.

UTF-8 Encoding to Chunk Stream

UTF-16 Encoding to Byte Stream

valueencodeUtf16le :: Monad m => Stream m Char -> Stream m Word16
Source#

Encode a stream of Unicode characters to a UTF-16 encoded stream. Any invalid characters in the input stream are replaced by the Unicode replacement character U+FFFD.

The resulting Word16s are encoded in little-endian byte order.

Transformation

valuelines :: Monad m => Fold m Char b -> Stream m Char -> Stream m b
Source#

Fold each line of the stream using the supplied Fold and stream the result.

Definition:

Example1 expression
lines f = Stream.foldMany (Fold.takeEndBy_ (== '\n') f)

Usage:

Example1 expression
Stream.toList $ Unicode.lines Fold.toList (Stream.fromList "line1\nline2\nline3\n\n\n")["line1","line2","line3","",""]

Pre-release

valuewords :: Monad m => Fold m Char b -> Stream m Char -> Stream m b
Source#

Fold each word of the stream using the supplied Fold.

Definition:

Example1 expression
words = Stream.wordsBy Char.isSpace

Usage:

Example1 expression
Stream.toList $ Unicode.words Fold.toList (Stream.fromList " ab  cd   ef ")["ab","cd","ef"]

Pre-release

valueunlines :: MonadIO m => Unfold m a Char -> Stream m a -> Stream m Char
Source#

Unfold a stream to character streams using the supplied Unfold and concat the results suffixing a newline character \n to each stream.

Definition:

Example2 expressions
unlines = Stream.unfoldEachEndBy '\n'unlines = Stream.unfoldEachEndBySeq "\n" Unfold.fromList

Pre-release

valueunwords :: MonadIO m => Unfold m a Char -> Stream m a -> Stream m Char
Source#

Unfold the elements of a stream to character streams using the supplied Unfold and concat the results with a whitespace character infixed between the streams.

Example2 expressions
unwords = Stream.unfoldEachSepBy ' 'unwords = Stream.unfoldEachSepBySeq " " Unfold.fromList

Pre-release

StreamD UTF8 Encoding / Decoding transformations.

Decoding String Literals

valuefromStr# :: MonadIO m => Addr# -> Stream m Char
Source#

Read UTF-8 encoded bytes as chars from an Addr# until a 0 byte is encountered, the 0 byte is not included in the stream.

Unsafe: The caller is responsible for safe addressing.

Note that this is completely safe when reading from Haskell string literals because they are guaranteed to be NULL terminated:

Example1 expression
Stream.fold Fold.toList (Unicode.fromStr# "Haskell"#)"Haskell"

Word16 Utilities

valuemkEvenW8Chunks
  1. :: Monad m
  2. => Stream m (Array Word8)
  3. -> Stream m (Array Word8)
Source#

Ensure chunks of even length. This can be used before casting the arrays to Word16. Use this API when interacting with external data.

The chunks are split and merged accordingly to create arrays of even length. If the sum of length of all the arrays in the stream is odd then the trailing byte of the last array is dropped.

Deprecations