UTF8

is a simple UTF-8 package for Carp. This nascent string replacement type allows you to use many of the functions you know from Carp strings while respecting unicode runes instead of just having bytes.

Installation

You can obtain this library like so:

(load "git@github.com:carpentry-org/utf8.carp@0.2.0")

Usage

First, let’s define a UTF-8 string to work with!

(let [s (UTF8.from-string "hεllö")]

That’s a cute, short string! Hm, I wonder how long it is!

(length s) ; => 5

That’s surprising! So the length is the actual number of runes, and not the number of bytes, you say? Most curious!

You know what, I want to see this second character there up close! It somehow looks all Greek to me!

(nth s 1) ; => ε

Hm, so this is what that looks like, huh? Interesting. And what’s its type?

(type UTF8.nth) ; => UTF8.nth : (λ [(Ref UTF8), Int] Rune)

So, it’s called a Rune, huh? Hm, they don’t seem to be super interesting, but I seem to be able to compare them and stringify them, and even take their length in bytes! Quite delicious!

I wonder what else I can do with these functions?

=

defn

(Fn [(Ref UTF8 a), (Ref UTF8 a)] Bool) 

                        (= a b)

append

defn

(Fn [(Ref UTF8 a), (Ref UTF8 b)] UTF8) 

                        (append a b)

appends two UTF-8-encoded strings.

copy

instantiate

(Fn [(Ref UTF8 a)] UTF8)

copies a UTF8.

decode-at

defn

(Fn [(Ref (Array Byte) a), Int] (Maybe (Pair Int Int))) 

                        (decode-at bytes pos)

decodes the single UTF-8 code point beginning at byte offset pos in bytes, validating it per RFC 3629 / Unicode Table 3-7: it rejects overlong encodings, UTF-16 surrogate code points (U+D800..U+DFFF), code points beyond U+10FFFF, and out-of-range continuation bytes. Returns (Just (Pair codepoint width)) — the scalar code-point value and its byte width (1..4) — or (Nothing) at end of input, on truncation, or on malformed input.

delete

instantiate

(Fn [UTF8] ())

deletes a UTF8. Should usually not be called manually.

empty?

defn

(Fn [(Ref UTF8 a)] Bool) 

                        (empty? u)

checks whether a UTF-8-encoded string is empty.

ends-with?

defn

(Fn [(Ref UTF8 a), (Ref UTF8 b)] Bool) 

                        (ends-with? u sub)

checks if the string u ends with the string sub.

from-bytes

defn

(Fn [(Ref (Array Byte) a)] (Maybe UTF8)) 

                        (from-bytes bytes)

creates a UTF-8 string from raw bytes, first checking with valid? that they are well-formed UTF-8. Returns Nothing when the bytes are not valid UTF-8, so untrusted input can be turned into a UTF8 safely — unlike from-string, which assumes well-formed data.

from-string

defn

(Fn [(Ref String a)] UTF8) 

                        (from-string s)

creates an UTF-8 string from a regular string.

init

instantiate

(Fn [(Array Rune)] UTF8)

creates a UTF8.

length

defn

(Fn [(Ref UTF8 a)] Int) 

                        (length u)

returns the length of a UTF-8-encoded string.

nth

defn

(Fn [(Ref UTF8 a), Int] (Maybe Rune)) 

                        (nth u n)

returns the nth rune from a UTF-8-encoded string.

prefix

defn

(Fn [(Ref UTF8 a), Int] UTF8) 

                        (prefix u n)

returns the first n characters of the string u.

prn

instantiate

(Fn [(Ref UTF8 a)] String)

converts a UTF8 to a string.

reverse

defn

(Fn [(Ref UTF8 a)] UTF8) 

                        (reverse u)

reverses a UTF-8-encoded string.

runes

instantiate

(Fn [(Ref UTF8 a)] (Ref (Array Rune) a))

gets the runes property of a UTF8.

set-runes

instantiate

(Fn [UTF8, (Array Rune)] UTF8)

sets the runes property of a UTF8.

set-runes!

instantiate

(Fn [(Ref UTF8 a), (Array Rune)] ())

sets the runes property of a UTF8 in place.

slice

defn

(Fn [(Ref UTF8 a), Int, Int] UTF8) 

                        (slice u a b)

returns a substring of the string from the index a to the index b.

starts-with?

defn

(Fn [(Ref UTF8 a), (Ref UTF8 b)] Bool) 

                        (starts-with? u sub)

checks if the string u begins with the string sub.

str

defn

(Fn [(Ref UTF8 a)] String) 

                        (str u)

creates a regular string from a UTF-8 string

suffix

defn

(Fn [(Ref UTF8 a), Int] UTF8) 

                        (suffix u n)

returns the runes of the string u from index n to the end, i.e. u with its first n runes removed. Mirrors Array.suffix and String.suffix: n is a starting index, not a count taken from the end.

unsafe-nth

defn

(Fn [(Ref UTF8 a), Int] Rune) 

                        (unsafe-nth u n)

returns the nth rune from a UTF-8-encoded string unsafely.

update-runes

instantiate

(Fn [UTF8, (Ref (Fn [(Array Rune)] (Array Rune) a) b)] UTF8)

updates the runes property of a UTF8 using a function f.

valid?

defn

(Fn [(Ref (Array Byte) a)] Bool) 

                        (valid? bytes)

checks whether bytes is well-formed UTF-8 per RFC 3629, rejecting overlong encodings, UTF-16 surrogate code points (U+D800..U+DFFF), and code points beyond U+10FFFF. Useful to guard untrusted input before treating it as UTF-8, since from-string and String.from-bytes assume well-formed data.

zero

defn

(Fn [] UTF8) 

                        (zero)

Returns the empty string.