Strings and Byte Buffers

How to work with strings, byte arrays, and buffer operations in Esper IR.

This page describes the current SSA builder and backend implementation. Its unversioned esp examples are compatibility snapshots, still accepted by the reader. Start with the new text walkthrough for esper 1 syntax and the roadmap for the library-first migration boundary.

Esper has no string type. Strings are byte sequences in linear memory, managed entirely by the frontend. This guide shows how a compiler lowers common string operations into Esper IR.

What we’re lowering

Consider a language with built-in string operations:

func greet(name: String) -> String {
    let greeting = "Hello, "
    return greeting + name
}

func countChars(s: String) -> Int32 {
    var count = 0
    for ch in s {
        count += 1
    }
    return count
}

At the source level, strings are opaque values with high-level operations like concatenation and iteration. The frontend must decide a concrete memory representation and lower every operation to pointer arithmetic, byte loads, and loops. Esper provides the building blocks; the frontend makes the design choices.

Choosing a representation

Before emitting any IR, the frontend must pick a string representation. The two most common are:

RepresentationLayoutProsCons
Null-terminated (C)ptr to bytes ending in 0x00Simple, interop with CLength requires a scan, no embedded nulls
Length-prefixed (Rust, Go)(ptr data, i32 len)O(1) length, embedded nulls ok8+ extra bytes per string

The rest of this guide shows both, starting with null-terminated strings since they are simpler to lower.

C-style null-terminated strings

Pseudo-code: string length

func strlen(s: *UInt8) -> Int32 {
    var len = 0
    while s[len] != 0 {
        len += 1
    }
    return len
}

Step-by-step lowering

  1. Parameters: s is a ptr to the first byte. We allocate space for a mutable counter len as i32.
  2. The loop condition (s[len] != 0): Compute the address s + len with add i64, load a single byte with load8.u (unsigned so 0x00-0xFF maps to 0-255), then test with eqz.
  3. Why load8.u? Characters are unsigned byte values. Using load8.s would sign-extend bytes above 0x7F to negative integers, which would break the zero check for valid UTF-8 continuation bytes.
  4. Incrementing: Plain add i32 on the counter, then jump back to the check.

Esper lowering

func $strlen(%0 ptr) -> i32 {
entry:
    %1 = ldi i32 4
    %2 = salloc %1             // allocate space for len
    %3 = ldi i32 0
    store i32 %2, %3           // len = 0
    jmp check

check:
    %6 = load i32 %2           // load len
    %7 = add i64 %0, %6       // s + len
    %8 = load8.u i32 %7       // load byte (zero-extended)
    %9 = eqz i32 %8           // is it the null terminator?
    jmp.if %9 done next

next:
    %11 = load i32 %2         // load len
    %12 = ldi i32 1
    %13 = add i32 %11, %12    // len + 1
    store i32 %2, %13         // len = len + 1
    jmp check

done:
    %16 = load i32 %2         // load len
    ret i32 %16
}

Pseudo-code: byte comparison

func memcmp(a: *UInt8, b: *UInt8, len: Int32) -> Int32 {
    for i in 0..<len {
        if a[i] != b[i] {
            return a[i] - b[i]
        }
    }
    return 0
}

Step-by-step lowering

  1. Loop structure: A counter i walks from 0 to len. Each iteration loads one byte from each buffer and compares them.
  2. Address computation: For each buffer, the byte address is base + i. Both use add i64 because pointer arithmetic operates at pointer width.
  3. Early exit: We subtract the two bytes. If the result is nonzero, we return it immediately (positive means a > b, negative means a < b). If zero, we advance to the next byte.
  4. Why sub i32 for the comparison? The difference of two unsigned bytes (0-255) always fits in a signed i32, and the sign of the result tells the caller which buffer is “greater.” This matches the C memcmp contract.

Esper lowering

func $memcmp(%0 ptr, %1 ptr, %2 i32) -> i32 {
entry:
    %3 = ldi i32 4
    %4 = salloc %3             // allocate space for i
    %5 = ldi i32 0
    store i32 %4, %5           // i = 0
    jmp check

check:
    %8 = load i32 %4           // load i
    %9 = ge.u i32 %8, %2      // i >= length?
    jmp.if %9 equal compare

compare:
    // Load byte from first buffer
    %11 = load i32 %4         // load i
    %12 = add i64 %0, %11     // a + i
    %13 = load8.u i32 %12     // a[i]

    // Load byte from second buffer
    %14 = load i32 %4         // load i
    %15 = add i64 %1, %14     // b + i
    %16 = load8.u i32 %15     // b[i]

    // Compare
    %17 = sub i32 %13, %16    // a[i] - b[i]
    %18 = eqz i32 %17         // is zero?
    jmp.if %18 advance diff

diff:
    ret i32 %17               // return nonzero difference

advance:
    %21 = load i32 %4         // load i
    %22 = ldi i32 1
    %23 = add i32 %21, %22    // i + 1
    store i32 %4, %23         // i = i + 1
    jmp check

equal:
    %26 = ldi i32 0
    ret i32 %26
}

Length-prefixed strings

Pseudo-code

struct StringView {
    var data: *UInt8    // offset 0, size 8 (ptr)
    var length: Int32   // offset 8, size 4
}

func stringLen(s: *StringView) -> Int32 {
    return s.length
}

func charAt(s: *StringView, i: Int32) -> UInt8 {
    return s.data[i]
}

Step-by-step lowering

  1. Struct layout: StringView is 12 bytes. data is a ptr at offset 0, length is an i32 at offset 8. The frontend computes these offsets at compile time.
  2. Length access (s.length): Load the parameter (a pointer to the struct), add offset 8, then load i32. This is O(1), unlike the null-terminated strlen which requires a loop.
  3. Character access (s.data[i]): First load the data pointer from offset 0 with load ptr, then add the index i, then load a byte with load8.u. This is two loads: one to dereference the struct, one to dereference the data pointer.
  4. Why a struct pointer? Passing the struct by pointer avoids copying both fields on every call. For hot paths, the frontend could also flatten the struct into two parameters (ptr, i32) to avoid the indirection.

Esper lowering

// Get the length of a string view
func $string_len(%0 ptr) -> i32 {
entry:
    %1 = ldi i32 8
    %2 = add i64 %0, %1
    %3 = load i32 %2          // load length field
    ret i32 %3
}

// Get a byte at index i
func $string_at(%0 ptr, %1 i32) -> i32 {
entry:
    %2 = load ptr %0           // load data pointer
    %3 = add i64 %2, %1       // data + index
    %4 = load8.u i32 %3       // load byte
    ret i32 %4
}

Copying memory (memcpy)

Pseudo-code

func memcpy(dst: *UInt8, src: *UInt8, len: Int32) {
    for i in 0..<len {
        dst[i] = src[i]
    }
}

Step-by-step lowering

  1. Loop counter: i counts from 0 to len, same pattern as memcmp.
  2. Reading a byte: Compute src + i, then load8.u to load one byte.
  3. Writing a byte: Compute dst + i, then store8 to store. store8 truncates the value to 8 bits before writing, so only one byte is stored regardless of the i32 value.
  4. Why byte-by-byte? This is the simplest correct lowering. A real backend would recognize this loop pattern and emit a block copy instruction (rep movsb on x86, memcpy call on ARM). The IR should be correct first; the optimizer makes it fast.

Esper lowering

func $memcpy(%0 ptr, %1 ptr, %2 i32) {
    // params: dst=%0, src=%1, len=%2
entry:
    %3 = ldi i32 4
    %4 = salloc %3             // allocate space for i
    %5 = ldi i32 0
    store i32 %4, %5           // i = 0
    jmp check

check:
    %8 = load i32 %4           // load i
    %9 = ge.u i32 %8, %2      // i >= len?
    jmp.if %9 done copy

copy:
    // src[i]
    %11 = load i32 %4         // load i
    %12 = add i64 %1, %11     // src + i
    %13 = load8.u i32 %12     // load byte from src

    // dst[i] = src[i]
    %14 = load i32 %4         // load i
    %15 = add i64 %0, %14     // dst + i
    store8 i32 %15, %13       // store byte to dst

    %17 = load i32 %4         // load i
    %18 = ldi i32 1
    %19 = add i32 %17, %18    // i + 1
    store i32 %4, %19         // i = i + 1
    jmp check

done:
    ret void
}

Key takeaways