Closures and Callbacks

This page describes the current SSA builder and backend implementation. Its unversioned esp examples are compatibility snapshots, still accepted by the reader. Start with the new text walkthrough for esper 1 syntax and the roadmap for the library-first migration boundary.

Esper has no built-in closure type. Closures are lowered by the frontend into a function pointer paired with an environment pointer, using call.ind for the indirect call. This guide walks through the lowering from simple function pointers to full closures with captured variables.

Each example is shown twice: the calls a frontend makes against the builder API in esper.h, and the text those calls produce. The two halves are checked against each other, so the C tab builds exactly what the Esper tab prints.

What we’re lowering

func makeAdder(n: Int32) -> (Int32) -> Int32 {
    return { x in x + n }  // captures n from enclosing scope
}

let add5 = makeAdder(5)
add5(10)  // returns 15
add5(20)  // returns 25

The closure { x in x + n } captures n from makeAdder’s scope. After makeAdder returns, the closure must still be able to access n. The frontend must allocate n somewhere that outlives the call and pass it to the closure at each invocation.

The standard lowering is:

Allocate an environment struct in memory containing the captured values
The closure function takes the environment pointer as a hidden first parameter
The closure value is a (function_index, env_pointer) pair

Before diving into closures, we start with the simpler case: function pointers with no captures.

Simple function pointers

Pseudo-code

func apply(fn: (Int32) -> Int32, value: Int32) -> Int32 {
    return fn(value)
}

func double(x: Int32) -> Int32 { return x * 2 }
func triple(x: Int32) -> Int32 { return x * 3 }

apply(double, 21)  // returns 42

Step-by-step lowering

Function indices: Each function in the Esper unit has a numeric index, assigned in the order the functions are added. $double is index 0 because it was built first. The frontend tracks this mapping.
Passing a function: Load the function’s index as a plain i32. From Esper’s perspective, it is an integer and nothing more.
Calling through a pointer: call.ind takes a function index value, a signature index, and then the arguments.
Why call.ind instead of call? call requires a statically known function name. call.ind resolves the target at runtime, enabling higher-order functions, vtable dispatch, and callbacks.

The signature index

A direct call reads its arity and its types off the callee. An indirect call has only an SSA value, so there is nothing to read them off. The shape is stated once in the unit’s signature table and named by index everywhere it is used.

esper_unit_add_signature records a shape and returns the index. esper_build_call_indirect takes that index and checks the argument count against it. A signature already in the table is returned rather than added again, so a frontend may call this per call site and still end up with one entry per distinct shape.

EsperTypeRef params[1] = {ESP_TYPE_I32};
EsperTypeRef rets[1] = {ESP_TYPE_I32};
uint32_t fn_sig = esper_unit_add_signature(unit, params, 1, rets, 1);

sig (i32) -> i32

The index a call site carries should always be the value this call returned. Writing the number by hand works until a second signature exists, at which point the argument-count check runs against the wrong shape.

Esper lowering

/// Builds one of the two functions `apply` can be handed.
static void build_scale(EsperUnit *unit, const char *name, int32_t factor) {
  EsperTypeRef params[1] = {ESP_TYPE_I32};
  EsperTypeRef rets[1] = {ESP_TYPE_I32};

  EsperBuilder *b = esper_build_begin(unit, name, params, 1, rets, 1);
  esper_build_bind(b, esper_build_block(b, "entry"));

  EsperVal k = esper_build_ldi(b, ESP_TYPE_I32, factor);
  EsperVal out = esper_build_binary(b, ESP_OP_MUL, ESP_TYPE_I32,
                                    esper_build_param(b, 0), k);
  esper_build_ret(b, &out, 1);

  esper_build_end(b);
  esper_builder_free(b);
}

/// Builds the higher-order function, which calls through its parameter.
static void build_apply(EsperUnit *unit, uint32_t sig) {
  EsperTypeRef params[2] = {ESP_TYPE_I32, ESP_TYPE_I32};
  EsperTypeRef rets[1] = {ESP_TYPE_I32};

  EsperBuilder *b = esper_build_begin(unit, "apply", params, 2, rets, 1);
  esper_build_bind(b, esper_build_block(b, "entry"));

  // Parameter 0 is the function index, parameter 1 the argument.
  EsperVal args[1] = {esper_build_param(b, 1)};
  EsperVal out = esper_build_call_indirect(b, ESP_TYPE_I32,
                                           esper_build_param(b, 0), sig,
                                           args, 1);
  esper_build_ret(b, &out, 1);

  esper_build_end(b);
  esper_builder_free(b);
}

static void esper_doc_build(EsperUnit *unit) {
  // The shape of every function apply can be handed.
  EsperTypeRef sig_params[1] = {ESP_TYPE_I32};
  EsperTypeRef sig_rets[1] = {ESP_TYPE_I32};
  uint32_t fn_sig = esper_unit_add_signature(unit, sig_params, 1, sig_rets, 1);

  build_scale(unit, "double", 2);
  build_scale(unit, "triple", 3);
  build_apply(unit, fn_sig);

  EsperTypeRef rets[1] = {ESP_TYPE_I32};
  EsperBuilder *b = esper_build_begin(unit, "main", NULL, 0, rets, 1);
  esper_build_bind(b, esper_build_block(b, "entry"));

  // $double was built first, so its index is zero.
  EsperVal target = esper_build_ldi(b, ESP_TYPE_I32, 0);
  EsperVal value = esper_build_ldi(b, ESP_TYPE_I32, 21);
  EsperVal args[2] = {target, value};
  EsperVal out = esper_build_call(b, ESP_TYPE_I32, "apply", args, 2);
  esper_build_ret(b, &out, 1);

  if (!esper_build_end(b))
    fprintf(stderr, "main: %s\n", esper_builder_last_error(b));

  esper_builder_free(b);
}

sig (i32) -> i32

func $double(%0 i32) -> i32 {
entry:
  %1 = ldi i32 2
  %2 = mul i32 %0, %1
  ret i32 %2
}

func $triple(%0 i32) -> i32 {
entry:
  %1 = ldi i32 3
  %2 = mul i32 %0, %1
  ret i32 %2
}

func $apply(%0 i32, %1 i32) -> i32 {
entry:
  %2 = call.ind i32 %0, 0, %1
  ret i32 %2
}

func $main() -> i32 {
entry:
  %0 = ldi i32 0
  %1 = ldi i32 21
  %2 = call i32 $apply, %0, %1
  ret i32 %2
}

Running $main on the interpreter returns 42, which is the check that matters: the 0 in ldi i32 0 really does reach $double, and $triple is reachable by passing 1 instead. The index is positional and the frontend owns the mapping, so a frontend that reorders its output has to renumber every function value it has already emitted.

Closures with captured variables

Pseudo-code

func makeAdder(n: Int32) -> (Int32) -> Int32 {
    return { x in x + n }  // captures n
}
let add5 = makeAdder(5)
add5(10)  // returns 15

Step-by-step lowering

Environment struct: The closure captures n, so the environment is a single i32 at offset 0 (4 bytes total). If the closure captured multiple variables, they would be laid out sequentially in the struct.
Closure function signature: The original lambda (Int32) -> Int32 becomes (ptr, Int32) -> Int32. The hidden first parameter is the environment pointer, and that is the shape the signature table records.
Accessing captures: Inside the closure body, n is loaded from the environment pointer with load i32.
Factory function: makeAdder allocates the environment, stores n into it, and returns the pointer. The caller holds that alongside the function index.
Calling the closure: call.ind with the environment as the first argument and the original argument after it.
Why a hidden first parameter? Every closure function then has the same signature shape, (ptr env, ...original params), so one signature covers all of them and a plain function pointer can join the same table by taking an ignored environment.

Esper lowering

/// Builds the lambda body. Parameter zero is the environment.
static void build_adder_closure(EsperUnit *unit) {
  EsperTypeRef params[2] = {ESP_TYPE_PTR, ESP_TYPE_I32};
  EsperTypeRef rets[1] = {ESP_TYPE_I32};

  EsperBuilder *b = esper_build_begin(unit, "adder_closure", params, 2, rets, 1);
  esper_build_bind(b, esper_build_block(b, "entry"));

  // n is the only capture, so it sits at offset 0 of the environment.
  EsperVal n = esper_build_load(b, ESP_OP_LOAD, ESP_TYPE_I32,
                                esper_build_param(b, 0));
  EsperVal out = esper_build_binary(b, ESP_OP_ADD, ESP_TYPE_I32,
                                    esper_build_param(b, 1), n);
  esper_build_ret(b, &out, 1);

  esper_build_end(b);
  esper_builder_free(b);
}

/// Builds the factory, which allocates the environment and fills it.
static void build_make_adder(EsperUnit *unit) {
  EsperTypeRef params[1] = {ESP_TYPE_I32};
  EsperTypeRef rets[1] = {ESP_TYPE_PTR};

  EsperBuilder *b = esper_build_begin(unit, "make_adder", params, 1, rets, 1);
  esper_build_bind(b, esper_build_block(b, "entry"));

  // The environment outlives this call, so it cannot be a stack allocation.
  EsperVal size = esper_build_ldi(b, ESP_TYPE_I32, 4);
  EsperVal env = esper_build_alloc(b, ESP_OP_HALLOC, size);
  esper_build_store(b, ESP_OP_STORE, ESP_TYPE_I32, env,
                    esper_build_param(b, 0));
  esper_build_ret(b, &env, 1);

  esper_build_end(b);
  esper_builder_free(b);
}

static void esper_doc_build(EsperUnit *unit) {
  // Every closure of this arity has the same shape once the environment
  // is counted as a parameter.
  EsperTypeRef sig_params[2] = {ESP_TYPE_PTR, ESP_TYPE_I32};
  EsperTypeRef sig_rets[1] = {ESP_TYPE_I32};
  uint32_t closure_sig = esper_unit_add_signature(unit, sig_params, 2,
                                                  sig_rets, 1);

  build_adder_closure(unit);
  build_make_adder(unit);

  EsperTypeRef rets[1] = {ESP_TYPE_I32};
  EsperBuilder *b = esper_build_begin(unit, "main", NULL, 0, rets, 1);
  esper_build_bind(b, esper_build_block(b, "entry"));

  // make_adder(5) yields the environment half of the closure value.
  EsperVal five = esper_build_ldi(b, ESP_TYPE_I32, 5);
  EsperVal env = esper_build_call(b, ESP_TYPE_PTR, "make_adder", &five, 1);

  // $adder_closure was built first, so the code half is index zero.
  EsperVal code = esper_build_ldi(b, ESP_TYPE_I32, 0);
  EsperVal x = esper_build_ldi(b, ESP_TYPE_I32, 10);
  EsperVal args[2] = {env, x};
  EsperVal out = esper_build_call_indirect(b, ESP_TYPE_I32, code, closure_sig,
                                           args, 2);
  esper_build_ret(b, &out, 1);

  if (!esper_build_end(b))
    fprintf(stderr, "main: %s\n", esper_builder_last_error(b));

  esper_builder_free(b);
}

sig (ptr, i32) -> i32

func $adder_closure(%0 ptr, %1 i32) -> i32 {
entry:
  %2 = load i32 %0
  %3 = add i32 %1, %2
  ret i32 %3
}

func $make_adder(%0 i32) -> ptr {
entry:
  %1 = ldi i32 4
  %2 = halloc %1
  store i32 %2, %0
  ret ptr %2
}

func $main() -> i32 {
entry:
  %0 = ldi i32 5
  %1 = call ptr $make_adder, %0
  %2 = ldi i32 0
  %3 = ldi i32 10
  %4 = call.ind i32 %2, 0, %1, %3
  ret i32 %4
}

The allocation in $make_adder is halloc and not salloc because a stack allocation is released when the function returns, and the environment is the one thing in a closure that must outlive its factory. esper_build_alloc takes the opcode for exactly this decision. ESP_OP_ALLOC is the third choice: it defers the question to escape analysis, which is the right one when the frontend does not know whether the closure escapes.

Nothing here ties %2 = ldi i32 0 to $adder_closure except the frontend’s own bookkeeping. The pair (0, %1) is the closure value; a language that stores closures in variables stores those two words together, in a struct laid out the same way as the callback slot below.

Callback registration

Pseudo-code

struct Callback {
    var handler: (Context) -> Int32
    var context: Context
}

func register(slot: *Callback, handler: (Context) -> Int32, ctx: Context) {
    slot.handler = handler
    slot.context = ctx
}

func invoke(slot: *Callback) -> Int32 {
    return slot.handler(slot.context)
}

Step-by-step lowering

Callback layout: A callback slot is 16 bytes: i32 func_index at offset 0, 4 bytes of padding, ptr context at offset 8.
Registration: Write the function index and the context pointer into the slot.
Invocation: Load the context, load the function index, then call.ind. The context pointer serves the same role as the closure environment.
Why the same pattern as closures? Callbacks, closures, and vtable dispatch are the same mechanism: an indirect call with some associated data. The only difference is who allocates the data and how long it lives.

This is the closure from the other side. $make_adder produced the pair and handed it out; $register_callback receives a pair someone else produced and stores it. The invocation is identical either way.

Esper lowering

/// Builds the handler a caller will register. Its only argument is the
/// context, which is the same role the closure environment played.
static void build_handler(EsperUnit *unit) {
  EsperTypeRef params[1] = {ESP_TYPE_PTR};
  EsperTypeRef rets[1] = {ESP_TYPE_I32};

  EsperBuilder *b = esper_build_begin(unit, "handler", params, 1, rets, 1);
  esper_build_bind(b, esper_build_block(b, "entry"));

  EsperVal v = esper_build_load(b, ESP_OP_LOAD, ESP_TYPE_I32,
                                esper_build_param(b, 0));
  esper_build_ret(b, &v, 1);

  esper_build_end(b);
  esper_builder_free(b);
}

/// Builds the writer: a function index at offset 0, a context at offset 8.
static void build_register(EsperUnit *unit) {
  EsperTypeRef params[3] = {ESP_TYPE_PTR, ESP_TYPE_I32, ESP_TYPE_PTR};

  EsperBuilder *b = esper_build_begin(unit, "register_callback", params, 3,
                                      NULL, 0);
  esper_build_bind(b, esper_build_block(b, "entry"));

  EsperVal slot = esper_build_param(b, 0);
  esper_build_store(b, ESP_OP_STORE, ESP_TYPE_I32, slot,
                    esper_build_param(b, 1));

  EsperVal eight = esper_build_ldi(b, ESP_TYPE_I32, 8);
  EsperVal ctx_at = esper_build_binary(b, ESP_OP_ADD, ESP_TYPE_I64, slot, eight);
  esper_build_store(b, ESP_OP_STORE, ESP_TYPE_PTR, ctx_at,
                    esper_build_param(b, 2));

  esper_build_ret(b, NULL, 0);
  esper_build_end(b);
  esper_builder_free(b);
}

/// Builds the reader, which is the same three steps a virtual call takes.
static void build_invoke(EsperUnit *unit, uint32_t sig) {
  EsperTypeRef params[1] = {ESP_TYPE_PTR};
  EsperTypeRef rets[1] = {ESP_TYPE_I32};

  EsperBuilder *b = esper_build_begin(unit, "invoke_callback", params, 1,
                                      rets, 1);
  esper_build_bind(b, esper_build_block(b, "entry"));

  EsperVal slot = esper_build_param(b, 0);
  EsperVal eight = esper_build_ldi(b, ESP_TYPE_I32, 8);
  EsperVal ctx_at = esper_build_binary(b, ESP_OP_ADD, ESP_TYPE_I64, slot, eight);
  EsperVal ctx = esper_build_load(b, ESP_OP_LOAD, ESP_TYPE_PTR, ctx_at);
  EsperVal target = esper_build_load(b, ESP_OP_LOAD, ESP_TYPE_I32, slot);

  EsperVal out = esper_build_call_indirect(b, ESP_TYPE_I32, target, sig,
                                           &ctx, 1);
  esper_build_ret(b, &out, 1);

  esper_build_end(b);
  esper_builder_free(b);
}

static void esper_doc_build(EsperUnit *unit) {
  EsperTypeRef sig_params[1] = {ESP_TYPE_PTR};
  EsperTypeRef sig_rets[1] = {ESP_TYPE_I32};
  uint32_t handler_sig = esper_unit_add_signature(unit, sig_params, 1,
                                                  sig_rets, 1);

  build_handler(unit);
  build_register(unit);
  build_invoke(unit, handler_sig);

  EsperTypeRef rets[1] = {ESP_TYPE_I32};
  EsperBuilder *b = esper_build_begin(unit, "main", NULL, 0, rets, 1);
  esper_build_bind(b, esper_build_block(b, "entry"));

  // A sixteen byte slot and a four byte context, both frame-local here.
  EsperVal slot_size = esper_build_ldi(b, ESP_TYPE_I32, 16);
  EsperVal slot = esper_build_alloc(b, ESP_OP_SALLOC, slot_size);
  EsperVal ctx_size = esper_build_ldi(b, ESP_TYPE_I32, 4);
  EsperVal ctx = esper_build_alloc(b, ESP_OP_SALLOC, ctx_size);
  EsperVal seven = esper_build_ldi(b, ESP_TYPE_I32, 7);
  esper_build_store(b, ESP_OP_STORE, ESP_TYPE_I32, ctx, seven);

  // $handler was built first, so its index is zero.
  EsperVal code = esper_build_ldi(b, ESP_TYPE_I32, 0);
  EsperVal reg[3] = {slot, code, ctx};
  esper_build_call(b, ESP_TYPE_VOID, "register_callback", reg, 3);

  EsperVal out = esper_build_call(b, ESP_TYPE_I32, "invoke_callback", &slot, 1);
  esper_build_ret(b, &out, 1);

  if (!esper_build_end(b))
    fprintf(stderr, "main: %s\n", esper_builder_last_error(b));

  esper_builder_free(b);
}

sig (ptr) -> i32

func $handler(%0 ptr) -> i32 {
entry:
  %1 = load i32 %0
  ret i32 %1
}

func $register_callback(%0 ptr, %1 i32, %2 ptr) {
entry:
  store i32 %0, %1
  %4 = ldi i32 8
  %5 = add i64 %0, %4
  store ptr %5, %2
  ret
}

func $invoke_callback(%0 ptr) -> i32 {
entry:
  %1 = ldi i32 8
  %2 = add i64 %0, %1
  %3 = load ptr %2
  %4 = load i32 %0
  %5 = call.ind i32 %4, 0, %3
  ret i32 %5
}

func $main() -> i32 {
entry:
  %0 = ldi i32 16
  %1 = salloc %0
  %2 = ldi i32 4
  %3 = salloc %2
  %4 = ldi i32 7
  store i32 %3, %4
  %6 = ldi i32 0
  call $register_callback, %1, %6, %3
  %8 = call i32 $invoke_callback, %1
  ret i32 %8
}

The gaps in $main, between %4 and %6 and between %6 and %8, are not errors. A void instruction still occupies an SSA id, so the store and the call to $register_callback each consume one. A frontend that numbers its own ids instead of using the value the builder returns will drift out of step here.

The slot and the context are salloc in this example because both live and die inside $main. A callback registered with something that outlives the registering frame needs halloc for both, and something has to free them: Esper has no garbage collector, and esper_build_free is the only release.

Key takeaways

A closure is a (function_index, environment_pointer) pair
The environment is allocated in memory, with halloc when it outlives its factory
The closure function receives the environment as a hidden first parameter
call.ind handles the indirect call, and needs a signature index from esper_unit_add_signature because there is no callee to read the shape off
The frontend owns the function-index mapping; nothing in the IR records which function an integer refers to
This pattern works for closures, callbacks, vtable dispatch, and coroutine resume functions