Embedding

This page describes the current SSA builder and backend implementation. Its unversioned esp examples are compatibility snapshots, still accepted by the reader. Start with the new text walkthrough for esper 1 syntax and the roadmap for the library-first migration boundary.

Library integration

Esper BE is a single static library (libesper.a) with two headers. Add them to your project and link against the library:

#include "esper.h"       // IR types, parser, printer, codegen, optimization
#include "esper_exec.h"  // VM interpreter, native bindings

clang -Iinclude -o my_tool my_tool.c libesper.a -lm

The portable core is C99. Native code and linking additionally use tinyasm and tinylinker. Standalone amalgamated headers are planned; they are not shipped by this checkout yet.

Parse and print

// Parse from a file
EsperUnit *unit = esper_parse("program.esp");

// Parse from an in-memory string
EsperUnit *unit = esper_parse_string(src, strlen(src), "program.esp");

// Print versioned IR with local assignments
esper_print_ir(unit, stdout);

// Free when done
esper_unit_free(unit);

Optimize

// Apply individual passes
esper_mem2reg(unit);
esper_const_fold(unit);
esper_peephole(unit);
esper_strength(unit, NULL);
esper_copy_prop(unit);
esper_cf_opt(unit);
esper_dce(unit);

// Or use the config-driven API
EsperConfig config = {0};
config.opt_const_fold = true;
config.opt_dce = true;
esper_optimize(unit, &config);

esper_mem2reg is listed first for a reason: it promotes stack slots that are only loaded from and stored to into SSA values, and the scalar passes below it are written to work on what it produces. A frontend that emits one slot per local and no merges at all needs it enabled; one that builds SSA itself can leave it off. esper_optimize always runs it before the others when opt_mem2reg is set. See Optimization Passes for the full pipeline and every flag.

Generate code

// Emit C source
esper_emit_c(unit, stdout);

// Emit LLVM IR
esper_emit_llvm(unit, stdout);

// Or use the config-driven API
EsperConfig config = {0};
config.backend = ESP_BACKEND_C;  // or ESP_BACKEND_LLVM
esper_codegen(unit, &config, stdout);

backend and target are separate fields, and separate enums. The backend is what to produce and is fixed when Esper is built; the target is what machine to produce it for and is chosen per call, which is what lets one binary cross-compile. Leaving target zero means ESP_TARGET_UNKNOWN, which skips the target check. See Build configuration for which backends a given build contains.

Straight to a file

The text emitters above write to a FILE *, which is what you want when something else is going to compile the result. When Esper is doing the whole job, four entry points go all the way:

// An object file in the target's own format, for a linker to consume.
esper_emit_native_object(unit, "prog.o", target, 0);

// The same, with a DWARF line table and compile unit written into it,
// so a debugger can bind a breakpoint to a line of your source.
esper_emit_native_object_debug(unit, "prog.o", target, 0, true);

// An executable. Writes the object, links it with the runtime archive
// when the program reaches into it, and removes the object again.
esper_emit_native_exe(unit, "prog", target, 0);

// A shared library: a .so, a .dylib or a .dll. A unit with no entry
// point is fine here, which is the difference that matters.
esper_emit_native_lib(unit, "libprog.so", target, 0);

The last two need ESPER_WITH_TINYLINKER; without it they report rather than writing a file that would not run. The features argument names processor capabilities to generate against, and zero means the target’s baseline.

esper_emit_llvm_for is the LLVM emitter with an explicit target, for the same reason: a module carrying no target is read as the machine that reads it, and the layout decides how wide a pointer is and how a struct is packed.

Asking about a target

The target enum is opaque; everything you might need to know about one is a call, so a frontend does not have to carry its own table:

EsperTarget target = esper_target_from_name("arm64-linux");

esper_target_name(target);              // back to "arm64-linux"
esper_target_arch(target);              // ESP_ARCH_ARM64
esper_target_os(target);                // ESP_OS_LINUX
esper_target_format(target);            // ESP_FORMAT_ELF
esper_target_pointer_size(target);      // 8
esper_target_is_little_endian(target);  // true
esper_target_llvm_triple(target);       // "aarch64-unknown-linux-gnu"
esper_target_llvm_layout(target);       // the datalayout string

esper_format_name names an object format, and esper_backend_from_name turns a backend name into the enum, which is what the CLI’s --emit= does.

Pointer size is the one that reaches into the IR: it decides every aggregate offset. Call esper_unit_layout_for when you change target, and esper_unit_layout_matches to ask whether the cached layout belongs to the target you are about to emit for. Every backend asks before emitting, because a mismatch that is not refused becomes offsets that are quietly wrong.

Checking you linked what you compiled against

EsperConfig is a struct a consumer allocates, so a header from one version and a library from another disagree about its size and the disagreement is silent.

if (!ESPER_ABI_OK()) {
  fprintf(stderr, "esper: built against %s, linked %s\n",
          ESPER_VERSION_STRING, esper_version());
  return 1;
}

ESPER_ABI_OK() expands to esper_abi_compatible(ESPER_VERSION_NUMBER, sizeof(EsperConfig)), comparing what the header said with what the library was built from. esper_version and esper_version_number report the library’s own version.

Execute with the VM

#include "esper_exec.h"

// Execute with default configuration
int32_t result = esper_exec(unit, "main");

// Execute with custom configuration and native bindings
EsperExecConfig config = esper_exec_config_default();

// Bind native functions to extern declarations
void my_print(EsperValue *args, uint32_t argc, EsperValue *rets,
              uint32_t retc, void *ctx) {
    printf("%d\n", args[0].as_i32);
}

EsperBinding bindings[] = {
    { .name = "print_i32", .fn = my_print, .ctx = NULL },
};
config.bindings = bindings;
config.binding_count = 1;

// A non-zero step budget bounds the run, so a program that does not
// terminate reports rather than hanging its caller.
config.step_limit = 10000000;

int32_t result = esper_exec_with(unit, "main", &config);

The public headers deliberately keep the standard C names. The short aliases (i32, u32, usize) are internal to the library’s own sources and are not installed, so they are not available to an embedder.

Panics, traps, and the symbols your language supplies

Esper has no notion of a panic, and deliberately so. A language decides what happens when an index is out of range or an addition overflows, and those decisions differ: Rust unwinds with a message, Swift traps, Zig prints a trace in debug and does nothing in release. Esper’s job is to emit the check and the branch; what sits at the end of the branch is yours.

There are two ways to end such a path, and they are not interchangeable.

trap stops the program with no explanation. It needs nothing from you, costs one instruction, and is the right answer when the language’s response is simply to die.

bounds_fail:
  trap

A call to your own function is the right answer when the language has something to say. Declare it as an extern, call it, and end the block with unreachable so the optimizer knows control does not come back:

func extern $rt_panic_bounds(i32, i32)

; ...

bounds_fail:
  call $rt_panic_bounds, %1, %2
  unreachable

Note what is not being passed: a message. Esper has no string constant, so the text of a panic lives in your runtime, and what crosses the boundary is whatever identifies it. An index into your own message table is the usual answer, and it is cheaper than a pointer besides.

Nothing above is special to Esper. $rt_panic_bounds is an ordinary extern call, so the same IR serves any language, and the difference between Rust’s behaviour and Zig’s is entirely in what gets linked.

Linking the compiled program

An extern is a name the object file leaves undefined, so your runtime has to supply it at link time like any other library:

cc program.o libmylang_rt.a -o program

If the symbol is missing the linker says so, naming it. That is the intended failure: a frontend that emits a call to a panic handler it never provides finds out at link time rather than at run time.

Two symbols come from Esper’s own runtime rather than yours, on the machine-code backend only: esp_trap, which a lowered trap calls, and the bit and byte reversal helpers. Link libesper_rt alongside your own runtime when you use that backend. The C and LLVM backends emit compiler builtins for these and need nothing.

Running the same program under the VM

The interpreter resolves an extern through a binding rather than a linker, so the callback shape you would otherwise reach for already exists. Bind the same name to a host function:

static void rt_panic_bounds(EsperValue *args, uint32_t argc, EsperValue *rets,
                            uint32_t retc, void *ctx) {
    (void)argc; (void)rets; (void)retc;
    fprintf(stderr, "index %d out of range\n", args[1].as_i32);
    abort();
}

EsperBinding bindings[] = {
    { .name = "rt_panic_bounds", .fn = rt_panic_bounds, .ctx = NULL },
};

So one IR runs both ways: linked against your runtime when compiled, bound to a host function when interpreted. An extern with neither is reported as exec.unbound_extern rather than crashing.

What the optimizer does with it, and what it will not do

The three endings behave differently on purpose, and the difference is worth knowing before you pick one.

An arm holding nothing but unreachable is an assertion that control never arrives. cf_opt folds the branch away and DCE takes the comparison with it, so a check the frontend has already proved cannot fail costs nothing:

entry:                              entry:
  %3 = lt.u i32 %1, %2        ->      nop
  jmp.if %3 ok fail                   nop
fail:                               ok:
  unreachable                         %5 = ldi i32 42

An arm that calls your panic handler keeps the branch. The call is an observable effect, and folding the branch away would delete it. That is the right answer and not a missed optimization: you asked for the call, so you get the check that guards it.

A trap is never folded either, for the same reason. Stopping the program is an effect, not a claim that the path is impossible.

So the two are not interchangeable. Reach for a bare unreachable only where the language guarantees the case cannot happen and you want the check to cost nothing; reach for a call or a trap where reaching it is a real outcome you want to observe. Getting this backwards is undefined behaviour rather than a missing message: a bare unreachable that is actually reached makes the whole function’s behaviour undefined, not just that path’s.

Diagnostics

Everything Esper has to say about a unit reaches the embedder the same way: a code, a severity, a place, and a sentence. The code is what carries the meaning. The sentence is Esper’s own wording and a fallback for a host that has not written its own yet.

Nothing is written to stderr unless no sink has been installed, which keeps the library out of its host’s output.

The sink

An EsperDiagSink is a function pointer and a context pointer:

typedef void (*EsperDiagFn)(const EsperDiag *diag, void *ctx);

typedef struct {
    EsperDiagFn fn;
    void *ctx;
} EsperDiagSink;

Each diagnostic arrives as an EsperDiag:

typedef struct {
    EsperDiagCode code;
    EsperDiagSeverity severity;   // ESP_DIAG_ERROR or ESP_DIAG_WARNING
    uint32_t source;              // index into the unit's source table
    uint32_t line;                // 1-based, 0 when there is no place
    uint32_t column;
    char *message;
} EsperDiag;

The diagnostic and its message belong to Esper and do not outlive the call, so copy anything you intend to keep. Reporting is not reentrant: do not call back into Esper on the same unit from inside the callback.

source indexes the unit’s source table, which esper_unit_add_source fills; resolve it with esper_unit_source. Both line and column are zero where there is nothing to point at, which today is every diagnostic except the parser’s and the builder’s. The builder carries the position last set with esper_build_set_loc.

Installing one on a unit

esper_unit_set_diags covers everything reported about a unit after parsing: building, optimization, code generation, and interpretation. Passing NULL reverts to stderr.

static void receive(const EsperDiag *diag, void *ctx) {
    struct MyCompiler *cc = ctx;
    my_report(cc, esper_diag_code_name(diag->code), diag->line, diag->column,
              diag->message);
}

EsperDiagSink sink = { receive, compiler };
esper_unit_set_diags(unit, &sink);

The sink is stored on the unit, so it must be installed before the phase whose diagnostics you want, and it survives until it is replaced.

Collecting instead of handling

esper_diaglist_sink wraps an EsperDiagList as a sink, for a caller that would rather gather diagnostics and inspect them afterwards. The list copies each message, so entries stay valid after the reporting call returns.

EsperDiagList list;
esper_diaglist_init(&list);

EsperDiagSink sink = esper_diaglist_sink(&list);
esper_unit_set_diags(unit, &sink);

// ... build, optimize, emit ...

for (uint32_t i = 0; i < list.count; i++) {
    const EsperDiag *d = &list.items[i];
    printf("%s: %s\n", esper_diag_code_name(d->code), d->message);
}
esper_diaglist_free(&list);

The list must outlive every use of the sink made from it, and esper_diaglist_free releases both the messages and the array.

Parsing

Parsing happens before a unit exists, so it cannot take a sink from one. It takes an EsperDiagList directly instead:

EsperDiagList diags;
esper_diaglist_init(&diags);

EsperUnit *unit = esper_parse_string_diag(src, len, "bad.esp", &diags);

for (uint32_t i = 0; i < diags.count; i++) {
    const EsperDiag *d = &diags.items[i];
    printf("bad.esp:%u:%u: %s: %s\n", d->line, d->column,
           esper_diag_code_name(d->code), d->message);
}
esper_diaglist_free(&diags);
esper_unit_free(unit);

The parser recovers at top-level boundaries, so one parse can report several errors: on a syntax error it discards the rest of the current declaration and resumes at the next top-level keyword. Function bodies are read after every declaration in the unit, and each has a recovery point of its own, so a body it cannot read costs only that body. It returns NULL when any error-severity diagnostic was recorded, and the list is the caller’s to free either way.

esper_parse_string is the older shim around this. It prints to stderr and returns NULL on error, with no way to inspect what happened. Prefer the _diag form in anything embedded.

Installing a sink on the unit afterwards is the normal follow-up, since the parse list covers only the parse.

Runtime diagnostics

The runtime library is linked into the program Esper compiled, not into the compiler, so it has no unit to hang a sink on. It reports through a separate global one:

void esp_rt_set_diags(const EsperDiagSink *sink);

The separation is deliberate: a host embedding Esper can send a compiled program’s allocation failures somewhere other than its own compile errors, which is what it wants when the program is a user’s and the errors are its own.

esp_rt_set_diags is not synchronized. Install it before the program runs, not while it is running, and not from more than one thread. Passing NULL reverts to stderr, where the runtime prints in the form [esper] rt.trace: ....

Runtime diagnostics never carry a position: source, line, and column are always zero, since there is no source table on that side.

Codes

EsperDiagCode is banded by phase, so an embedder can switch on a range as readily as on a value:

The full set as of this writing:

esper_diag_code_name maps a code to the short name in the second column. It never returns NULL, and answers "?" for a code it does not recognize, which includes ESP_D_NONE and every gap between the bands.

Switching on the band rather than the value is the intended shape for a host that wants a coarse classification:

static const char *phase_of(EsperDiagCode code) {
    if (code >= 500) return "runtime";
    if (code >= 400) return "codegen";
    if (code >= 300) return "exec";
    if (code >= 200) return "build";
    if (code >= 100) return "parse";
    return "none";
}

What is guaranteed

An embedder mapping Esper’s codes onto its own needs to know what will not move. Stated plainly:

Guaranteed.

• A numeric code is never reused for a different meaning. A diagnostic that stops being reported leaves its code behind rather than letting a later one inherit it.
• The band a phase owns does not move. parse stays at 100, build at 200, exec at 300, codegen at 400, rt at 500. Each band is anchored by an explicit initializer in the enum, so appending to one band cannot shift another.
• The short name esper_diag_code_name returns for a given code is stable, and is the form intended for logs and for a host’s own lookup table.
• esper_diag_code_name returns a non-NULL static string for every input, including a value that is not a code at all.

Not guaranteed.

• The set is not closed. New codes are appended to the end of their band, so a host must have a default arm. Every code added since the enum was introduced has been an append: build.scope, exec.step_limit, and the whole 500 band arrived that way, and nothing has been renumbered or removed.
• Message wording is not stable. It is Esper’s own phrasing, and a compiler built on Esper is expected to re-say it in its own words. Do not match on it.
• Which code a given input produces is not fixed. A diagnostic may be reclassified from one code to a more specific one within its band as the library gets better at telling cases apart. This matters most in the parse band, where the classification is currently derived from the message text: a change to the wording can therefore change the code, even though neither the code’s meaning nor its number changes.
• Severity may change. A condition reported as a warning today may become an error, or the reverse.
• Position fields are best-effort. A diagnostic that carries no position reports line and column as zero, and more diagnostics may gain positions over time. Do not treat a zero line as a distinguishing feature.

Two consequences for a host. Switch on the code with a default arm rather than matching text, and treat a code you do not know by its band. And do not assume a parse.syntax will not become a parse.bad_type in a later release; the reverse mapping, from code to your own diagnostic, is the direction that holds.

A worked example

Installing one sink and provoking each band:

static void receive(const EsperDiag *diag, void *ctx) {
    printf("[%s] %s (%d) at %u:%u: %s\n", (const char *)ctx,
           esper_diag_code_name(diag->code), (int)diag->code,
           diag->line, diag->column, diag->message);
}

EsperDiagSink sink = { receive, (void *)"compiler" };

EsperUnit *unit = esper_unit_new("m");
esper_unit_set_diags(unit, &sink);

uint32_t file = esper_unit_add_source(unit, "hello.src");

EsperTypeRef rets[1] = { ESP_TYPE_I32 };
EsperBuilder *b = esper_build_begin(unit, "f", NULL, 0, rets, 1);
esper_build_set_loc(b, file, 12, 5);
esper_build_call(b, ESP_TYPE_I32, "nope", NULL, 0);   // build.unknown_callee
esper_build_end(b);
esper_builder_free(b);

EsperConfig cfg = {0};
cfg.backend = ESP_BACKEND_GLSL;
esper_codegen(unit, &cfg, stdout);                    // codegen.no_backend

esper_exec(unit, "missing");                          // exec.no_entry

EsperDiagSink rt = { receive, (void *)"program" };
esp_rt_set_diags(&rt);
esp_enable_trace(true);
esp_free(esp_halloc(16));                             // rt.trace, twice

That prints:

[compiler] build.unknown_callee (203) at 12:5: call to undeclared function 'nope'; declare it first
[compiler] codegen.no_backend (400) at 0:0: backend 'glsl' is not available in this build
[compiler] exec.no_entry (300) at 0:0: entry function '$missing' does not exist
[program] rt.trace (501) at 0:0: halloc 16 bytes -> 0x100e25e50
[program] rt.trace (501) at 0:0: free 0x100e25e50

The builder diagnostic carries the 12:5 that esper_build_set_loc supplied. The others report 0:0, and the runtime pair went to the other sink.

Runtime library

The runtime provides allocation wrappers with optional debug tracing:

// Enable allocation tracing to stderr
esp_enable_trace(true);

// Use runtime allocators (backends emit calls to these)
void *ptr = esp_halloc(1024);
esp_free(ptr);

// Check for leaks
uint64_t allocs, frees, bytes;
esp_alloc_stats(&allocs, &frees, &bytes);
printf("allocs=%llu frees=%llu leaked=%llu\n",
       (unsigned long long)allocs, (unsigned long long)frees,
       (unsigned long long)(allocs - frees));

The runtime reports through its own sink rather than the unit’s. See Runtime diagnostics above.

Pluggable runtimes (planned)

The runtime currently ships as src/runtime/runtime.c and is linked into every output. A pluggable Runtime ABI is planned so callers can swap implementations without rebuilding the backends:

# Default: tracing runtime with stats and leak detection
esper compile program.esp --target c

# Thin libc wrappers (zero overhead, no tracing)
esper compile program.esp --target c --libc

# Named flavor
esper compile program.esp --target c --runtime=freestanding

# Custom symbol prefix for embedding into a host that owns esp_*
esper compile program.esp --target c --runtime-prefix=mygame_

The ABI surface is intentionally small. Any host language (Rust, Zig, C++, hand-written asm) can satisfy it by exposing these symbols:

void *esp_rt_alloc(u64 size);          // May trap on OOM
void  esp_rt_free(void *ptr);          // Must accept NULL
void  esp_rt_abort(const char *msg);   // Panic / trap path
i64   esp_rt_now_ns(void);             // Optional, only when trace-mode is on

Shipped flavors will include libc, traced (current behavior), freestanding (arena-only), wasm (bump allocator), and none (caller provides everything). See PROGRESS.md for the implementation roadmap.

Type reference

Saving and combining units

esper_write and esper_read save and load a complete core IR archive through caller-owned streams. esper_serialize and esper_deserialize provide the same format in memory. esper_merge combines loaded or directly built units into a new unit, resolving imports before you call esper_optimize.

See the translation-unit guide for complete ownership examples, private symbols, diagnostics, and current indirect-call limits.