Testing Plugins

Extism plugins are WebAssembly modules, and as such, need to be tested within a WebAssembly runtime in order to truly verify how they behave and perform.

To do this, a new test runner CLI must be used. This tool is called xtp. To install xtp, you can run:

curl https://static.dylibso.com/cli/install.sh | sudo sh

NOTE: Dylibso is the core maintainer and creator of Extism. You can find more information about the company at https://dylibso.com, and if you have any questions about xtp or Extism, please reach us at support@dylibso.com or in the Extism Discord. The xtp CLI is a free developer tool, part of a larger product, which you can learn about here - but you can use xtp CLI without using the XTP platform.

Once you have xtp, you can now run unit tests that call your Extism plugins and assert various things about them, such as outputs for given inputs, plugin state, or timing the performance of plugin function calls. To do this, we provide test harness libraries to create these unit tests for you to run with xtp.

To begin testing your Extism plugins, use any of the following to write tests in:

Within each of these repositories, you will find detailed instructions on how to write, compile, and run tests. It's important to note, that while these libraries are available in JS, Rust, Go, and Zig, you can use them to test Extism plugins which were written in any of the Extism PDK languages.

Testing a Plugin with Host Functions

When your plugins need to make calls to Host Functions, the obvious question introduced is "who is implementing the host functions?".

To solve for this, we provide an optional --mock-host argument to the xtp CLI when you execute tests. Here's an end-to-end example, split into 3 different Wasm projects:

a KV datastore (kvhost), exporting kv_read and kv_write functions (these act in place of real host functions and are imported by the plugin)
an Extism plugin (kvplugin), which interacts with the KV datastore via host function imports
an XTP test plugin (kvtest), which verifies the behavior of the Extism plugin

all of the following code can be found in full here: xtp-test-go/examples

Mock Host Functions
Plugin using Host Functions
XTP Test Plugin

The kvhost project is a simple key-value store that exports kv_read and kv_write functions. These functions are used by the kvplugin project to read and write key-value pairs. When this project is compiled, it will produce a Wasm file that can be used as a host for the kvplugin project, passed as the --mock-host argument to the xtp CLI.

kvhost/main.go
package main

import (
    // to simulate Host Functions, use the PDK to manage host/guest memory.
    pdk "github.com/extism/go-pdk"
)

// this is our in-memory KV store (e.g. a mock database)
var kv map[string]string = make(map[string]string)

// This export will be made available to the plugin as an import function
//go:export kv_read
func kv_read(key uint64) uint64 {
    // find the memory block that contains the key, read the bytes, and look up the 
    // corresponding value in the KV store
    keyMem := pdk.FindMemory(key)
    k := string(keyMem.ReadBytes())
    // if the entry is not found, return 0
    v, ok := kv[k]
    if !ok {
        return 0
    }

    // allocate a new memory block for the value, write the value bytes, and return the offset
    valMem := pdk.AllocateString(v)
    return valMem.Offset()
}

// This export will be made available to the plugin as an import function
//go:export kv_write
func kv_write(key uint64, value uint64) {
    // find the memory block that contains the key and value, read their bytes, 
    // and store the key-value pair in the KV store
    keyMem := pdk.FindMemory(key)
    valueMem := pdk.FindMemory(value)
    k := string(keyMem.ReadBytes())
    v := string(valueMem.ReadBytes())
    kv[k] = v
}

func main() {}

The kvplugin project is an Extism plugin that interacts with the KV datastore via host function imports. This plugin reads and writes key-value pairs and stores them in the plugin's state. We will verify this behavior using the kvtest project in the next tab.

kvplugin/main.go
package main

import (
    // this is a normal Extism plugin, so we import the PDK
    pdk "github.com/extism/go-pdk"
)

// the `kv_read` and `kv_write` functions below are imported from the host,
// (which we exported in the previous tab)

//go:wasmimport extism:host/user kv_read
func kv_read(key uint64) uint64

//go:wasmimport extism:host/user kv_write
func kv_write(key uint64, value uint64)

//go:export run
func run() int32 {
    // allocate a key and value to write to the KV store
    key := pdk.AllocateString("key")
    value := pdk.AllocateString("value")
    kv_write(key.Offset(), value.Offset())

    // immediately read the key back from the KV store, just to verify that it was written
    readVal := kv_read(key.Offset())
    if readVal != 0 {
        // if we found a value, read it from memory and append it to the plugin's state
        // (we'll test the state output in the `kvtest` project)
        readValMem := pdk.FindMemory(readVal)
        varVal := pdk.GetVar("key")
        // grow the var state in the plugin by appending the read value
        pdk.SetVar("key", append(varVal, readValMem.ReadBytes()...))
    } else {
        pdk.SetVar("key", []byte(""))
    }

    // return the plugin's state as the output so we can test it in the `kvtest` project
    pdk.Output(pdk.GetVar("key"))
    return 0
}

func main() {}

The kvtest project is an XTP test plugin that verifies the behavior of the kvplugin project. When we run this test, we will pass the kvhost project as the host, so that the kvplugin project can interact with the KV store we simulate in the kvhost project.

kvtest/main.go
package main

import (
    "fmt"
    "strings"

    // import the test harness library
    xtptest "github.com/dylibso/xtp-test-go"
)

//go:export test
func test() int32 {
    // call the `run` function in the `kvplugin` project, & verify the output
    output := xtptest.CallString("run", nil)
    xtptest.AssertEq("initial call to 'run' returns the correct value", output, "value")

    // call the `run` function in the `kvplugin` project 10 times, & verify the output each time
    xtptest.Group("multiple kv read/write calls produce correct state", func() {
        for i := 0; i < 10; i++ {
            output := xtptest.CallString("run", nil)
            expected := strings.Repeat("value", i+1)
            msg := fmt.Sprintf("repeat call to 'run' returns the correct value: %s", expected)
            xtptest.AssertEq(msg, output, expected)
        }
    })

    return 0
}

func main() {}

After compiling each of these Go projects to Wasm, you can run the test using the xtp CLI, and pass the --mock-host argument to specify the host Wasm file, to stitch all the pieces together:

xtp plugin test kvplugin.wasm --with kvtest.wasm --mock-host kvhost.wasm

You should see output like this:

🧪 Testing plugin.wasm

PASS ...... initial call to 'run' returns the correct value

1/1 tests passed (completed in 17µs)

📦 Group: multiple kv read/write calls produce correct state

PASS ...... repeat call to 'run' returns the correct value: value
PASS ...... repeat call to 'run' returns the correct value: valuevalue
PASS ...... repeat call to 'run' returns the correct value: valuevaluevalue
PASS ...... repeat call to 'run' returns the correct value: valuevaluevaluevalue
PASS ...... repeat call to 'run' returns the correct value: valuevaluevaluevaluevalue
PASS ...... repeat call to 'run' returns the correct value: valuevaluevaluevaluevaluevalue
PASS ...... repeat call to 'run' returns the correct value: valuevaluevaluevaluevaluevaluevalue
PASS ...... repeat call to 'run' returns the correct value: valuevaluevaluevaluevaluevaluevaluevalue
PASS ...... repeat call to 'run' returns the correct value: valuevaluevaluevaluevaluevaluevaluevaluevalue
PASS ...... repeat call to 'run' returns the correct value: valuevaluevaluevaluevaluevaluevaluevaluevaluevalue

10/10 tests passed (completed in 845µs)

all tests completed in 862µs

Mocking input data to plugin test calls

Configure your test with dynamic input provided by a xtp CLI parameter or xtp.toml file. Read runtime-provided input that mocks the actual input when a plugin is called:

Note: this is available in each of the JavaScript/TypeScript, Rust, Go, & Zig test harness libraries.

//go:export test
func test() int32 {
    // use the MockInputBytes() function to read the input data provided by the test runner
    // (there are variations of this function in other xtp-test libraries)
    notEmpty := xtptest.CallString("count_vowels", xtptest.MockInputBytes())
    xtptest.AssertNe("with mock, not empty", notEmpty, "")
  // ...
}

Providing mock input data

There are two ways to provide input data to the plugin test calls:

xtp CLI, using --mock-input-data or --mock-input-file
xtp.toml file

Using the `xtp` CLI

CLI supports args --mock-input-data and --mock-input-file to pass text or load a file.

e.g

xtp plugin test plugin.wasm --with test.wasm --mock-input-data "this is my mock input data"
# or a path to a file for --mock-input-file

Using `xtp.toml`

xtp.toml supports syntax such as:

# path or url locating the wasm plugin to test
bin = "https://raw.githubusercontent.com/extism/extism/main/wasm/code.wasm"

[[test]]
# label this test something recognizable to see in CLI output
name = "basic"
# build the test wasm module, is run before the test
build = "cd examples/countvowels && tinygo build -o test.wasm -target wasi test.go"
# the wasm module to use as the test
with = "examples/countvowels/test.wasm"
# provide mock input data to the plugin test call, returned to a 'MockInput' type of function call
mock_input = { data = "this is my mock input data" }

[[test]]
name = "basic - file input"
build = "cd examples/countvowels && tinygo build -o test.wasm -target wasi test.go"
with = "examples/countvowels/test.wasm"
# load mock input data from a file instead of inline
mock_input = { file = "examples/countvowels/test.go" }

(see examples used in examples/countvowels)

Overriding `xtp.toml` location

When running xtp plugin test, if xtp.toml is present in the current directory, it will be used to configure the test. The location of the file can be overridden using --path:

xtp plugin test --path tests/countvowels

Usage in tests

The various XTP libraries provide convenient functions to dynamically read input from the host, mocked out by the supported options above.

For more context about this tool, the test harness libraries, and why we created them, please read the announcement blog post at: https://dylibso.com/blog/testing-extism-plugins/ or, watch the introduction video:

Testing a Plugin with Host Functions​

Mocking input data to plugin test calls​

Providing mock input data​

Using the xtp CLI​

Using xtp.toml​

Overriding xtp.toml location​

Usage in tests​