Here at Enalean, we are dedicated to make Tuleap evolve in a way that satisfies all of our clients, especially by always implementing new requested features. But sometime, some requested features are just too niche to be relevant to all of our users, and since our R&D team consist of a finite number of people we cannot implement all of them.
In order to tackle this problem once and for all, we want to build a system inside Tuleap to allow administrators to execute their own code, with the aim of extending our platform’s functionalities. However, with this solution comes one big point of focus: security. It shouldn’t come as a surprise that letting people execute code on your servers without some sort of restrictions isn’t a good idea. Even if you only have to deal with people of trust, it’s good to make sure that nobody can crash the server by accident.
Aiming to implement this feature in Tuleap, we first set out to create a prototype, so as to be able to properly answer the question: How to execute untrusted code on our PHP based platform in a secure and timely manner ?
Last but not least, the resulting mechanism must guarantee these three technical constraints: security, performance and flexibility.
Bear in mind that this article is a high level overview of our work. For more details we advise you to take a look at our complete commented code on GitHub, the link is at the end of the article so you can keep on reading peacefully ! Futhermore, everything that we wrote related to PHP should be compatible with other languages, as long as they have a similar FFI system.
State of the art: WebAssembly at the rescue
To find the right technology for our needs, we conducted a state of the art analysis of the available solutions. We identified three technologies that seemed interesting:
- gVisor: Google’s sandboxed containers solution to run untrusted code
- firecracker: Fast micro-VMs developed by Amazon
- WebAssembly: This new, sandboxed by default, Cross-platform assembly language that everyone is talking about
We conducted various experimentations with gVisor and firecracker, both seemed very good, especially gVisor that was very simple to use and looked a lot like the containers that we are already using for Tuleap. Sadly, even though they satisfied our two most important constraints: security and performance, the flexibility was lacking.
Indeed, to better accommodate our clients, Tuleap runs in very diverse production environments (VMs, Docker container, Kubernetes, bare-metal). In the case of Virtual Machines, nested virtualization in not always an option, depending on the hosting service you’re using. It is also known to greatly reduce performance, this eliminates firecracker as a potential candidate. Nowadays, Tuleap usually runs in a Docker container or a Kubernetes cluster, we found out that executing a gVisor sandboxed container in this environment proved to be impossible without decreasing the level of security offered by Google’s technology.
WebAssembly
Disappointed, but full of hope for WebAssembly, we went on with our experimentations, but first, let us describe this technology in a few words:
WebAssembly (WASM), was first announced by the W3C in 2015, it is a standard that defines a bytecode, its textual representation alongside with a runtime environment inside a sandbox. WASM was designed to enable near-native code execution speed in web browser, but it doesn’t make any Web-specific assumptions and it can therefore be used outside of the browser. Moreover, since WebAssembly only specifies a low-level language, the bytecode is almost always produced by compiling a higher-level language (like C, Rust, Java…) to WASM, it is what we call a ’compilation target’. Since WebAssembly’s runtime environments are low-level virtual stack machines (akin to JVM) that can be embedded into host applications, some of them have found a way to standalone runtime environments.
We’re particularly interested in this technology because it provides sandboxing at an application level and other nice security features (avoiding the aforementioned problems with gVisor and Firecracker). It requires well-defined imports to interact with the host, it is a compilation target for higher-level languages and therefore simply asking for a WASM binary could allow us to support many different languages with ease.
In order to test this technology we used the most popular and well-maintained runtimes for WebAssembly, namely: WasmEdge, Wasmer and Wasmtime. We decided to kill two birds with one stone by testing the technology and comparing the different runtimes in the same process.
- To make sure that we would not run in the same flexibility problems that we encountered with gVisor and firecracker, we directly tested whether or not it is possible to execute a WebAssembly module on different platforms. Execution of WASM modules in containers and VMs worked out perfectly, one big constraint out of the way!
- While doing our first test we checked out the container’s execution time and distinguished important variations between runtimes. This gave us the idea to create a benchmark for the runtimes, doing this gave us the occasion to test the performance criteria but also to see which runtime was faster.
Here’s the results we obtained (note that we used a logarithmic scale to represent the time):
We can see that Wasmtime has an advantage when it comes to Ahead of Time (AoT) compilation, which we’re planning to use. - Last but not least, the final decision criteria was security.
The only runtime that is clearly addressing security outside of the default structure of WebAssembly is Wasmtime. There is a base layer of security features defined in the WebAssembly Core specification, but runtimes can add new features as well. Of the three runtime we’ve discussed, it is the only one to add defense in-depth features as well as providing some mitigation for the Spectre attack. Hence we can consider that Wasmtime is the best runtime in term of security, and the one that is the most fitted for the task at hand.
Our prototype
Now that we’ve decided to use Wasmtime to execute WebAssembly modules from Tuleap, we started to design and create our prototype.
Here’s the abstract design idea for the final product, it guided us to create our prototype:
Technical aspects
In order to build a realistic prototype, that we could easily extend to work with Tuleap, we chose to have a base layer in PHP. Wasmtime offers a CLI tool alongside with libraries to work with different languages (Rust, C/C++, Python, .NET and Go), unsurprisingly it does not offer support for PHP. Since Wasmtime is written in Rust and the focus is primarily put on the Rust library, it is the one we decided to use. With it, we wrote a program that can instantiate, pass input to a WASM module, execute it and retrieve the generated output. This program is our Rust library called wasmtime-wrapper-lib. To interact between PHP and our library we used PHP’s Foreign Function Interface (FFI).
The FFI is an extension that allows the loading of shared libraries (.DLL or .so), calling of C functions and accessing of C data structures in pure PHP. Even though we’re working with a Rust library, Rust’s cbindgen can generate a correct C header for our code, which is sufficient for PHP to be able to interact with our lib through FFI.
Before we dwell upon the technical aspects, here is an overview of what we created.
Our prototype takes a WebAssembly binary (a .wat, .wasm or precompiled .cwasm file) and the input it’s going to process (a JSON string), then the WASM module is executed safely using Wasmtime’s runtime and it returns the produced output.
It consists of three files:
- ffi-wasmtime.php: Since Tuleap’s backend is mostly written in PHP, this file is the starting point, it acts as a Tuleap instance so we can create a realistic prototype.
- wasmtime-wrapper-lib: This is a wrapper of the Wasmtime Rust library that we’ve written in order to simplify the interactions between PHP and Wasmtime. This is the piece of code that is responsible for the instantiation and execution of the WASM modules.
- add-json-rs: A rust program that we compile to WebAssembly and execute through ffi-wasmtime.php. It is our example WASM module to prove that everything is working correctly.
At first we thought that we could implement a small program to execute a WASM module directly in PHP, by simply calling some functions of the Wasmtime Rust lib through the FFI, when needed. But this proved to be much more difficult than anticipated. The very simple C parser from PHP prevented us from doing a lot of things, we encountered a lot of errors related to types when calling Wasmtime’s functions. Thus, we decided to write a Rust wrapper for the Wasmtime library. This wrapper only exposes one function. Since there is only one external function that we need to call from PHP, we could make it work much more easily. We detail how our wrapper works in the next section.
$ffi = FFI::load(__DIR__ . '/wasmtimewrapper.h');
$out = $ffi->compile_and_exec($wasm_path, $json_in);
$json_out = json_decode($out);Calling our Rust function from PHP
#include <cstdarg>
#include <cstdint>
#include <cstdlib>
static const uint64_t MAX_EXEC_TIME = 80;
extern "C" {
const char *compile_and_exec(const char *filename, const char *json);
} // extern "C"Content of the wasmtimewrapper.h header that make it possible for PHP to interact with our lib
The heart of our prototype: wasmtime-wrapper-lib
First off, we had to settle on an Input/Output format. We decided to use JSON, this choice is pretty logical because it is a data format that we already use a lot in Tuleap and it is easily serializable. On top of that, it is a very simple format to parse in every language and it gives us great flexibility. We can represent a lot of different arguments, with various types, by simply passing a JSON string to our WebAssembly modules.
With that settled, we started to work on the main part.
pub extern "C" fn compile_and_exec(filename: *const c_char, json: *const c_char) -> *const c_char {};The single function exposed by our wrapper
As input, it takes the name of the path to the WASM file and a JSON string. The function also returns a JSON string, so it can be exploited on the PHP side.
As you will see bellow, we have used two methods to pass data to the WASM module. Since the first one was too inconvenient, we opted to switch later on. Let us explain why:
This is because, at the time of writting, WebAssembly doesn’t have complex types like strings. You can find the full list of types here, most often, these are the ones you will see used in functions:
- i32 : 32 Bit Integer
- i64 : 64 Bit Integer
- f32 : 32 Bit Floating Point Number
- f64 : 64 Bit Floating Point Number
Thereby, getting data in and out of WebAssembly, specifically server-side WebAssembly run with Wasmtime, can be quite tricky. Hopefuly, the work that is currently being done on interface type proposal and the component model design will quickly change the situation. But for now we have to resort to these methods.
Initial approach:
Our first approach to solve this problem, the red path in the image, was to require the module to export two functions: one to allocate memory on the module’s memory, and another one to start processing what we input in the module’s memory (the ‘computation function’ if you will).
This method worked but was quite constraining, for a WASM module to be ‘compatible’ with our system, it had to export two functions with a precise prototype just to get a proper input. Not satisfied, we experimented with another method that proved to be much simpler and better in every way.
let alloc = instance
.get_typed_func::<i32, i32, _>(&mut store, ALLOC_FN).expect("expected alloc function not found");
let alloc_ptr = alloc.call(&mut store, alloc_size as i32)?;Exemple of our old method to fetch and call the exported ‘alloc’ function of the WASM module.
Current method:
The second method, the green path in the image, uses the WebAssembly System Interface (WASI) to access the standard input and output of the WebAssembly module. By doing this we do not need to pass complex types through functions, the program just needs to implement a main and get the input from stdin. This technique requires coming up with a data format which supports serialization and de-serialization, since we already opted to use JSON, that was perfect for us. It also abstract the memory aspects and doesn’t require to implement specific functions, this is why we decided to adopt this method and ditch the other one. Additionally, with this technique, implementing a logging system will be much more easier. When an error is encountered, the modules can simply write it to stderr, and we could forward this output to the Tuleap’s interface for the developer to see.
let stdin = ReadPipe::from(JSON_input.to_owned());
wasi: WasiCtxBuilder::new()
.stdin(Box::new(stdin.clone()))
.stdout(Box::new(stdout.clone()))
.inherit_stderr()
.build()We configure WASI to send the input JSON string to the stdin of the WASM module
let mut input: JsonData = serde_json::from_reader(stdin()).map_err(|e| {
eprintln!("ser: {e}");
e
})?;add-json-rs reads the input JSON string on stdin and serializes it with Serde
Limiting the resources a module can use
Now that we can safely execute a WebAssembly module, we have to limit the amount of resources it can use to avoid accidents and denial of service attacks. This was a rather simple process thanks to Wasmtime’s clever design.
Currently, to limit the execution time we decided to use the epoch_interruption mechanism. The concept is simple, we allocate a number of epoch y to the WebAssembly module, then, we start a separate thread that will bump the epoch after x amount of time. When x = y, the module is terminated. Because the goal is to execute modules that will impact the data displayed in Tuleap, it is really important for us to guarantee that this can be done in a timely manner. No ones wants to wait one second or more for a webpage to load, this echoes also with the security constraints mentioned above. At the time of writing, we set this limit to 80 ms, but this might change in the future.
To limit the amount of memory a WebAssembly module can use, we implemented the ResourceLimiter trait. By doing so we can put an upper limit on the amount of memory a module can allocate, for our prototype we chose 2 Mo, again this is probably going to evolve in the future. It is very important to limit this parameter, otherwise, malicious people could crash the whole server by allocating too much memory.
Demo WebAssembly module
To make sure that the process of creating a WebAssembly module for Tuleap is as simple as we envisioned, we created a demo program called add-json-rs. It is a program that we wrote in Rust and that we compiled to WebAssembly. It expects a JSON file of the following form as input:
{
"number1":<int>,
"number2":<int>,
"res":0
}With that input, it proceeds to add the values in the two first fields and put the result of the addition in the ‘res’ field, the updated JSON string is then returned. It is important to note that if the JSON does not have the expected form, or is improperly formatted, the module will return an error.
Here is what’s happening in this program in a chronological order:
Here’s the output in the CLI when executing add-json-rs with ffi-wasmtime.php:
Optimization pipeline
To guarantee that the WebAssembly binaries that we receive are as fast as possible, we run them through a small optimization pipeline. Firstly, we optimize our binary with the wasm-opt tool of the Binaryen suite. It conducts an impressive amount of optimization pass, to both reduce the size of the binary and make it faster, especially with the O4 setting that we’re using. You can check out the full list of all the passes here.
Afterwards, as we’ve discussed above, we use Wasmtime’s Ahead of Time (AoT) compilation feature. This speeds up the execution time dramatically, as we’ve shown in our benchmark up ahead.
We went from 22.59s for the execution of the vanilla add-json-rs module to a whopping 0.07s, for the module that went through wasm-opt and the pre-compilation ! All that said, wasm-opt only make us gain 0.01s here. When you add 2 integers, there is only so much you can do in term of optimization. We foresee much greater gain with larger modules. We will keep you updated!
Conclusion
To recapitulate, we now have a realistic prototype that’s capable of executing safely any WebAssembly module while handling their input/output. These modules only need to implement a main function, but there are no other expectations. Now it is time to make it interface little by little with Tuleap. For the first step of this process, we plan on linking our mechanism with the follow-up comments system. The goal being that users can execute their own code in order to operate transformations on the text field of follow-up comments.
Gradually we will link it with every part of Tuleap, making it possible to extend the functionalities of our platform to realise more and more complex tasks.
For more details, you can find the complete commented code on GitHub.
Stay tuned for more !
Sources
A big thank you to the authors of these ressources, they helped us a lot during our journey:
- Getting data in and out of WASI modules
- A practical guide to WebAssembly memory
- Wasmtime documentation
- From Rust to beyond: The PHP Galaxy
September 5, 2022 Modification: Syrius Akbary, CEO of Wasmer, noticed that the timings we observed when using Wasmer with AoT compilation were not expected. We have been able to confirm that the reported timings were indeed incorrect and updated our result chart. We have also published the code we have used to get those numbers if you want to reproduce the experience.