Topics:

Serving PyTorch Predictions with Rocket via @DanielPBank

Repo: https://github.com/danielbank/rust-tch-server

I made an example web server with Rocket which serves up linear regression predictions using tch, a crate providing Rust bindings for PyTorch. The project consists of two parts:

  • An example program called train which trains a linear regression model and saves its weights in a file so that it can be loaded.

  • The main program, a web server that responds to POST requests to /predict with a prediction.

I found that the best place to learn how to use tch to make a linear regression model was the tch project's tests.

The main challenge I faced in making the example was when I wanted to utilize the linear regression model in the prediction route. The model is created in main():

let mut vs = nn::VarStore::new(Device::Cpu);
let linear = net(&vs.root());

The first (naive) approach I attempted was to pass it directly to the prediction route by having Rocket manage it as state:

rocket::ignite()
  .manage(linear)
  .mount("/", routes![index]).launch();

The type of linear is impl Module or "a type that implement nn::Module". Trouble arises when I try to wrap this in State in the definition for the route handler:

fn predict(bmi: Form<BMI>, classifier_state: State<impl Module>) -> std::string::String { ... }

This line then reports an error:

`impl Module` cannot be shared between threads safely

help: consider further restricting this bound: ` + std::marker::Sync`rustc(E0277)
main.rs(33, 46): `impl Module` cannot be shared between threads safely
state.rs(106, 32): required by this bound in `rocket::State`
`impl Trait` not allowed outside of function and inherent method return types

In this case, the advice for this error is not particularly helpful because further restricting the bounds only changes the error:

the size for values of type `(dyn tch::nn::Module + std::marker::Sync + 'static)` cannot be known at compilation time

doesn't have a size known at compile-time

help: the trait `std::marker::Sized` is not implemented for `(dyn tch::nn::Module + std::marker::Sync + 'static)`

The solution I found was to use a mutex to ensure that shared access to the model in the route and main thread is protected. The main() function becomes:

rocket::ignite()
  .manage(Mutex::new(Box::new(linear) as Box<dyn Module>))
  .mount("/", routes![index, predict]).launch();

And the definition of the route handler becomes:

fn predict(bmi: Form<BMI>, classifier_mutex: State<Mutex<Box<dyn Module>>>) -> std::string::String { ... }

Another solution would be to make a concrete type that implements tch::nn::Module and std::marker::Sync. This would take a little more work but has the advantage that the route handler definition would be a little clearer with State<MyConcreteType>.

Chris notes that the Shared State section of the Rust book recommends using Arc<Send + Sync>.

Extending a File Renaming CLI Tool via @RyanFickenscher

Ryan shared his contribution to fdname, a CLI tool for batch renaming files and directories. He needed the ability to rename file names as hashes (to prevent duplicate files) as well as removing whitespace from the file names. In order to implement his changes, he needed to learn how to use a hasher.

Boa JavaScript Lexer, Parser, Compiler built with Rust via @RyanFickenscher

Another project that Ryan is contributing to is Boa, an experimental Javascript lexer, parser, and compiler written in Rust.

One point that came up during his talk was the question how this project is different from Deno. Deno is a secure runtime for JavaScript and TypeScript. Whereas Deno has portions of its architecture written in Rust like the Tokio thread pool and its execution context (deno::Isolate), it is still using the V8 engine which is implemented in C++.

Deno Architecture Diagram

Rust on PyGamer Updates via @jacobrosenthal

Jacob created a new repository for the PyGamer board support crate. This makes it easier to get started than trying to find the pygamer folder nested two levels deep in the atsamd repo.

He also made improvements to the hf2 command so now you can simply cargo run --example blah or hit the Play button in VSCode to build and upload to the board.

One final improvement Jacob made in the examples repo was to light the red LED on the back of the PyGamer during a panic by stealing the LED from the underlying device singleton. You won't know where you panic'ed but at least you can know that you did.

Smol Async Example via @jacobrosenthal

smol is a fast async runtime with a small API. It was made to address the ecosystem split between tokio and async-std async libraries. The author wrote a blog post about why he built it.

Jacob made an example program using smol which monitors an SHTC3 sensor on Linux in the terminal. The program shares a thread between 3 different tasks, polling a sensor in one async task, printing it in another async task, and watching for control-c signal in a third, joining them all with a futures::or!.

smol::block_on is single-threaded by default. An alternative would be to spawn three threads for each of the processes. However why should we deal with the overhead of additional syscalls?

smol::channel::unbounded gives you a sender and receiver:

    smol::block_on(async {
        ...
        let (sender, receiver) = smol::channel::unbounded();
        let _ = futures_micro::or!(ctrl_c.recv(), poll(sender), show(receiver)).await;
    });

smol::Timer::after is the async version of std::thread::sleep

async fn show(
    receiver: Receiver<(Measurement, Measurement)>,
) -> Result<(), smol::channel::RecvError> {
    loop {
        // Drain any data updating the buffer
        for (normal, _) in receiver.try_recv() {
            println!("{:?}", normal);
        }
        smol::Timer::after(UI_REFRESH_DELAY).await;
    }
}

Caveat on Using rust-analyzer in VSCode

rust-analyzer is an experimental modular compiler frontend for Rust. It's really good but if you are using it in VSCode, there is an important caveat: As of August 26th 2020, it has not been merged with the Rust language extension. You need to uninstall the Rust language extension when using rust-analyzer. It should get merged eventually.

Dark Forest via @BlaineBublitz

Blaine wasted his week playing / editing the open source, serverless, smart contract game Dark Forest instead of writing any Rust.

Crates You Should Know

  • tch: Rust wrappers for the PyTorch C++ api (libtorch)
  • ctrlc: Ctrl-C handler that works on Unix and Windows