Topics:

Rants via @NelsonChen and @BlaineBublitz

Too Many Similar Crates, No Official One

Nelson rants that there are too many wrapper crates for the Windows API, all of them are subtely different, and none of which providing everything he needs. The closest one is winapi.

Daniel asked what kind of support Microsoft might be providing, considering projects like WinRT, Rust for the Windows Runtime. Here though, WinRT is wrapping the COM interfaces which is a higher level abstraction than the Windows API.

Macros are a Crutch

Blaine rants that macros are a crutch used by Rust developers when they don't know how to make a good code API. Libraries like Diesel particularly seem run afoul of complicated macro hell.

Jacob notes that the only ways to implement config are to either use a config struct or builder pattern. If your API doesn't make sense with either of these constructs, you are kind of screwed and macros are your only solution. This is something that Raph Levien mentioned in his wishlist for Rust in 2021:

Fourth, if I could have one wishlist from the Rust language, it would be proper keyword arguments, as the builder pattern feels clunky, and the struct-with-default pattern is no better. But overall there's a lot to be said for the language itself basically being stable, and the primary focus being on implementation, tools, and ecosystem.

Linking with C Libraries via @ChristopherSebastian

Chris was having some major problems with Rust builds when trying to link against a (fairly complex) C library. Basically, the problem boiled down to the fact that the C library consists of a bunch of static libs (.a) files and they re-define common symbols. By default, Rust uses the "--whole-archive" linker option which causes these redefinitions to conflict. Also, the C libs have cyclical dependencies and they really need to be linked with "--start-group" but again Cargo gives no way to control this. Finally, the only way to communicate with Rust's linker is to use cargo rustc linklib which limits the ability to control how things are linked.

It's actually easy to manually issue the right command and get the thing to build -- but if we are building a crate which needs to be easily usable from other Rust projects, we need to find a nice cargo-compatible build solution.

Chris's solution consisted of two steps:

Use an Archiver to Make a Library Rust Can Use

Chris finally ended up using the cc crate which figures out how to build C++ files into a static archive and the ar crate to convert the complex libary into a simpler, merged libary that is compatible with Cargo. One important note is that he needed to use a Thin Archive, as a Standard Archive will not work because it is too deeply nested.

Cross Compile for Windows using a DLL that Exports a Simple API

The build process has an extra step for supporting Windows. Since Windows uses /GS for buffer security checks as well as some other flags that MinGW doesn't support, compiling produces unresolved symbols for things like security cookies, etc. To get MinGW to work, Chris made a DLL (compiled with Visual Studio) which understands the security checks and makes a simplified API that MinGW can use.

While initially feeling like the Rust toolchain was not providing enough, Chris came to appreciate the level of abstraction that Rust provided. If they tried to do more, it would be too much. He didn't end up needing a fancy API after all.

Code Summary

// ==== build.rs ====
// We manually run the 'cc' compiler and 'ar' archiver to convert our complex
// C libraries into a simpler, merged library that is compatible with Cargo's limited
// linker abilities.  Then we tell Cargo about the lib with the "cargo:rustc-link-..." prints.
//
// NOTE: The stdout of this script gets stored at target/debug/build/$name-$hash/output.

use std::path::PathBuf;

fn run<S:AsRef<str>>(cmd_str:S) -> Result<(), std::process::ExitStatus> {
    // Split 'cmd_str' by spaces and create a Command object:
    let mut pieces = cmd_str.as_ref().split(' ');
    let mut cmd = std::process::Command::new(pieces.next().unwrap());
    for piece in pieces {
        if piece.len()==0 { continue; }
        cmd.arg(piece);
    }

    // Run the Command and report the result:
    let status = cmd.status().unwrap();
    if status.success() {
        Ok(())
    } else {
        Err(status)
    }
}

fn main() {
    // Helpful for debugging and understanding our context:
    assert!(std::process::Command::new("env").status().unwrap().success());

    let proj_dir : PathBuf = std::env::var("CARGO_MANIFEST_DIR").unwrap().into();
    let out_dir  : PathBuf = std::env::var("OUT_DIR").unwrap().into();
    let lib_dir            = proj_dir.join("x64_lsb-11.17.1.0_v6");

    run(format!("cc -I{lib_dir} -c {proj_dir}/src/c_api.c -o {out_dir}/c_api.o",
                proj_dir=proj_dir.to_str().unwrap(),
                lib_dir=lib_dir.to_str().unwrap(),
                out_dir=out_dir.to_str().unwrap(),
               )).unwrap();

    run(format!("cc -I{lib_dir} -c {lib_dir}/x64_lsb/lm_new.c -o {out_dir}/lm_new.o",
                lib_dir=lib_dir.to_str().unwrap(),
                out_dir=out_dir.to_str().unwrap(),
               )).unwrap();

    // Note that we are using a "thin" archive to bundle all libs together because Rust's linker controls are extremely limited and unable to handle this kind of circular-dependency + redundant-definitions structure:
    run(format!("ar crT {out_dir}/libflexbundle.a {out_dir}/c_api.o {out_dir}/lm_new.o {lib_dir}/x64_lsb/liblmgr_pic.a {lib_dir}/x64_lsb/libcrvs_pic.a {lib_dir}/x64_lsb/libsb_pic.a {lib_dir}/x64_lsb/libFNPload_pic.a {lib_dir}/x64_lsb/liblmgr_dongle_pic.a",
                lib_dir=lib_dir.to_str().unwrap(),
                out_dir=out_dir.to_str().unwrap(),
               )).unwrap();

    println!("cargo:rustc-link-search={}", out_dir.to_str().unwrap());
    println!("cargo:rustc-link-lib=flexbundle");
}

An Async Rocket Server with OAuth, Cookies, Diesel, and Templates via @mysteriouspants

Repo: https://github.com/iDevGames/uDevGames.com

Chris worked on an async Rocket server for a gaming community he is a part of. Some of the features of the server are:

GitHub OAuth Authentication

Chris rolled his own OAuth client instead of using the oauth2 crate. GitHub's docs are so awesome that it is easy and fun to implement the OAuth dance yourself.

Diesel ORM for SQL

Chris didn't like sqlx because it seemed like it was doing too much with the preprocessor and proc_macros. So instead he used diesel even though he had some issues with it's API. Specifically, Chris found it difficult to figure out what DSL to use or DSL type you get back from a function call. He thought they could learn a lot from the active record pattern from Ruby on Rails.

One benefit of using diesel was the diesel::embed_migrations which reads the migrations at compile time and puts them in the binary. This makes it easy to execute migrations on the remote server without needing to cart all the .sql files around. Simply scp up the server binary and run it with a migration command.

Async Rocket

Chris played with using rocket's async API. In his main(), he used the rocket::main macro, an alias for diesel is not async. Then the route handlers could use async await commands:

async fn get_user_detail(
    gh_client: &ReqwestClient, access_token: &String
) -> Result<UserResponse, GetUserDetailError> {
   let r = gh_client.get("https://api.github.com/user")
        .header("Authorization", format!("token {}", access_token))
        .header("Accept", "application/json")
        .send()
        .await?
        .json()
        .await?;

    Ok(r)
}

Templating

Chris used tera for a templating engine and Rocket's template context.

Dark Forest Game via @jacobrosenthal and @BlaineBublitz

Repo: https://github.com/phated/sophon

Dark Forest, named after the book with the same name, is a serverless game based on Ethereum smart contracts. The game is similar to EVE Online, where you are exploring and conquering planets in the universe.

Mining

There is a fog of war around your planets and the visible area of the universe is initially tiny. You have a miner which you can place at any coordinates and it reveals what's there. It is called a miner because it is performing a similar function as mining Bitcoin, it computes hashes for a piece of data until it finds a signed hash where the first N bits begin with a 10000000.... The hash result of the coordinates determine what is there (a planet, an asteroid field, etc). It can also give you other information...

Moves made by players are communicated on the blockchain. This leaks meta- and actual data about the player. Once you have found computed the hash result for a given (x, y) coordinate, you can use it with this information to monitor the activity of other players (moving energy between planets, inactivity, etc).

Optimizing the Miner

The game's client is open source and can be edited. There is no way to cheat with the miner code because all it does is move to a coordinate and compute a hash result. However, the code can be optimized to compute more hashes per second (a more efficient miner). There is even a Rust-based miner that is available on GitHub.

The Rust miner is a Rocket server with an entry point that is calling a main hash function.

Jacob was iterating on it and did some benchmarking to compare his code's performance (e.g. how much time is being wasted on the scheduler, how much is rayon taking, etc.). In addition to measuring results for different changes in the architecture, Jacob also measured results against different block_sizes for the hash function to see which block_size yielded the best results. Another way he analyzed performance was to use flamegraph to check which function calls were taking the most time.

Chris mentioned that, for serious benchmarking, it is important to control the conditions under which the performance test is run. He provided the following commands he uses to control his cpu-scaling and frequency, in order to stabilize benchmarks:

cpu-show:
        for P in /sys/devices/system/cpu/cpufreq/policy*; do echo $$P; echo scaling_governor=$$(cat $$P/scaling_governor) scaling_min_freq=$$(cat $$P/scaling_min_freq) scaling_max_freq=$$(cat $$P/scaling_max_freq
) scaling_cur_freq=$$(cat $$P/scaling_cur_freq); done

cpu-normal:
        # Run with 'sudo'.
        for P in /sys/devices/system/cpu/cpufreq/policy*; do echo powersave >$$P/scaling_governor; cp $$P/cpuinfo_min_freq $$P/scaling_min_freq; cp $$P/cpuinfo_max_freq $$P/scaling_max_freq; done
cpu-fast:
        # Run with 'sudo'.
        for P in /sys/devices/system/cpu/cpufreq/policy*; do echo performance >$$P/scaling_governor; cp $$P/cpuinfo_max_freq $$P/scaling_max_freq; cp $$P/cpuinfo_max_freq $$P/scaling_min_freq; done
cpu-slow:
        # Run with 'sudo'.
        for P in /sys/devices/system/cpu/cpufreq/policy*; do echo powersave >$$P/scaling_governor; cp $$P/cpuinfo_min_freq $$P/scaling_min_freq; cp $$P/cpuinfo_min_freq $$P/scaling_max_freq; done

Miner Manager with Native Node Modules in Rust

Blaine made a port of the game's Node.js server that runs the miner manager. He initially tried re-engineering it in Rust but quickly gave up because it was difficult to adapt the promise logic of Node.js to async paradigms in Rust.

Instead, he used Neon which provides automated Rust bindings for writing safe and fast native Node.js modules. This allowed him to use Rust for its typesafe for when performance mattered, and Node.js when he needed the cheap asynchronicity.

Crates You Should Know

  • winapi: FFI Bindings to all of the Windows API
  • cc: Library to compile C/C++/assembly into a Rust library
  • ar: Library for encoding / decoding Unix archives (.a) files
  • neon: Rust bindings for writing safe and fast native Node.js modules