Lecture 4

Contents

Unsafe Code

This section is a very short version of what you can also read in the rust book

In rust, memory safety is checked by the compiler. It does this using a complicated set of rules that will reject some code and accept other code. However, not all memory safe code is accepted. The rules that govern the memory safety are a bit too conservative. Especially when you want to start doing low-level things, you may run into this.

However, in Rust it is possible to temporarily disable some (but not all!!!) of the compiler's checks so you can do things which you know is safe but the compiler cannot check. We mark such sections with the keyword unsafe:

fn main() {
unsafe {
    // this code has fewer checks
}
}

Inside these unsafe blocks, you are allowed to do the following things:

  • Dereference a raw pointer (can lead to user-after-free or buffer overflows!)
  • Access or modify a mutable static variable (can lead to race conditions!)
  • Access fields of unions (this allows you to interpret data that should be an int as for example a float)
  • Call an unsafe function or method (an unsafe function can do any of the above)
  • Implement an unsafe trait (some traits have certain requirements which will cause memory unsafety when not satisfied)

Raw pointers are something different than references. We write references as &T or &mut T while we write a raw pointer as *const T or *mut T. However, unlike references, pointers can be null, may alias (even *mut) and rust won't make any assumptions about them

As mentioned earlier, using unsafe code doesn't imply that it's wrong. In other words, not all unsafe code is unsound. However, safe code can never be unsound, if it were the compiler would catch it. Inside unsafe blocks, the compiler will not always detect unsound code.

Undefined behavior

Some operations in a language are explicitly undefined behavior. That's not because the designers of the language were lazy, but because this gives the compiler freedom to optimize certain programs. For example:

fn some_fn() -> Option<u32> {unimplemented!()}
fn main() {
let a: Option<u32> = example();
a.unwrap();
}

The unwrap call here has to check whether a is Some or None. However, if we know that example will never return None we can use unsafe{a.unwrap_unchecked()} instead. When a is Some(x) this will return x. However, what happens when suddenly a turns out to be None anyway? In unwrap_unchecked, this is undefined behavior.

With unwrap, this would check this and panic. But with unwrap_unchecked, the compiler is perfectly allowed to leave out that check (hence it is unsafe to use). What happens when you do use it on None? We don't know. Most likely, it will just give us a random integer here, from somewhere else on the stack. This is exactly what you are responsible for when you use unsafe code. You should make sure that this can never actually happen (by for example ensuring example really never can return None).

Assumptions

Often, unsafe code relies on certain assumptions. These assumptions may be true in your code, but the compiler cannot reason about them being true in general. For example:

  • This variable can never be zero
  • This pointer is always within bounds
  • This pointer is never null
  • This variable is only ever accessed from one thread
  • There is only one core on this machine
  • Interrupts are turned off here
  • This memory is still allocated
  • This function doesn't decrease the value of my variable

In unsafe code, you, the programmer are responsible for making sure that in your code your assumptions are true. You can do this by explicitly checking that they are true like below:

#![allow(unused)]
fn main() {
 // unsound when a is null
unsafe fn dereference(a: *mut usize) -> usize {
    *a
}

// TODO: is this the only assumption we have to check?
fn safe_dereference(a: *mut usize) -> usize {
    // make sure a is not null
    assert_ne!(a as usize, 0);
    unsafe {
        dereference(a)
    }
}
}

Alternatively, you can just reason through your code to prove that your code cannot break your assumptions.

These two techniques are the same as what the compiler does: reasoning and inserting checks. However, in unsafe code that's your responsibility, since the compiler cannot always reason as well as you can.

In the code example above, we also see a function marked as unsafe. That means that the function can only be called inside of an unsafe block. This is often done when the soundness of a function depends on the caller making sure that certain conditions are true

IMPORTANT

When you use an unsafe block, it is often a good idea to put a comment above reasoning through your assumptions and why they are true. If you are then later debugging, you can check your assumptions again.

More importantly, it is a very good practice to put such comments above unsafe functions explaining what a caller must check before it is sound to call that function.

In fact, clippy, Rust's linter can warn when you don't have such a comment

Abstraction

Almost everything in Rust is an abstraction over unsafe code. Abstractions over unsafe code can be fully safe to use, as long as they make sure to check all their assumptions before running unsafe code.

For example, a Box has to call malloc and free to allocate and deallocate memory. These functions give back raw pointers, which sometimes may even be NULL. However, Box will check that this is not the case. As long as you have access to the Box, the Box will also not free the memory associated with it. That means you can never accidentally use the memory after it is freed (since that can only happen when you have no access to it anymore).

Similarly to how Box abstracts over managing memory:

  • Mutex abstracts shared mutability
  • Vec abstracts growing and shrinking memory
  • println! and File abstracts system calls
  • std::io::Error abstracts errno
  • Channel abstracts

Foreign Function Interfaces

Sometimes, it becomes necessary to communicate with programs written in other languages besides Rust. There are two good options for this:

  1. Passing around messages through some kind of shared channel. This channel could be a network socket, http api, unix socket, file, unix pipe, etc. As long as source and destination agree on what bytes mean what this works.
  2. Going through C

The latter is what we will talk about in this section. C has become, besides just a programming language, also a common way of communicating between languages. Many languages provide some kind of API which allows that language to call a C function. In some languages that's easier than in others (python is very easy for this for example), but it often exists. Data passed from and to C functions will be converted to use C's data representation.

This approach has some downsides. For example, we are limited to the data types which C supports. Some datatypes in Rust for example aren't very nicely supported. For example, enums or slices. But that's not exclusive to Rust, C doesn't really have a concept of classes for example. So any language that uses classes has to work around that.

Such interfaces to other languages like C are called foreign function interfaces. They work by linking code together. Each language uses their own compiler to compile their own code, and later the resulting object files are pasted together into one big binary. The individual object files will define symbols (basically names of things in them, like functions).

Some object files will be "looking" for a symbol that they don't define themselves. The linker will connect the object files such that when one object file is "looking" for a symbol and another provides that symbol, the resulting binary connects those two together.

Above we describe static linking. There's also dynamic linking, where at runtime the program can import more symbols from a shared object file.

When a program in another language (like C) wants to use some Rust code, that code will be "looking" for a certain symbol during linking.

// says "there exists a function somewhere called do_rust_thing".
// however, we did not define it in c. Later, during linking
extern void do_rust_thing(int a);

int c_thing() {
    // call that function
    do_rust_thing(int a);
}

do_rust_thing is declared here, but we gave no definition. However, we can do that in Rust. As long as the name is the same, the linker will resolve this.

#![allow(unused)]
fn main() {
use std::ffi::c_int;

pub fn do_rust_thing(a: c_int) {
    println!("{}", a);
}
}

However, here we have to be careful. Rust uses a process called name-mangling. In the final object file, there won't be a symbol do_rust_thing. Instead, there will be something like _ZN7example13do_rust_thing17hccc9c26c6bcb35a8E (which you can see does contains the name of our function). This mangling is necessary since in Rust it's possible to define multiple different functions with the same name. This for example happens when you implement the same trait for multiple types.

We can avoid this by using #[no_mangle]

#![allow(unused)]
fn main() {
use std::ffi::c_int;

#[no_mangle]
pub fn do_rust_thing(a: c_int) {
    println!("{}", a);
}
}

Next, C uses a different ABI than Rust. That basically means that the way it passes arguments to functions and gets return values from functions is slightly different. Different ABIs may for example use different registers for certain parameters, and C supports functions with a variable number of argument, which Rust does not.

That means that for all the parameters to do_rust_thing to be interpreted correctly by both sides, we need to make sure that the ABI is the same. The easiest in this case is to ask Rust to change the ABI for the do_rust_thing function, which we can do as follows:

#![allow(unused)]
fn main() {
use std::ffi::c_int;

#[no_mangle]
pub extern "C" fn do_rust_thing(a: c_int) {
    println!("{}", a);
}
}

You may have noticed that we use a type called c_int here. That's a type that is guaranteed to be equivalent to an int in c.

Here we specify the ABI to be "C". There are other ABIs we can choose, but we want this function to be callable from C so that's what we will choose.

We can do the same the other way around. In Rust we can say that we "expect" a c function to be defined somewhere. We can write that as follows:

use std::ffi::c_int;

// tell the Rust compiler to expect 
extern "C" {
    pub fn some_c_function(
        a: c_int,
    );
}

fn main() {
    unsafe {some_c_function(3)}
}
#include <stdio.h>

void some_c_function(int a) {
    printf("%d", a);
}

Again, we have to specify that we're working with the "C" ABI. Inside the extern block we can define a number of function signatures without a function body. In our main function we can now simply call this function, the linker will take care of actually connecting this to the function defined in C. As you can see, calling a C function in Rust is actually unsafe. That is because the c compiler has much fewer safety guarantees than Rust has. Simply calling a C function can cause memory unsafety.

To actually make sure the programs are linked together, we need to instruct the compiler to do some work for us

When we want to call Rust code from C, we may need to compile our Rust binary as a "shared object file" and we can even use cbindgen to automatically generate a C header (.h) file.

Alternatively, we may want to automatically compile and link our C files when we want to call functions in that C file from Rust. We can use build scripts (little Rust programs that are run before compilation, which can for example call another compiler) to automatically compile C files as part of cargo run. A crate called cc can be used to actually call the c compiler.

Nightly features and release cycle

The Rust compiler is made in Rust. That means that to compile a new Rust compiler, you need an old Rust compiler. This means that when new features are added to Rust, they can't immediately be used in the Rust compiler sourcecode itself, since the older compiler doesn't support them yet. This process of using an old compiler to create a new compiler is called bootstrapping.

Rust compilers are constantly being built and published. Most often there's the nightly compiler, which is automatically created every night base on the current master branch of rust's git repository. You can use these nightly compilers if you want to use upcoming features or still unstable features of the Rust language.

Then, every 6 weeks, the current nightly released is branched off which becomes the beta release. Beta releases are mostly meant for testing. If bugs are found in beta, they are removed before they get into stable, the last type of rust compiler we have to talk about. New stable compilers are released every 6 weeks (at the same time as the next beta branches off of main). The hope is that most bugs in new features are removed when a stable version is released. Therefore, stable Rust is the version you'll use most often.

You can read more about this here

There is however a very good reason to use nightly sometimes. New features are constantly added to Rust. Even when they are already fully implemented, they may not be scheduled to be in the next beta or stable compiler for a long time. However, with a nightly compiler you can enable these experimental features anyway! (use at your own risk)

For example, it is currently (sorry if this is outdated by the time you're reading this!) not possible to create a type alias with impl SometTrait in them For example, the following code is not legal.

#![allow(unused)]
fn main() {
type U32Iterator = impl Iterator<Item = u32>;
}

However, there does exist an implementation of this in the nightly compiler. There's an issue for it and although it's not yet stabilized and there's little documentation for it, it is fully implemented.

To use it, you download a nightly compiler (rustup toolchain add nightly), put this annotation at the top of your main.rs or lib.rs file:

#![feature(type_alias_impl_trait)]

and compile with cargo +nightly run. Now the example from above is accepted by the Rust compiler.

Cross compilation

Cross compilation in Rust allows us to compile our Rust programs for different target architectures, such as ARM or RiscV, or maybe just for different operating systems (compiling for Windows from Linux) without needing to have that target architecture (or operating system) on the machine we compile on. This can be useful for creating binaries for devices with limited resources (maybe it's not possible to compile on the target device itself!) or for testing our programs on different architectures (within emulators for example).

To set up cross compilation in Rust, we need to first install the necessary tools for the target architecture. For an ARM architecture, this may include the ARM cross compiler and the necessary libraries. We can do that through rustup:

rustup target add thumbv6m-none-eabi

Once the tools are installed, we can add the target architecture to our .cargo/config file. This file allows us to configure cargo, the Rust package manager, to automatically cross compile our programs.

To add the target architecture to our .cargo/config file, we can add a section like the following:

[build]
target = "thumbv6m-none-eabi"

This configures cargo to use the target we want it to: thumbv6m-none-eabi which is also the one we just installed.

We could also add a runner for this specific architecture by adding the following in the same .cargo/config file:

[target.thumbv6m-none-eabi]
runner = "run some command here, maybe start an emulator"

The command gets the location of the compiled binary as its first parameter.

Now we can simply run

cargo run

This will automatically use build the binary for the ARM architecture and run it with our configured runner.

Messaging protocols

When we talked about foreign function interfaces, we briefly mentioned the fact that sometimes we can also use some kind of channel to communicate between languages. For example, TCP or UDP sockets, an HTTP api, unix sockets, the list goes on. Even the UART interface you are making is a communication channel. Of course such communication channels are not just useful for communicating between languages. Maybe you just want to communicate between two programs on the same, or on different machines. For the assignment you have to talk from an emulator to a host computer over UART.

However, we can't just send anything over a communication channel 1. Bytes or byte sequences or are usually easy, but this depends a bit on what kind of communication channel you are using. In any case, sending things like pointers or references is usually quite hard. They're just numbers you might say! But they refer to a location on your computer. For a pointer to make sense to someone we send it to, we also have to send whatever it is pointing to. Let's assume we use UART which only transmits single bytes. How can we transmit arbitrary types over such a communication channel?

The process of converting an arbitrary type into bytes is called serialization. The opposite process of taking a sequence of bytes. Very simple serialization you can do by hand. For example, you can just turn a u32 into 4 bytes. However, anything more complex can quickly become hard. Luckily, there is serde. (literally serialization deserialization).

With serde we can annotate any type that we want to be able to serialize. It will automatically generate all the required to serialize it. But not just into bytes. Serde always works in two parts. It always requires some kind of backend that takes types annotated by serde, and turns those into the actual serialized representation. That's not just bytes.

So, let's look at an example on how we can turn our types into bytes:

use serde::{Serialize, Deserialize};
use postcard::{to_vec, from_bytes};

#[derive(Serialize, Deserialize, PartialEq)]
struct Ping {
    // Some data fields for the ping message
    timestamp: u64,
    payload: Vec<u8>,
}

fn main() {
    let original = Ping {
        timestamp: 0x123456789abcdef,
        payload: vec![0, 1, 2, 3, 4, 5, 6, 7, 8],
    };

    let serialized = to_vec(&original);
    
    // serialized now is just a Vec<u8> representing the original message
    
    // can fail because our bytes could not represent our Ping message
    let deserialized: Ping = from_bytes(serialized.deref()).unwrap();
    
    assert_eq!(original, deserialized);
}

Serde will recursively go through your structs and enums and make sure that when you serialize a type it will be turned into its bytes in its entirety.

Usually, for these kinds of protocols, it's useful to make our message type an enum. That way we can add more message types later by just adding variants to our enum.

With Serde, both the sending end and receiving end need to agree on the datatypes they use. When the sending end sends a message with a datatype containing an u8, while the receiving end has a type with a u32 in that place, deserializing will fail.

Therefore, it can be useful to make the messages you send a library that's imported by both the sending end and the receiving end. That way the types are guaranteed to be the same.

Starting and ending messages

When sending messages over a channel like UART, you have to be careful. UART only allows you to transmit and receive single bytes of information. How does a receiver know where messages that take up multiple bytes start? There are multiple options, for example you could define some kind of header that defines the start of a message. You have to make sure here that the header never appears in your message which can be hard. You might need to do some escaping/byte stuffing/bit stuffing. It may also help to include the length of the next message at the start of every message to let the receiver know how many bytes to expect.

In any case, it's impossible to have the messages that are transmitted to have all the following properties:

  • any byte can be sent over the wire. Data bytes are always seen as data and control bytes are always seen as control bytes
  • simple to transmit. It's easiest if we can just write our serialized message directly, however if we have to do escaping then we may need to analyze the message first
  • easy to find starts and ends of messages

Message integrity

For many, if not all systems it's critical that we don't misinterpret messages we receive. Therefore we want to make sure that whatever we receive is actually the message that the receiver wanted to send. Imagine if a few bytes are corrupted and instead of putting a motor on a drone to 20%, we put it to 100%.

To ensure that what we receive is actually correct we can use checksums. A checksum is a sort of hash. The sender calculates a number, usually a few bytes based on what they send. The receiving end then does the same for the bytes that it received. If the checksum the receiver calculates does not match the checksum that was sent along with the message, the receiver knows that some part of the message was corrupted (or maybe the checksum itself was corrupted!)

What the receiver does with that may depend on the message. It may choose to ignore messages like that, or could ask for a retransmission.