After several months of silence, I’m pleased to announce that we are starting to test the first usable version of Pijul.
We are not quite ready to release, as we’d like to get a bug reporting system at the same time as our first release, so that releasing is actually helpful for the community and for us, and no one gets too angry if we don’t fix problems in time.
In this post, I’d like to review some of the achievements which have led to this release.
Certainly the hardest part was to make Sanakirja production-ready. One thing that made it incredibly hard was that it all happened incrementally: I was first trying to write a drop-in replacement of LMDB in Pijul. With the existing Pijul, this meant using “binary strings” (or &[u8]) everywhere.
Noticing that this wasn’t really going anywhere, Florent then improved the API a lot by adding types everywhere. At the same time, I was having a hard time getting the memory management right: Sanakirja uses a B tree to store reference counts for blocks, and also needs to change the reference counts when changing B trees (including the “reference counter B tree”).
Around September 2016, I started to rewrite Sanakirja almost from scratch, which was completed in early December, including an important change in the format to merge Florent’s improvements at a deeper level: we are now using types to define the binary encoding of keys and values. In particular, since Pijul uses a lot of constant-sized strings, we can simply omit their sizes, and know it at compile time. Obviously, this means that Sanakirja databases need to be written and read by programs that use the same type.
This allowed us to rewrite libpijul’s backend module with things like:
type NodesDb = sanakirja::Db<self::key::UnsafeKey, self::edge::UnsafeEdge>;
pub struct Dbs {
nodes: sanakirja::Db<self::small_string::UnsafeSmallStr, NodesDb>
}In other words, our nodes database is now a dictionary mapping branch names to a graph, where a graph is a dictionary mapping a Key to an Edge. In the new Sanakirja API, all these are instances of trait Representable.
The main caveat is that there’s no way in Rust (because of the lack of HKT, see e.g. this blog post by Niko Matsakis) to correctly express lifetimes in the deserializing method (see read_value) of Representable. Indeed, these methods would need to return an associated type with a lifetime equal to the transaction’s.
In Pijul, for each type we wanted to store in the database, we have needed to define two types: the one actually used in libpijul, and an unsafe type equivalent to a raw pointer, private to our backend module. This is also the reason why all methods of that trait are marked unsafe and take raw pointers: they cannot possibly be safe anyway.
This situation is relatively unfortunate, as it makes it difficult to use Sanakirja in prototypes, where encodings tend to change a lot. However, in Pijul, where all the tables had already been thought through for a number of months, implementing this dual-type system took no more than one afternoon.
I started Thrussh in June 2016 after failing to write short bindings for the server part of libssh. And even though this project was started out of frustration, the Rust community recently turned it into a much better project than my most optimistic expectations!
In particular, two major improvements made it a fun project:
The first one was contributed by Brian Smith: he learnt darcs (which really impressed me) and sent me a replacement of the crypto library I was using (libsodium) with his *ring* crate.
I’m really grateful he took the time to understand my code, and I’m amazed by this contribution. *ring* is in my opinion (but I’m not a cryptographer) a great way to use Rust for cryptography, as (1) it complies with all of the arguments usually used to justify the use of C for crypto, but (2) avoids exactly that class of problems in C that have led to so many important security issues in other crypto libraries.
I was proud to be invited to the Rust Belt Rust conference, where I could meet awesome fellow rustaceans.
In particular, Paul Colomiets, who was also a speaker there, gave me a strong argument to use the tokio and futures crates instead of my own event loop.
His argument was as follows: in an asynchronous implementation of a protocol, an event loop processes events, and usually calls “callback functions” in reply to these. What Paul made me notice, is that mixing several protocols is non-trivial in such a scheme. For instance, if the callback code uses a different protocol to communicate with a database, how can we use the same event loop to process that protocol?
Then, Alex Crichton and Carl Lerche ran a workshop on the topic at Rust Belt Rust, which I used as an opportunity to try and move Thrussh to tokio, as I could ask the dumb questions I needed to answer before I could do it.
I want to share a few observations about this process:
Implementing asynchronous functions using the Future trait made me notice a few bugs in my asynchronous code. This is because the poll method cannot be mixed with other stuff, which highlights the asynchronous parts (in particular the reading parts) of the code. This helped me see new cases where NotReady errors could occur, which I was not handling before.
One cool feature of Tokio is that not everything has to be asynchronous from the beginning. What helped me a lot was that I could implement asynchronous parts gradually, and test each new improvement individually.
The main downside is that I’ve had to hard-code the use of network sockets instead of Read and Write. Of course, SSH is usually run over sockets, but this complicates testing: indeed, I was planning to run the protocol in pairs of Vec<u8> in the tests, and now I have to open ports.
Obviously, it was hard for me to resist using my new asynchronous callbacks, so I wrote a futures-based minimal http server (in part inspired by minihttp), and a postgreSQL driver (I’ll release it soon, I promise) to authenticate users, and it works!
Conclusion: Tokio and Futures really rock.
After getting useful feedback on my prototype PGP crate, we decided to drop the support for PGP. Florent also made the important point that signing patches can be harmful (because anyone knowing the patch and the signature could apply it to a branch on the patch author’s behalf): what we need to sign is the action of applying a patch to a branch, which is fulfilled by SSH.
When we have a bug tracker, i.e. really soon. My current prototype of the nest already accepts patches pushed via SSH, and can display files from a repository. I just need to tweak a few things, test a bit, and we should be ready.
The hard bits are behind, our tests are working, our theory is consistent. Quite a relief, after more than a year of work.