Binary compatibility is a feature which is often overlooked in programming languages. These days, many languages use some form of interpreter or virtual machine (e.g. the JVM, Python’s interpreter, the CLR), which means they don’t have to deal with it.

However, in languages which compile to native code (e.g. C, C++, D, Plinth), there are often many restrictions on what a programmer can change in a library without breaking binary compatibility.

What is binary compatibility?

Binary compatibility relates to backwards compatibility for libraries. For a library to be backwards compatible, programs which were compiled against an old version of it should still work with a newer version, without crashing.

In practise, this means that a newer version of the library has to contain all of the same methods and fields that the old one contained. But not only that: the binary representation of every object must also be compatible, so for example the order of fields inside the binary object may not change. To illustrate the sorts of things you can and can’t do while maintaining binary compatibility, a list of rules for C++/Qt is here: KDE Binary Compatibility Dos and Don’ts

How does plinth handle binary compatibility?

Plinth tries to allow as much as possible. However, there will always be some constraints. For example, it will always be forbidden to remove a function which existed in a previous version. Here is a list of some of the things which can and can’t be done while maintaining binary compatibility:

You can:

• Add new static methods
• Add new static fields
• Add new non-static (virtual) methods, if you use a since(…) specifier
• Override a virtual method from a superclass, if you use a since(…) specifier
• Add new non-static fields to a sealed class (one which cannot be extended), if you use a since(…) specifier
• Add new type definitions (classes, compound types, interfaces, enums)
• Change an existing method into a native upcall or downcall
• Make a field or method more visible (e.g. change it from private to public)
• Reorder the fields or methods in a class
• Remove a private static field or method, if it is never used

You can not:

• Add a new non-static field to a class which is not sealed
• Change a since(…) specifier in a way that would change the order of fields or virtual methods
• Change a since(…) specifier on a constructor or a static method
• Change the type signature of a method
• Decrease the visibility of a field or method (e.g. change it from public to private)
• Remove a static field which could have been used externally
• Remove a static method which could have been used externally
• Remove a non-static field
• Remove a non-static method
• Remove a public class, or change it from public to package visibility
• Rename any types, fields, or methods

One important thing to point out in this list is that you can always add new virtual methods, and override existing ones. This is impossible in languages like C++, because they generally only have one virtual function table per object.

Virtual Function Tables (VFTs)

A virtual function table is just a list of pointers to functions. In languages like C++, every object which has virtual functions has a VFT to look them up in; inside the object is a pointer to that object’s VFT.

In order to call a virtual function, you must have a pointer to an object, and you must know the function’s index in the VFT. Given this, you can find the VFT by looking inside the object, and find the function by looking at this index into the VFT. Here is an example:

class Foo
{
  uint fooField;
  void print()
  {
    stdout::println("Hello, World!");
  }
}

class Bar extends Foo
{
  string barField;
  uint add(uint x, uint y)
  {
    return x + y;
  }
}

As you can see, Bar’s VFT has the same layout as Foo’s VFT, but with some extra methods added on the end. This makes it impossible to add virtual methods to Foo later on and maintain binary compatibility, because anything which extends Foo has to know exactly how many virtual methods Foo has in order to put its own methods after the end of that list.

So how does plinth manage to allow adding virtual methods? The key is to have several different VFTs, one for its actual class, and one for each class which it inherits from:

Now, since the VFTs are separate, Bar’s does not have to start with Foo’s. This means that, as long as you always add new virtual functions at the end of a VFT, then adding them is binary compatible. This is where since(…) specifiers come in.

Since Specifiers

Whenever you use a since(…) specifier, you provide a version number in brackets, such as since(8.4.2) or since(4.10). The compiler uses this to sort the methods before they are added to the VFT, so that methods with the largest since specifier will always go at the end of the VFT (methods without a since specifier always go at the start). So if you use a since specifier to add methods at the end of the VFT, adding methods is binary compatible.

Fields are more difficult, because each object must start with its superclass’s fields and VFT pointers. This makes it impossible to add new fields to a class which might be extended. However, if a class is sealed (i.e. it cannot be extended), then new fields can be added to the end of it without breaking binary compatibility. This is done in the same way as for virtual methods, by using a since(…) specifier.

Adjusting superclass VFTs

In the example above, there is a “Pointer to Foo’s VFT (adjusted for Bar)”. In that example the adjusting process had no effect, but when methods are overridden, this step is important as it ensures the correct method is always called. Consider this example:

class Foo
{
  uint fooField;
  void print()
  {
    stdout::println("Hello, World!");
  }
  since(1.8) uint add(uint x, uint y)
  {
    stderr::println("Foo.add()");
    return x + y;
  }
}

class Bar extends Foo
{
  string barField;
  uint add(uint x, uint y)
  {
    stderr::println("Bar.add()");
    return x + y;
  }
}

Without the adjusting process, if we had an object which we knew was a Foo but was actually a Bar as well, we would look in Foo’s VFT for the add method and call Foo.add(), even though the actual object was a Bar – defeating the purpose of virtual functions entirely. With the adjusting process, we would call Bar.add() through the adjusted VFT.

But what if we compiled Bar against version 1.7 of Foo, and then ran it with version 1.8? If the adjustment happened at compile time, the adjusted VFT would only have one element: “Foo.print” (since that’s all that existed in version 1.7), and so calling Foo.add() would make us crash.

Clearly, the adjustment needs to happen at run time, and this is exactly what plinth does. Before execution of any code, plinth will go through each superclass method and search for any overrides of it in subclasses. It does this using VFT descriptor tables which are stored along with the original VFT for that class.

Static members and constructors

During compilation, static members are translated into global variables and functions. This means that accessing them never depends on the layout of an object. Because of this, we can easily have more than one version of a static method in the same class.

Imagine you have a static method which performs something badly, which you want to deprecate and replace with a newer version which does things slightly differently. Usually, you would have to change the name or the type signature of the method so that code compiled against the old version of your library would still work, while forcing new code to use your new version. However, with plinth you can just create a new version of the method with a more recent since specifier. This allows both versions of the method to exist in the library, but things compiled against this version will always use the most recent one, and using older versions of the method is not possible when a new once is available.

The same thing is possible with constructors, since they are called in the same way as static methods, without using a VFT. Static fields are not affected at all by since specifiers (although they are allowed), since only one field can exist with any given name.

Other considerations

Plinth is built with binary compatibility in mind throughout, and features which could cause compatibility problems are avoided where possible.

For example, my last post was about nullability and initialisation. In the Freedom Before Commitment scheme that was outlined in the Summers & Müller paper, the check for whether an object had been initialised depended on which of the fields inside that object had been initialised. The problem with this is that when you call a constructor, you do not necessarily know about all of the fields of the object. In particular, new fields could have been added to the end of a sealed class, which could allow a new object to be used before all of the new fields were initialised. To combat this, the proposal outlined in last weeks post did not allow not-null fields to be first initialised outside a constructor – constructors must always give values to all not-null fields.

Next week’s post will be about interfaces and how they are implemented.