Dual Cache Key Modes

Note: This milestone needs to be thought out some more. The real goal here is to have a separate build planning mode which does not require that every reverse dependency get rebuilt when something changes at the cost of less trustworthy builds. However the point has been raised that we should at least be recording what our dependencies were built against and continue to produce a cryptographically strong cache key for these builds. I.e. we should not really have weak cache keys, or if we do have weak cache keys, we should still produce and retain the feature of having a strong cache key.

Cache keys are central to how BuildStream determines what needs to be built and serve to help ensure deterministic build output.

Currently however, we only have support for a deterministic build of a full project, which is to say that whenever any dependency of an element changes, everything depending on that element must be rebuilt in order to pedantically ensure that build results are deterministic.

This is not desirable however for tight developer loops, where a user desires to modify some lines of code down in glib and boot a fully assembled GNOME VM, there is no desire in that case to build 2 WebKitGtk modules and rebuild the whole world just to add a printf() statement to some code they are debugging.

Further, GNOME Continuous does not follow a deterministic stack build, instead the Continuous use case is to build only what has changed as fast as possible and then proceed to run installed tests. This responsiveness comes at some cost of failing builds and false test failures, but is well worth the cost for the responsiveness of immediate tests; usually in the case of false failures, Continuous will anyway fall back on its feet shortly thereafter.

We aim to solve this problem by offering an alternative, less deterministic (weak) cache key calculation algorithm, one which simply does not account for dependencies when calculating the key.

This should be a user configuration and not a project configuration (since any project can be built either deterministically or in weak mode).

Artifacts hereafter can still be shared, and this is important so that withing a community such as GNOME, one can simply say "I made a change in GTK+, just rebuild your VM with BuildStream to test it", and a user need not build everything just to try it.

However, the created artifacts should be encoded with whether they are weak or strong keys, for instance we now use paths such as {project-name}/{element-name}/{cache-key}, artifacts should be addressable with an additional path element to indicate whether they are a 'weak' or a 'strong' artifact.

This is Done, we have an implementation which:

• Calculates both weak and strong keys for each artifact
• Stores the keys in the artifact data, so they can be resolved from downloaded artifacts

• Builds in what is now called strict mode by default, with an option to disable that, allowing full builds which do not require rebuilds of reverse dependencies