Technical roadmap¶

Web UI modernisation¶

The current main web UI is based on the TurboGears 1 stack (although it uses SQLAlchemy rather than SQLObject for the database access layer). This makes some aspects of development more awkward than they might be with a more recent web framework.

Starting with the Beaker 28 release, the main web server is in the process of being migrated to Flask, by allowing endpoints to be implemented Flask handlers. We are also aiming to replace the front end components with cleaner alternatives based on React/Patternfly.

As part of those activities, we are planning to build web UI as separate service.

Reference harness modernisation¶

At present all Beaker recipes are run with the same harness, Beah. This harness includes a number of features solely for backwards compatibility with a legacy test execution system that is now long obsolete.

The Reference Harness design proposal covers the creation of a new reference harness that not only eliminates those no longer needed backwards compatibility features, but is also able to operate independently of Beaker and adds new capabilities (such as executing tests directly from source control) that would be difficult to implement as part of the current harness.

Issues with running the existing Beah harness inside a container has raised the importance of making the new reference harness more readily available to Beaker users.

Task oriented guides for users and administrators¶

Beaker offers a lot of power and flexibility, but that can often come at the price of making the right way to do certain things non-obvious. Beaker’s documentation is likely to benefit from additional sections that take a “task-based” approach to documentation and answer questions like “How do I limit my recipe to systems with a graphics adapter?” or “How do I require that my recipe run directly on bare metal and not in a VM?”.

This will include a general “troubleshooting guide” to help users and administrators collaborate effectively in tracking down the more obscure failures that can occur with the kind of integration testing Beaker supports.

Systematic self-tests for provisioning and beah¶

As a tool for better validating new Beaker releases, as well as making it easier to check for the correct operation of new Beaker installations, a set of self-test Beaker tasks will be made readily available. These tasks should come with helper scripts scripts for installing them into a Beaker installation and the appropriate job definitions to execute them across all configured architectures and distro trees.

Explicit system pool selection¶

Beaker currently schedules jobs on any system the user has access to, preferring the users own systems over group systems, over the generally accessible system pool.

This approach isn’t always desirable, since some systems have special features that should only be used when explicitly requested, or a user may wish to target a specific job at a particular set of machines.

Allowing systems to be grouped into pools (independent of the access policies used to grant or deny access to the systems) will allow users to express more abstract preferences about machines that aren’t directly related to the system itself.

This idea is covered by the Scheduler Integration for System Pools design proposal.

More flexible job prioritisation¶

Armed with the new user group and access policy models, and the new event driven approach to scheduling, it becomes possible to offer system owners much greater control over which recipes are selected to run on their systems.

This idea is covered by the Effective Job Priorities design proposal.

Job based recipe access limitations¶

Running recipes can currently inadvertently interfere with systems running recipes for unrelated jobs. While it is intentional that recipes can control systems other than the one they are running on, there should really be a mechanism that limits this access to only those systems running other recipes within the same recipe set.

Unifying hostRequires filtering and web UI search functionality¶

Beaker’s job processing and the web UI both allow a user to identify a subset of interest within the full set of available systems. The user interface for these is necessarily different, as once is based on the XML file defining a job, while the other is defined through an interactive web form.

However, rather than being thin wrappers around a shared internal filter creation API, the dynamic filter creation implementations in these components are almost completely separate. This means that capabilities are sometimes added to the hostRequires processing and not to the web UI, or vice-versa.

It seems like it should be possible to substantially reduce the level of duplication between these two components, and thus make it easier to add new filtering and sorting criteria in the future.

Automated classification of intermittent and spurious test failures¶

The OpenStack CI infrastructure includes a tool called “Elastic Recheck”. Essentially what they do is take the automated logs from particular OpenStack CI runs, feed them into an ElasticSearch instance, and then run various classifiers over those logs. Elastic Recheck then posts back to the failed change proposal, indicating the likely cause of the failure (and potentially triggering a second check attempt).

While such a tool wouldn’t need to be part of Beaker itself, it may still be a useful feature to explore, and there may be a place for publishing suitable classifiers in a related project.

A Beaker installation could potentially make use of such a tool in two ways. Firstly, Beaker includes the concept of “result acknowledgements”, where users can “NAK” a result to indicate that it wasn’t a valid test run (for example, there was an error in the test else, or something failed in the lab environment). An Bayesian classifier could be used to scan the logs of NAKed results, looking for patterns that are likely to indicate these kinds of “failures”, which don’t actually reflect a fault in the software being tested.

Secondly, for genuine test failures, a Bayesian classifier could be used to identify log data that is likely to correspond with a failed test, and suggest that as a probable cause when a test fails, rather than requiring users to trawl through the logs themselves. This is one of the key approaches the OpenStack CI team used to build their Elastic Recheck tool - many of the common failures were identified by automated scanning of previous failed test runs rather than by identifying the causes of the failure directly.

Provisioning other hypervisors¶

Beaker provides rich “guest recipe” functionality for testing installation and other operations within a KVM based virtual machine. Testing against non-KVM hypervisors is possible, but more awkward, as the guest VMs must be precreated and registered with Beaker as full systems with appropriate custom power scripts that handle the process of starting and stopping the underlying virtual machines. This is an unfortunate limitation.

Asynchronous message queues¶

The provisioning service on the lab controllers currently receives commands by polling a command queue stored on the main server. Similarly, the main task scheduler polls the database to determine when new and queued recipes can be assigned to systems.

It may be worth adopting fedmsg, or something similar, to help get rid of these polling calls.

Alternate database backend¶

The only currently supported database backend for the main server is MySQL (or an equivalent, like MariaDB). There are all sorts of reasons why this isn’t good, but migrating to PostgreSQL isn’t straightforward. The two main issues to be addressed are the handling of queries where MySQL and PostgreSQL have drastically difference performance characteristics (and there’s no solution that performs well in both), and the challenge of actually doing a data migration for any existing Beaker installations.