Genie Home

Developing Genie

Introduction

Genie is an Open Source and Free Software project. The source code is openly available from the Gnome software repository and you are free to copy and use the software under the terms of the GNU Lesser General Public License (LGPLv2.1+).

Genie acts as a parser for the Vala compiler and makes up only three files within the Vala repository. The relation between Genie, the Vala compiler and the C compiler is illustrated and explained below.

There are several ways to contribute to and enhance Genie. These include submitting bug reports, submitting patches and reviewing patches. This page gives more detail on these processes to make it quicker and easier to contribute to the project.

What to Expect

Gnome Bugzilla and Mailing List

Bugs and feature requests are recorded using the Gnome bugzilla:

• Open Genie Bugs

• All Genie Bugs

Before filing a new bug use the links above to check if a bug or feature request is not already recorded. When filing a bug include a simple test case that illustrates the problem.

The links also show changes to bugs in date order, newest first. This is a useful way to check for new bugs and updates if you are intending to be a reviewer.

A patch to the Genie parser can be submitted by attaching it to the bug report.

There is a mailing list for Vala and Genie where questions can be asked.

Genie's Relation to valac

When starting to analyze a problem with compiling a program it is useful to break the compilation process down into three stages. The first stage is carried out by the Genie parser and translates the program source file into a Vala abstract syntax tree (AST). The second stage is carried out by the valac compiler and translates the AST into C code. The third stage is carried out by a C compiler, gcc by default, and compiles the C source file into a binary.

These three stages are illustrated below. Further details are then given on how to examine the output of each stage. Note that valac can process other input files at the same time as Genie source files.

Stage 1 - Genie Parser

The Genie parser is made up of three files in the valac source code. The files can be browsed from the main Vala git repository with the following links:

valagenietokentype.vala

This file contains an enumerated list of tokens. This file is unlikely to need modifying unless adding new functionality

valageniescanner.vala

The scanner reads .gs files and converts the text into tokens

valagenieparser.vala

This file parses the tokens and creates a Vala AST

See Hackers's Guide to Vala - Parser for a list of classes used in the Vala AST

A project can include other files that are then included in the AST. These can be Vala source files and Vala API bindings to C libraries. So it is advisable to develop a simple Genie test program that only includes the feature you are testing. This avoids introducing any unnecessary uncertainties into the testing. The only two Vala API files that are included automatically are for GLib and GLib's GObject system.

Use the --dump-tree=filename.vala switch with valac to see the AST converted to Vala code. This is useful to check if the parser is working as expected.

Stage 2 - C Code Generation

The Vala compiler will use the AST to generate C code. At this stage the Genie parser has finished its work, so any bugs seen here are likely to be within the Vala compiler itself or an error in the C bindings. The Genie parser, however, may not have created the AST correctly and this can be ruled out by coding the test in Vala to compare with the Genie results. Bug hunting is sometimes a long and slow art! Hopefully there is satisfaction when the cause of the bug is tracked down.

To only see the C code and not produce a binary use the --ccode switch with valac.

Stage 3 - C Compiler

At this stage you are probably a long way from a bug in the Genie parser itself, but it is useful to understand there are powerful debugging tools available for C compiled programs.

gdb, the GNU Project debugger, or a graphical front end to gdb, such as Nemiver, can be used to step through binaries produced from Genie code. Use the --debug switch with valac. This inserts line numbers from the Genie code into the C code, which are then included in the binary and are displayed with gdb's output. To both see the C code and produce the binary use the --save-temps switch with valac.

valgrind can also be used for dynamic analysis of the program. This is useful for tracking down memory leaks. A blog post Testing for memory leakage with Vala and GLib by Richard Schwarting gives an idea of what can be achieved.

Unit Tests

When submitting a bug report it is helpful to attach a test case to show the problem. This can be used to test any patch submitted.

Vala has a test suite that can be browsed at https://git.gnome.org/browse/vala/tree/tests

There is currently no test suite for Genie in the Vala source code.

Building valac from Source

To start tracking down the cause of a bug and developing a fix for it you need to build the Vala compiler from its source code. The instructions that follow are for a GNU/Linux operating system, such as Fedora or Ubuntu, but Vala can also be built on Windows or Mac OS X with the right tools.

These instructions place the Vala compiler in your local user directory. This makes your development valac independent of the valac you would normally use to compile Genie programs. This allows normal Genie coding to continue, while keeping development of the fixed compiler separate.

Setting up a Build Environment

The Vala compiler is self compiling so a copy of valac will be needed. To manage the large number of files in the compile process Vala uses the GNU build system, also known as Autotools. The commands that follow have been tested on a virtual machine running a clean install of Fedora 20. The Ubuntu commands were found from a search, but are untested.

Anyone already compiling Genie programs will have the following installed, but they are listed here for completeness:

You will also need the GNU build system, the Git version control system and some additional packages for GObject introspection:

Compiling valac to a Local Directory

The following instructions will download Vala into your Unix home directory ($HOME) and create the directory structure:

• $HOME/vala/source/ - contains the Vala source code

• $HOME/vala/installed/ - contains the local build of Vala

The first stage is to download a copy of the Vala source code repository and configure the build system. This stage is done once:

The next stage is to build and install valac from source. This will be repeated each time you want to test your modifications to the source code:

Creating a Patch with git

Check Your Identity

Patches should be submitted with your full name and a valid email address. Check these have been set up in git with:

git config user.name
git config user.email

To set them use:

git config --global user.name "your name"
git config --global user.email your@email

To set these the just for the current project replace --global with --local

Refresh Your valac and Create a Branch

It is best to work with an up to date version of the Vala compiler. This is so when it comes to submitting your patch it is less likely to need revision because of changes upstream.

A great advantage of git is the ease of creating branches. It is easy to create a branch, try out some ideas, commit the changes in the branch and if you decide you don't like them just delete the branch.

A typical way of starting hacking would be:

git checkout master
git pull origin
git checkout -b name_of_working_branch

Tips on Working

While developing the idea using just make and then make install to build valac is quicker than running the full test suite each time.

The aim is to create a clear and simple patch or set of patches. git rebase --interactive master provides a powerful way of re-organising commits in a local branch so what was originally developed over four or five commits, for example, can be squashed into one or two simpler commits. A reviewer of the patch doesn't need to know the development stages, but the patch needs to be clear in what it does and how it does it.

Final Checks

Before generating the patch it is best to make sure valac is built from the most up to date version and that the changes you have made work against a clean source. Something like this:

git checkout master
git pull origin
git checkout -b name_of_working_branch
git rebase master
make clean
make
make test
rm $HOME/vala/installed/* -rf
make install

Then run your test cases against the version of valac just generated. So long as everything works it is time to generate your patch.

Generate and Submit the Patch

To list, for example, the last ten commits use:

git log --oneline -10

This will show the abbreviated commit hashes. Choose the commit just before your work and note its abbreviated hash. This hash is then used to identify the commits after that to generate the patch or patches:

git format-patch abbreviated_hash

You should now have one or more files starting with a number and ending in .patch. These are the files that will be uploaded to the Vala bugzilla.

Reviewing a Patch

A submitted patch needs to be reviewed by someone else to check that it:

• resolves the bug / feature request
• maintains compatibility with previous versions of Genie
• meets the security standards of the project
• provides a simple and elegant solution

• meets the coding standards of the project (see Vala Hacking - Coding Style)

Refresh the Build Environment

Before applying the patch remove the old build of valac and make sure the vala source is clean and up to date:

rm $HOME/vala/installed/* -rf
cd $HOME/vala/source
make clean
git checkout master
git pull

Download and Apply the Patch

Usually a patch is added as an attachment to Bugzilla. This can be downloaded with a web browser. In Firefox, for example, click on the name of the attached file to view the content in raw format. Then choose File -> Save Page As from the Firefox menu and save the patch to the local Vala source directory.

Create a new branch to test the patch in, apply the patch and build a new local copy of valac:

cd $HOME/vala/source
git checkout -b testing_branch
git apply name_of_patch_downloaded.patch
make
make install