I work on the EFL, a cross-platform graphical toolkit written in C. I recently decided to improve one aspect of our users' (developers using our API) experience by making the EFL provide them with more information and stricter sanity checks when developing and debugging their applications. A key requirement was ease of use. With these requirements, the solution was obvious, unfortunately obvious doesn't always work as well as you expect.

The obvious solution: an environment variable

This sounds like a good idea, but it comes with one major, unacceptable, flaw: a significant performance impact. Unfortunately, one of the places where we wanted to collect debug information in was Eo, a hot-path in the EFL. Believe it or not, adding a simple check if debug-mode is enabled was enough to degrade performance. To illustrate, consider the following code:

Note: thanks to Krister Walfridsson for pointing out a mistake in the following example.

#include <stdio.h>
#include <stdlib.h>

#define N 100000000 // Adjust for your machine

int main(int argc, char *argv[])
{
   int debug_on = !!getenv("DEBUG");
   unsigned long sum = 0;
   unsigned int i;

   for (i = 0 ; i < N ; i++)
     {
        unsigned int j;
        for (j = 0 ; j < N ; j++)
          {
#ifdef WITH_DEBUG
             if (debug_on)
                printf("Debug print.\n");
#endif

             sum += i;
          }
     }
   printf("Sum: %lu\n", sum);

   return 0;
}

Which when executed would result in:

$ # Debug disabled
$ gcc -O3 sum.c -o sum
$ time ./sum
Sum: 996882102603448320
./sum  0.18s user 0.00s system 95% cpu 0.186 total

$ # Debug enabled
$ gcc -O3 -DWITH_DEBUG=1 sum.c -o sum
$ time ./sum
Sum: 996882102603448320
./sum 0.26s user 0.00s system 98% cpu 0.267 total

This is almost a 50% performance penalty to just have it compiled in, not even enabled! While this is an extreme example, code-paths with a similar impact do exist in our codebase.

This performance penalty was unacceptable, so I went back to the drawing board.

An alternative solution: a compilation option when compiling the library

This idea is to add an option to compile the library in debug mode, so whenever it is used, it will always collect the data. In the EFL source code we would have something like:

EAPI Eo *efl_ref(Eo *obj)
{
   // snip ...
#ifdef EFL_DEBUG
   collect_debug_info();
#endif
   // snip ...
}

Users of the API will have to recompile the library with debugging on and use it instead of the distro provided version. This introduces a few issues, some of which are solvable, some are not. The main remaining issue is that the user would have to download and compile the library. A big enough hurdle to render this solution almost useless.

We had this implemented for quite some time because it was good enough for the most of us, but recently I decided to finally implement the better user-friendly solution I had in mind.

A better solution: automatically providing both versions

Eo is small, so compiling it twice wouldn't have a significant impact on neither the build time nor the deliverable size. I therefore modified our build-system to build Eo twice, once as libeo.so and once as libeo_dbg.so: the first version compiled as normal, and the second with EFL_DEBUG set.

This was already a better solution. It meant users of our API could just link against the new library when compiling their applications to use the debug version, and when they are ready to release, link against the non-debug one. This, while much better, still requires some advanced know-how from the users that I wanted to eliminate.

An even better solution: using DLL injection to ease usage

I wanted to create an easy way for users to toggle between the two on runtime, without having to relink their applications. I then realised I could use LD_PRELOAD (more info) in order to force the debug version of the library to be loaded first. Problem solved.

I therefore wrote a small script to ease usage:

#!/bin/sh
# The variables surrounded by @ are replaced by autotools
prefix="@prefix@"
exec_prefix="@exec_prefix@"
if [ $# -lt 1 ]
then
   echo "Usage: $0 <executable> [executable parameters]"
else
   LD_PRELOAD="@libdir@/libeo_dbg.so" "$@"
fi

Usage is now as simple as wrapping execution with the script above:

$ # Debug off
$ ./myapp param1 param2

$ # Debug on
$ eo_debug ./myapp param1 param2

Mission accomplished!

Note: The script and libeo_dbg.so are automatically built and installed with the rest of the EFL. This means that they will be available for everyone installing the efl package, or in some distributions the efl-dev package.

Conclusion

We now have an easy-to-use tool for users of our API to debug and check correctness of their applications. As a side effect, we also earned a useful tool for end users (users using applications written with the EFL) for providing better information when reporting bugs. All of this without any impact on normal execution.

Please let me know if you spotted any mistakes or have any suggestions, and follow me on Twitter or RSS for updates.