SCons: An Introduction - DRAFT COPY

Chosen for its cross-platform capabilities, SCons has been selected as Science Analysis Systems' next-generation software build tool, replacing CMT. It has been described as "an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache".

It is Open Source software using Python scripts to create configuration files. SConscript files are the equivalent of the requirements files currently used with CMT, and they define the targets that Scons will create during the build process. (For a more detailed introduction, go to http://scons.org/.)

Tagging Convention

With SCons, the vXrYpZ tagging convention will be replaced by packageName-XX-YY-ZZ standard. For example, the old tagging convention might apply v1r2p3 to the Likelihood package. Using the the new convention, this tag would become Likelihood-01-02-03, adding the package name to the tag and restricting use to 2 digit version numbering.

Note: Currently, this convention is automatically enforced by the old RM. Whenever a package is tagged with the vXrYpZ format, the old RM automatically tags the same package with packageName-XX-YY-ZZ. In the near future this convention will be reversed.

Minimal SConscript File

The following example shows a minimal SConscript file that will build nothing, but which will import some of the necessary tools for when build targets are added. This file should be stored in the top level directory of the package.

# $Id: xx_sconsOvrvw_working.htm,v 1.1 2009/10/13 23:00:10 chuckp Exp $
import platform

Import('baseEnv')
Import('listFiles')
progEnv = baseEnv.Clone()
libEnv = baseEnv.Clone()

The first line contains the file's CVS information. The next line will import the platform Python module. (This module is only necessary if you wish to add conditions that depend on the OS on which SCons is running.)

The two Import() calls import two SCons objects that have been defined at the top level; baseEnv is the basic compile environment created by the top level, and includes all necessary compile options such as debug, optimized, third party library locations, etc.

WARNING! The base environment must NEVER be modified. Any changes to the base environment are applied to all packages, independent of the order in which they were called.

The listFiles call imports a function defined at the top level. This function should be used any time you wish to pass a list of files to be included for creating a shared library, static library, program, etc.

WARNING! Due to technical limitations/bugs/features of SCons, it is imperative that two copies of the base environment be made if both libraries (of any kind) and applications are created.

Since this is almost always the case, it is strongly recommended that two copies of the base environment always be created.

Simple Static Library

After creating the SConscript file, targets to be compiled by SCons are created in a static library. The new code that should appear next is shown below:

myLib = libEnv.StaticLibrary('myLib', listFiles(['src/*.cxx']))
progEnv.Tool('registerObjects', package = 'myPackage', libraries = [myLib], includes = 
['myPackage/myLib.h'])

The first line creates a static library object containing all the information necessary to build the library at a later point. Notice the libEnv variable that is used, creating one of the copies made of the base environment. This copy should be used for any libraries to be compiled. It should NEVER be used for creating an application.

WARNING! If this rule is violated, it will cause errors in other packages that will be hard to track back to this source.

The StaticLibrary function call takes two arguments. The first specifies the name that should be given to the library. This name should not include any prefixes/suffixes that are platform specific. SCons will take care of adding those automatically. On Unix-based systems, this example would create a library named libmyLib.a.

The second argument, is a list of files to be compiled into the library. In this case, it is specified that the files to be included are in the src directory, relative to the top directory of the package, and are named *.cxx (i.e., anything ending with .cxx). Should only a single file be needed for the compilation of that library, you can either specify a single file to the listFiles function call [e.g., listFiles('src/myLib.cxx')], or you can simply skip the use of the listFiles() call and replace the entire listFiles function call with ['src/myLib.cxx'].

The second line registers objects at the top level to be compiled when appropriate. (This is not a standard SCons ability; rather, it is a custom extension to SCons.) Note that the progEnv copy of the environment is used to register objects, even if the objects are libraries, and it is strongly recommended that you abide by this convention.

Arguments used by the Tool() call are as follows:

• The first argument is the name of the tool to be called.

This argument needs to always be specified 'registerObjects'.

• The second argument is the name of the package.

• The third argument is a list of library objects to be registered for this package.

These can be shared or static. The name of the variable used is the same as the one used to store the library object returned by libEnv.StaticLibrary() call on the previous line.

• The next argument is a list of include files to be registered.

These are ONLY the public include files necessary to use the libraries that are going to be registered. If only a single header file needs to be registered, it is specified with ['myPackage/myLib.h']. If a list of header files, such as *.h, needs to be specified, the listFiles() call can be used.

Note: Other arguments can also be specified, and a complete list is provided in a later section.

Simple Shared Library

Similar to the code for a static library, the code for creating a shared library is:

mySharedLib = libEnv.SharedLibrary('mySharedLib', ['src/file1.cxx', 'src/file2.cxx'])

Note that the call libEnv.SharedLibrary() is used instead of the libEnv.StaticLibrary() call; however, the arguments to the SharedLibrary() call are identical.

Note also that this time we also to list the files to compile into the library individually, and the shared library will be compiled from the two source files src/flie1.cxx and src/file2.cxx.

As before, this object must be registered to be included in any SCons builds. Assuming the static library and the shared library have both been created, the registration function would be:

progEnv.Tool('registerObjects', package = 'myPackage', libraries = [myLib, mySharedLib], 
includes = ['myPackage/myLib.h', 'myPackage/file1.h'])  

Like the call from the previous section, this call is the same except that mySharedLib has been added to the list of libraries to be registered; an additional header file has also been added that needs to be registered in order to use the shared library.

Simple Application

Creating a simple application requires that we add a section similar to:

myApp = progEnv.Program('myProgram', ['src/myProgram.cxx'])

The progEnv copy of the base environment is used to create a program objection.

WARNING! For this task, never use the copy of the environment for creating libraries. If this rule is violated, it will cause errors in other packages that will be hard to track back to this source.

The arguments provided to the progEnv.Program() call are similar to those for libraries:

• The first argument is the name of the executable, making sure to exclude any platform specific prefixes or suffixes.

Note: On Windows SCons will create a program called myProgram.exe.

• The second argument is a list of source code files to compile the program. All three methods described in the previous two sections are valid here as well:
  • (listFiles()
  • ['single file']
  • ['list', 'of', 'files'])

And, just as in the previous sections, the program object must be registered with the top level before it can be used. Assuming the static and shared libraries from before still exist and we want to add this program to the registration call, we would modify the registration line as follows:

progEnv.Tool('registerObjects', package = 'myPackage', libraries = [myLib, mySharedLib], 
includes = ['myPackage/myLib.h', 'myPackage/file1.h']
             binaries = [myApp])

Observe that a new argument was added that will list all of the binaries to be used.

Note: When creating test applications, register them with the
testApps = [myApp] argument instead of the binaries = [myApp]
argument.

OS Specific Conditions

To perform functions on certain platforms only, use regular python conditionals around the functions. For example, to define the TRAP_FPE macro only on Linux platforms, append:

if platform.system() == 'Linux':
        progEnv.Append(CPPDEFINES = 'TRAP_FPE')

      

The platform.system() call returns the name of the OS. In this case, we wish to know if we are running on a Linux platform and, if that is the case, we wish to add a -DTRAP_FPE to the gcc command line.

Note: The progEnv.Append() call is explained later. (See Compiler Options.)

Libraries that Depend on Other Libraries

SCons performs dependency computations at the source code level. It does not compute dependencies of various binary packages such as the dependency of library A on library B when compiling library A into application A. A package maintainer writing application A does not wish to know all dependencies of all libraries; he only needs to know the DIRECT dependencies of the application.

Similarly, the package maintainer of library A should not have to know all the dependencies of library B when creating library A; he only needs to know that library A depends on library B. SCons, by default, does not have this ability.

??? The package maintainer for A only has to know that application A ??? depends on library A, but the owner also has to know that library A depends on library B, ... and so on, until all dependencies have been met.

Fortunately, SCons provides a "tool" that simplifies this problem so that only direct dependencies need be specified. When the package owner creates library A in the SConscript files, the package owner also creates an additional file to record the DIRECT dependencies of library A. This file must have a specific name called <package>Lib.py, and it has to be located in the top level of the package, together with the SConscript file.

In our example, the owner of 'myPackage' has two libraries: myLib and mySharedLib. Assume that myLib DIRECTLY depends on some other library of some other package (called someOtherPackage) and myLib also depends on some external library xerces. The contents of myPackageLib.py would be as follows:

def generate(env, **kw):
    if not kw.get('depsOnly', 0):
        env.Tool('addLibrary', library = ['libA'])
    env.Tool('someOtherPackageLib')
    env.Tool('addLibrary', library = env['xercesLibs'])

def exists(env):
    return 1

Both of these python functions need to exist at all times. The second of these functions is for features currently not used by use, so it should always be specified as shown. The first function is what creates the recursive computation of library dependencies. The first line in the if statement adds libA to the dependencies. This line is put inside an if statement that determines if it should be added or not, because the dependencies of libA need to be specified when libA is created.

However, when libA is being created, we can't specify that it should include libA. This would create a recursive dependency. As a result, when we want to build libA and we want to catch all the dependencies, we call this function; but we also pass an additional argument setting depsOnly = 1, so that the recursive dependency isn't created.

The second argument (and successive arguments if more are needed) adds the dependencies of libA to other libraries created by other packages. These must be DIRECT dependencies to keep computation fast.

Note: Unnecessary listings will slow SCons down considerably.

The last line of the function lists one (or more) external libraries that libA DIRECTLY depends on.

When we create libA, we want to link all of libA's dependencies into libA without creating a recursive dependency. In order to achieve this, we added the if statement around the addition of libA to the link line. In our SConscript file – prior to creating libA – we add the dependencies with the line:

libEnv.Tool('myPackageLib', depsOnly = 1)

Note: This line is identical to that in the example below, i.e., when we want to link libraries into the application. The only difference is that – when we create libA, we don't want to link in libA; so we add the extra argument of depsOnly = 1.

TODO: Currently there's no way to specify which library created by a single package we wish to use. This feature will be added at a later stage since the problem has not arisen yet in ScienceTools.

Application Dependence on Libraries

With the dependency tree generated by the previous section for libraries, package owners wishing to create dependencies on libraries for their applications need only list the DIRECT dependencies of their applications.

In our example, the owner of myPackage currently has one application myApp. Assuming this application depends on some library from myPackage as well as the external library ROOT DIRECTLY, he would add this line to his SConscript file:

progEnv.Tool('myPackageLib')
progEnv.Tool('addLibrary', library = env['ROOTLibs'])

Technically, the ordering of these two calls does not matter. However, it is strongly recommended that these calls be made prior to the call for generating the application. This will make it easier for a human to understand the code at a later point.

The first line will call the generate function from myPackageLib created as shown in the previous section. That function will add the library from that package to the dependencies of myApp, along with any other libraries that myPackage's library depends on.

The second call adds ROOT to the libraries that myApp depends on DIRECTLY. Should myApp not need ROOT directly but through some other package's library, it should be left out here. It is the responsibility of that package to add the ROOT dependency.

Arguments to Register Objects

Registration functionality is an extension of SCons created by us. The registration is done by a call to progeEnv.Tool('registerObjects, 'mypackage', ...). There are a minimum of two arguments to that function.

The first argument has to always be 'registerObjects', and the second argument always has to be the name of the package performing the function call. Additional arguments can be from the following:

Other arguments will be added as the need for them arises. The ordering of these arguments are not important.

Compiler Options

SCons provides some compiler independent options, which should be used as much as possible. Less common options have to be specified in a compiler dependent way; therefore, prior to setting a compiler option for a less common option, one should perform a check for which OS is being executed. (See OS Specific Conditions.)

Caution! In almost all cases these compiler options are lists that need to be appended to. If they are simply assigned new values, they will overwrite older options already defined. And..., in order to prevent repeating the same compile options several times, one should only append if the option does not exist. These options are available through a single function call: AppendUnique().

For example, to add a unique preprocessor definition to the compiler when compiling an application you would do:

progeEnv.AppendUnique(CPPFLAGS = ['TRAP_FPE')

This would add a -DTRAPF_FPE to the compiler options if one didn't already exist.

For a complete version of these compiler options, see http://scons.org.

Common Platform-independent Compiler Options

The following compiler options appear frequently enough that SCons takes care of converting them to the format required by the target compiler, and it is recommended that you use them when possible:

Less Common Platform-dependent Compiler Options

The following compiler options need to be specified differently, depending on which compiler is being used, and should therefore be wrapped around if statements:

External Libraries

Note: SCons uses a different directory structure than that used by CMT. The SCons structure was modified so that Unix- and Windows-based structures are more compatible than before. Only one version of external libraries is used per OS (i.e., both optimized and debug builds use the same libraries). As a result, the CMT external library structure cannot be used with SCons.

All external libraries used are automatically added to the library path of the compiler. Each package only needs to add the libraries it will need to the library path.

For example, to add CLHEP to list of libraries linked to, you would use:

progEnv.Tool('addLibrary', library = progEnv['clhepLibs']) 

Note: An example of external libraries similar to those listed below is used in Libraries that Depend on Other Libraries. The same names for accessing the external libraries should be used in that section as the ones defined further down in this section. ??????????????

Each external library the libraries required for linking against it stored in a variable like the above for clhepLibs. ???????????? Currently these are the external libraries available to be linked against. The progEnv['clhepLibs'] should be replaced with the name of the libraries you wish to add.

Running SCons

SCons is installed at slac in /afs/slac/g/glast/applications/SCons/0.97.0d20070809/bin/scons. To run SCons, you must be in the directory that contains the SConstruct file.

For example, to check out ScienceTools version LATEST1.2220 you'd issue the cvs command:

cvs co -r ScienceTools-LATEST-1-2220 ScienceTools-scons

You would then enter the ScienceTools-scon directory that was created. The SConstruct file that is read by SCons is in this directory.

To get a list of options added to SCons specifically for ScienceTools, issue the command:

scons --help

Note: These options have been programmed into the SConstruct and support files for ScienceTools, and were not written – and are not supported – by the SCons developers. To see the options that are written and supported by the SCons developers, issue the command:

scons -H

Building a Single, Specific Package (and its dependencies)

To build a single package, simply specify the package name as the target.

For example, to build facilities (and its dependencies), issue the following command along with any other options necessary:

scons facilities

Ignoring Errors

By default, SCons will stop compilation after the first error it encounters. Since the first error is probably not in a package of interest, you can tell SCons to continue building after it encounters an error by appending the -i option to the SCons command.

Parallel Builds

Since SCons reads every file it will build, it has very detailed knowledge of the build structure. As a result, SCons can safely perform parallel builds and not violate any dependency issues.

To tell SCons to do parallel builds, specify the -j option. The -j option is followed by a number to specify how many concurrent builds should be done.

For example, -j 2 would tell SCons to perform execute two g++ commands at the same time.

SCons Build Output

When SCons is performing the build process, it will put files in the following subdirectories that are located in the same directory as the SConstruct file:

Note: Additional output directories will be created in the future, and they will follow the same convention:

• If the contents of the directory are dependent on the OS, or on the compile options, SCons will include the variant subdirectory.

• If the contents are independent of such changes, SCons will not include the variant subdirectory.