Difference between revisions of "builder"

Revision as of 07:11, 25 August 2017

Unless you are creating a Linux distro that is a fork of another one with a well stocked software repository, the thought of having to build one from scratch is breath taking. To tackle the thousands of source code downloads, applying necessary patches, figuring out the proper compile flags and various make systems (autogen, autoconf, bootstrap, cmake, etc), and applying file and directory ownership and permissions is quite a job! Fortunately builder will handle all of these tasks for you and already comes with a large pool of software profiles that can be tailored to your individual needs, or left to assume defaults.

License

This projects' codebase is licensed under the 2-clause BSD which can be found at "Wikipedia".

Features

While there may be more features than are currently listed below, this should give potential users at least a portion of the currently implemented features.

* Simple syntax for easy compiling no matter build system

* Works with autogen, autoconf, bootstrap, cmake, and traditional

* Extreme flexibility using various pre and post step scripts

* Can pull source code from private or public locations

* Small footprint - contains only two files!

* Portable using POSIX shell

Addresses

As stated in the preamble, managing a software repository is a tremendous task that can get out of control if great effort and detail are not constantly applied. Several of the issues related to maintaining an online repo are handled automatically by builder and include:

Names not being uniform

This occurs when the 'foo' package in the i32 repo might be 'foo-ng' in i64. Not only can this cause confusion for the end user, but also makes it difficult for package maintainers to manage the proper dependency lists per software title.

Packages are not split the same way

Just like when the names don't match, this can equally cause problems managing dependencies lists because 'libfoo' in the r32 repo is actually the combined contents from 'libfoo' and 'libfoo-dev' from the i32 repo. When there are inconsistent packages, end users can again become confused and frustrated by not understanding appropriate steps to take in order to get a working system.

Inconsistent existence of a package between repos

What happens when 'foo' exists in i64, but not anywhere else? Not only do you end up with complaints from the users, but you risk the growth of your project because requested or required functionality is not completely instituted.

Source code used to build the packages may be different

A common problem where the 'foo' package in the r32 repo might have been compiled using the source code version 1.2.3, but i64 may have 1.4.6 and i32 might have 1.2.6. This can lead to various problems being present under certain circumstances, but not others due to the vast different in used code bases.

Compile flags are not the same

Similar to the above point, 'foo' in i64 might have been compiled using "--option1=xyz --option2 --option3", whereas "--option2 --option5" may be used to compile for i32 causing a mismatch of functionality between the different repositories available to users. This will simply lead to more questions from your community as to why this works in this situation, but not this one.

File and directory permissions and ownership are incorrect

There is a reason that these attributes were developed and implemented. Documents and graphics should not be executable and require proper ownership so users can read and/or modify them. Contrarily, binaries do need to be executable along with and SUID abilities. When these settings are incorrect, it not only causes a problem with programs that check for these rules, but also causes major security concerns!

Benefits

Not only do the above concerns get addressed by using builder, but several benefits become present as well. Some of them will be outlined below:

Uniform updates

Since builder is so easy to use, once a profile is developed for a software title, simply calling builder with '-a all' (or for each individual CPU architecture on the proper hardware or VM) will make all the packages available to simply upload to your repo!

Access to various CPU architectures

By using builder with available profiles, your Linux distro can immediately be used on all the devices with the supported silicon! This can make your project have a larger pool of users instead of only producing for a single architecture.

Access to a large repository of software

Instead of having to figure out how to build all the desired software you would like to include with your distro, you can save yourself a ton of effort and resources by using any or all of the software available to be built with builder!

Parameters

Among the standard 'help' and 'version' actions, this project also contains several others that will be covered below in greater detail.

-a allows the user to specify which CPU architecture to use during the compile process. Currently builder supports three including ARM v6/7 (r32) and AMD/Intel 32-bit (i32) and 64-bit (i64). The former is typically used for Raspberry Pi boards.

-d is used to build packages based on directory names instead of the supplied package name via the '-n' parameter. So, for example, if you need to create several packages from a single execution of builder such as alsa and alsa-config, you can pass this parameter to accomplish this task. It is important to note, however, that the directory names will become the package names and will need to be setup in the various pre/post scripts.

-D in the instance that you have an online repo setup with the source code files already located there, passing this switch can automatically download those files instead of the package maintainer having to manually find and download it beforehand. It uses the URL_CODE value from the builder.conf file in order to locate the proper source tarball from you repo.

-L similar to the '-D' option, this one will use a local (networked|attached) resource to fetch the source code files instead of having to go over the Internet. This one works with the DIR_REPO variable value in the builder.conf file.

-n indicates the name of the package to create and requires a builder profile under /etc/builder that has the same directory name.

-p provides the number of passes to take in order to fully compile, sort, and package a particular software. For example, php can have multiple passes to create various related packages without having to entirely recompile the source code. There are also some packages that have to start a compilation, then move files around, then re-compile in order for the end result to be correct - typically involving database interaction.

-P states that on a follow-up attempt at building a software due to a failure, that the patches should not be re-applied since that will cause another failure in the build process. Please note, that this is not the appropriate parameter to use if you don't want patches applied for a particular pass. See the information on the various builder files below.

-r defines phrases that can be removed from the package name and can contain regular expressions. So, for example, lets say you want to have 'lib' removed from 'libassuan', you can simply pass "-r '^lib'" and it will remove 'lib' from the beginning of the name!

-r informs builder that no sorting or packaging attempts should be made until the final pass in a multi-pass build. Otherwise, these steps will occur every pass and will require any pre/post scripts to manually move and rename a small set of files and directories (instead of making mass adjustments).

-s like the '-r' parameter, using this one will allow the sorting to occur, but will withhold packaging the contents (and applying ownership and permissions to the package contents).

-S another option that can be used to automatically obtain the source code files for package maintainers, this one will actually fetch the one from the official website. HOWEVER, this will require a properly configured associated information file containing a line beginning with 'download:' and 'filename:' representing the URL and filename respectively of the source code file to obtain. This switch uses the URL_INFO variable in the builder.conf to find the information file.

-t defines the type of compile framework that is used for the particular package and includes: autoconf, autogen, bootstrap, cmake, custom, and default.

-U lets builder know that it should not unload any dependencies required the build the package upon a successful run. If the SPM_UNL variable is blank, this step is ignored no matter if this is passed or not.

Installation

The installation of the software could not be easier. Simply download the package, uncompress it, and run the 'install.sh' script with an elevated account (e.g. root) or prefixing with 'sudo' such as:

$ cd /tmp
$ tar zxf 2017.08.22-builder.tgz
$ cd builder
$ sudo ./install.sh

Checking the effective UID: [done]

Installing the software: [done]

Congrats! Your software is installed and ready to use!

Example

'builder' only requires a profile and the source code to successfully build software. Below the steps will be outlined to show you how easy it is to actually use it! In this example, we will be using libbsd with the assumption that it is downloaded in /tmp.

First lets make sure we have a profile for the software:

/etc/builder $ ls
builder.conf   libbsd/

Now lets make sure we are in the proper directory to compile (one level up from the source code directory if CODE has a proper value, or within the source code directory without it)

/etc/builder $ cat libbsd/config | grep CODE
CODE='lib|libbsd-'
/etc/builder $ cd /tmp
/tmp $ ls
libbsd-0.8.6/   libbsd-0.8.6.tar.gz

Now lets call 'builder' to make the packages!

/tmp $ builder -n libbsd -a i64

Performing filesystem checks...

Installing compile-time dependencies...

Entering source code directory...

Applying software patches...

Generating the compile script...

Compiling the software...

Copying in the extra files...

Separating into various packages...

Applying ownership and permissions...

Creating the various packages...
   bsd [dev]
   bsd [doc]
   bsd [lib]

Congrats, the software has been packaged successfully!

Developers

Dave Henderson [dhenderson (at) cliquesoft (dot) org] - main developer polikuo [michael5329208 (at) gmail (dot) com] - testing and contributions