Difference between revisions of "builder"

From Cliquesoft
Jump to: navigation, search
Line 21: Line 21:
 
an immediate access to a large repository of software
 
an immediate access to a large repository of software
  
A simple utility script used for de-duplication of data pools such as your pictures or documents.  There's no deletion of the original files, but instead all unique data is copied to a separate directory so the originals can be deleted, backed up, or some other desired task.  Not unlike our other bash-script-based software, this project relies on our clAPI framework for various functionality, so be sure this dependency is satisfied before using.  It's also worth mentioning that the placement of the OPTION's <b>must</b> follow their respective ACTION (or parent script) which can be determined via the <i>--help</i> output.  It might also help to read over the basics of [http://wiki.cliquesoft.org/index.php?title=clAPI clAPI] to get a better understanding when running software from the command line.
 
  
  
==Terms==
+
==License==
 
This projects' codebase is licensed under the 2-clause BSD which can be found at "[https://en.wikipedia.org/wiki/BSD_licenses#2-clause_license_.28.22Simplified_BSD_License.22_or_.22FreeBSD_License.22.29 Wikipedia]".
 
This projects' codebase is licensed under the 2-clause BSD which can be found at "[https://en.wikipedia.org/wiki/BSD_licenses#2-clause_license_.28.22Simplified_BSD_License.22_or_.22FreeBSD_License.22.29 Wikipedia]".
  
Line 30: Line 29:
  
  
==ACTION's==
+
==Features==
Among our standard 'help', 'version', and 'update' ACTIONs, this project also contains two others - 'install' and 'sort'.  The 'install' ACTION simply installs the script in the "~/.bin" directory for XiniX and "/usr/bin" for typical GNU/Linux distros.  To see how easy it is to install, see the examples section.
+
While there may be more features than are currently listed below, this should give potential users at least a portion of the currently implemented features.
 +
* Simple syntax for easy compiling no matter build system
 +
* Works with autogen, autoconf, bootstrap, cmake, and traditional
 +
* Extreme flexibility using various pre and post step scripts
 +
* Can pull source code from private or public locations
 +
* Small footprint - contains only two files!
 +
* Portable using POSIX shell
  
The other ACTION, 'sort', will perform most of the desired work.  It will basically run through the entire directory heirarchy of the <i>--source</i> OPTION and create a database of MD5 hashes per compared file.  During this comparison, if the DB does <b>not</b> contain the generated MD5 hash, then the file is unique and will be copied to the <i>--target</i> directory - retaining it's location.  On the other hand, if the MD5 hash is present in the database, then the file is a duplicate and no further action is taken against it.  After 'dedup' has completed its run, all unique data is now located under the <i>--target</i> directory.
 
  
  
===OPTION's===
 
There aren't very many options currently available for the 'sort' ACTION, but more are planned for future versions.  Below we'll cover each to provide you with a detailed explaination of their function.
 
  
* <b>--source</b> allows the user to specify the "parent" directory containing the various data pools.  For example, lets say that you've ended up with your collection of pictures spread throughout different directories (aka folders) in the file system.  Since this project doesn't allow for multiple <i>--source</i> OPTIONs, all the directories containing your pictures will need to be moved or copied into a single "parent" directory and passed as this OPTIONs value.
+
==PARAMETERS==
 +
Among the standard 'help' and 'version' actions, this project also contains several others that will be covered below in greater detail.
  
* <b>--target</b> provides the location where all unique files will be copied from the <i>--source</i> directoryIt's important to note that each unique file will retain it's location after its copy.  For example, if we've executed 'dedup --source=/tmp/source --target=/tmp/target ...' and the source directory contains the unique file /tmp/<b>source</b>/garden/2011/sunflowers.jpg, when it's copied into the target, the directory would be /tmp/<b>target</b>/garden/2011/sunflowers.jpg.
+
: * <b>-a</b> allows the user to specify which CPU architecture to use during the compile processCurrently builder supports three including ARM v6/7 (r32) and AMD/Intel 32-bit (i32) and 64-bit (i64).  The former is typically used for [https://en.wikipedia.org/wiki/Raspberry_Pi Raspberry Pi] boards.
  
* <b>--temp</b> is used to indicate an alternative temporary directory for processingThis parameter is optional, but should be a full path of the desired location.
+
: * <b>-d</b> is used to build packages based on directory names instead of the supplied package name via the '-n' parameterSo, for example, if you need to create several packages from a single execution of builder such as alsa and alsa-config, you can pass this parameter to accomplish this task.  It is important to note, however, that the directory names will become the package names and will need to be setup in the various pre/post scripts.
  
* <b>--uniquedb</b> defines the unique database of files from a prior run.  Like the <i>--temp</i> directory, this is optional, but allows a user to compare additional (possibly) duplicate files against the database from a prior run.  So, given the pictures example above, this would allow a person to compare any newly found pictures against a prior database containing the unque file hashes from a previous run.
+
: * <b>-D</b> in the instance that you have an online repo setup with the source code files already located there, passing this switch can automatically download those files instead of the package maintainer having to manually find and download it beforehand.  It uses the URL_CODE value from the builder.conf file in order to locate the proper source tarball from you repo.
 +
 
 +
: * <b>-L</b> similar to the '-D' option, this one will use a local (networked|attached) resource to fetch the source code files instead of having to go over the Internet.  This one works with the DIR_REPO variable value in the builder.conf file.
 +
 
 +
is used to indicate an alternative temporary directory for processing.  This parameter is optional, but should be a full path of the desired location.
 +
provides the location where all unique files will be copied from the <i>--source</i> directory.  It's important to note that each unique file will retain it's location after its copy.  For example, if we've executed 'dedup --source=/tmp/source --target=/tmp/target ...' and the source directory contains the unique file /tmp/<b>source</b>/garden/2011/sunflowers.jpg, when it's copied into the target, the directory would be /tmp/<b>target</b>/garden/2011/sunflowers.jpg.
 +
 
 +
: * <b>-L</b> defines the unique database of files from a prior run.  Like the <i>--temp</i> directory, this is optional, but allows a user to compare additional (possibly) duplicate files against the database from a prior run.  So, given the pictures example above, this would allow a person to compare any newly found pictures against a prior database containing the unque file hashes from a previous run.
  
  

Revision as of 11:38, 23 August 2017

Unless you are creating a Linux distro that is a fork of another one with a well stocked software repository, the thought of having to build one from scratch is breath taking. To tackle the thousands of source code downloads, applying necessary patches, figuring out the proper compile flags and various make systems (autogen, autoconf, bootstrap, cmake, etc), and applying file and directory ownership and permissions is quite a job! Fortunately builder will handle all of these tasks for you and already comes with a large pool of software profiles that can be tailored to your individual needs, or left to assume defaults.

 What about if your repo is already a mess

- names are not uniform

    Example: foo.tcz in x86 might be foo-ng.tcz in x86_64

- packages are not split the same way

    Example: foo.tcz in x86 might include binaries, documentation, and libraries, whereas there might be foo.tcz, foo-lib.tcz, foo-doc.tcz in x86_64)

- the existence of a package in one repo does not mean it is available in another

    Example: foo.tcz exists for x86, but not for x86_64 or rpi

- the source code used to build the packages may not be the same

    Example: foo.tcz in x86 might be using 1.2.3 version of source code and the x86_64 might use 1.3.4

- the compile flags are not the same

    Example: foo.tcz in x86 might use "--option1=xyz --option2 --option3", whereas "--option2 --option5" may be used to compile for x86_64

- the file and directory permissions are not set correctly


benefits: constant updates consistency between CPU architectures an immediate access to a large repository of software


License

This projects' codebase is licensed under the 2-clause BSD which can be found at "Wikipedia".



Features

While there may be more features than are currently listed below, this should give potential users at least a portion of the currently implemented features.

  • Simple syntax for easy compiling no matter build system
  • Works with autogen, autoconf, bootstrap, cmake, and traditional
  • Extreme flexibility using various pre and post step scripts
  • Can pull source code from private or public locations
  • Small footprint - contains only two files!
  • Portable using POSIX shell



PARAMETERS

Among the standard 'help' and 'version' actions, this project also contains several others that will be covered below in greater detail.

* -a allows the user to specify which CPU architecture to use during the compile process. Currently builder supports three including ARM v6/7 (r32) and AMD/Intel 32-bit (i32) and 64-bit (i64). The former is typically used for Raspberry Pi boards.
* -d is used to build packages based on directory names instead of the supplied package name via the '-n' parameter. So, for example, if you need to create several packages from a single execution of builder such as alsa and alsa-config, you can pass this parameter to accomplish this task. It is important to note, however, that the directory names will become the package names and will need to be setup in the various pre/post scripts.
* -D in the instance that you have an online repo setup with the source code files already located there, passing this switch can automatically download those files instead of the package maintainer having to manually find and download it beforehand. It uses the URL_CODE value from the builder.conf file in order to locate the proper source tarball from you repo.
* -L similar to the '-D' option, this one will use a local (networked|attached) resource to fetch the source code files instead of having to go over the Internet. This one works with the DIR_REPO variable value in the builder.conf file.

is used to indicate an alternative temporary directory for processing. This parameter is optional, but should be a full path of the desired location. provides the location where all unique files will be copied from the --source directory. It's important to note that each unique file will retain it's location after its copy. For example, if we've executed 'dedup --source=/tmp/source --target=/tmp/target ...' and the source directory contains the unique file /tmp/source/garden/2011/sunflowers.jpg, when it's copied into the target, the directory would be /tmp/target/garden/2011/sunflowers.jpg.

* -L defines the unique database of files from a prior run. Like the --temp directory, this is optional, but allows a user to compare additional (possibly) duplicate files against the database from a prior run. So, given the pictures example above, this would allow a person to compare any newly found pictures against a prior database containing the unque file hashes from a previous run.



Installation

The installation of the software could not be easier. Simply download the package, uncompress it, and run the 'install.sh' script with an elevated account (e.g. root) or prefixing with 'sudo' such as:

$ cd /tmp
$ tar zxf 2017.08.22-builder.tgz
$ cd builder
$ sudo ./install.sh

Checking the effective UID: [done]

Installing the software: [done]

Congrats! Your software is installed and ready to use!



Example

In this example, we are going to assume that the libbsd source code has been downloaded and extracted under the /tmp directory

/etc/builder $ ls
builder.conf   libbsd
/etc/builder $ cd /tmp
/tmp $ builder -n libbsd -a i64

Performing filesystem checks...

Installing compile-time dependencies...

Entering source code directory...

Applying software patches...

Generating the compile script...

Compiling the software...

Copying in the extra files...

Separating into various packages...

Applying ownership and permissions...

Creating the various packages...
   bsd [dev]
   bsd [doc]
   bsd [lib]

Congrats, the software has been packaged successfully!



Developers

Dave Henderson [dhenderson (at) cliquesoft (dot) org]