builder
Unless you are creating a Linux distro that is a fork of another one with a well stocked software repository, the thought of having to build one from scratch is breath taking. To tackle the thousands of source code downloads, applying necessary patches, figuring out the proper compile flags and various make systems (autogen, autoconf, bootstrap, cmake, etc), and applying file and directory ownership and permissions is quite a job! Fortunately builder will handle all of these tasks for you and already comes with a large pool of software profiles that can be tailored to your individual needs, or left to assume defaults.
What about if your repo is already a mess
- names are not uniform
Example: foo.tcz in x86 might be foo-ng.tcz in x86_64
- packages are not split the same way
Example: foo.tcz in x86 might include binaries, documentation, and libraries, whereas there might be foo.tcz, foo-lib.tcz, foo-doc.tcz in x86_64)
- the existence of a package in one repo does not mean it is available in another
Example: foo.tcz exists for x86, but not for x86_64 or rpi
- the source code used to build the packages may not be the same
Example: foo.tcz in x86 might be using 1.2.3 version of source code and the x86_64 might use 1.3.4
- the compile flags are not the same
Example: foo.tcz in x86 might use "--option1=xyz --option2 --option3", whereas "--option2 --option5" may be used to compile for x86_64
- the file and directory permissions are not set correctly
benefits:
constant updates
consistency between CPU architectures
an immediate access to a large repository of software
A simple utility script used for de-duplication of data pools such as your pictures or documents. There's no deletion of the original files, but instead all unique data is copied to a separate directory so the originals can be deleted, backed up, or some other desired task. Not unlike our other bash-script-based software, this project relies on our clAPI framework for various functionality, so be sure this dependency is satisfied before using. It's also worth mentioning that the placement of the OPTION's must follow their respective ACTION (or parent script) which can be determined via the --help output. It might also help to read over the basics of clAPI to get a better understanding when running software from the command line.
Terms
This projects' codebase is licensed under the 2-clause BSD which can be found at "Wikipedia".
ACTION's
Among our standard 'help', 'version', and 'update' ACTIONs, this project also contains two others - 'install' and 'sort'. The 'install' ACTION simply installs the script in the "~/.bin" directory for XiniX and "/usr/bin" for typical GNU/Linux distros. To see how easy it is to install, see the examples section.
The other ACTION, 'sort', will perform most of the desired work. It will basically run through the entire directory heirarchy of the --source OPTION and create a database of MD5 hashes per compared file. During this comparison, if the DB does not contain the generated MD5 hash, then the file is unique and will be copied to the --target directory - retaining it's location. On the other hand, if the MD5 hash is present in the database, then the file is a duplicate and no further action is taken against it. After 'dedup' has completed its run, all unique data is now located under the --target directory.
OPTION's
There aren't very many options currently available for the 'sort' ACTION, but more are planned for future versions. Below we'll cover each to provide you with a detailed explaination of their function.
- --source allows the user to specify the "parent" directory containing the various data pools. For example, lets say that you've ended up with your collection of pictures spread throughout different directories (aka folders) in the file system. Since this project doesn't allow for multiple --source OPTIONs, all the directories containing your pictures will need to be moved or copied into a single "parent" directory and passed as this OPTIONs value.
- --target provides the location where all unique files will be copied from the --source directory. It's important to note that each unique file will retain it's location after its copy. For example, if we've executed 'dedup --source=/tmp/source --target=/tmp/target ...' and the source directory contains the unique file /tmp/source/garden/2011/sunflowers.jpg, when it's copied into the target, the directory would be /tmp/target/garden/2011/sunflowers.jpg.
- --temp is used to indicate an alternative temporary directory for processing. This parameter is optional, but should be a full path of the desired location.
- --uniquedb defines the unique database of files from a prior run. Like the --temp directory, this is optional, but allows a user to compare additional (possibly) duplicate files against the database from a prior run. So, given the pictures example above, this would allow a person to compare any newly found pictures against a prior database containing the unque file hashes from a previous run.
Installation
The installation of the software could not be easier. Simply download the package, uncompress it, and run the 'install.sh' script with an elevated account (e.g. root) or prefixing with 'sudo' such as:
$ cd /tmp $ tar zxf 2017.08.22-builder.tgz $ cd builder $ sudo ./install.sh Checking the effective UID: [done] Installing the software: [done] Congrats! Your software is installed and ready to use!
Example
In this example, we are going to assume that the libbsd source code has been downloaded and extracted under the /tmp directory
/etc/builder $ ls builder.conf libbsd /etc/builder $ cd /tmp /tmp $ builder -n libbsd -a i64 Performing filesystem checks... Installing compile-time dependencies... Entering source code directory... Applying software patches... Generating the compile script... Compiling the software... Copying in the extra files... Separating into various packages... Applying ownership and permissions... Creating the various packages... bsd [dev] bsd [doc] bsd [lib] Congrats, the software has been packaged successfully!
Developers
Dave Henderson [dhenderson (at) cliquesoft (dot) org]
