backup.lite

This project allows a user to backup and/or sync their data between multiple locations including local directories, network shares, and remote nodes using SSH with optional email contact upon success and/or failure of job. Not unlike our other bash-script-based software, this project relies on our clAPI framework for various functionality, so be sure this dependency is installed before using. It's also worth mentioning, when using this software from the command prompt, that the placement of the OPTION's must follow their respective ACTION (or parent script) which can be determined via the --help output. It might also help to read over the basics of clAPI to get a better understanding when running software from the command line.

Terms

This projects' codebase is licensed under the AGPLv3 unless a valid CPL has been purchased. More information about both of these licenses can be found under the "Our Licenses" link of our homepage.

Using the CLI

ACTION's

Among our standard 'help', 'version', and 'update' command line ACTIONs, this project also contains several others that are available - 'backup', 'sync', and 'restore'. It's probably pretty obvious what task each of those handles, so we'll go ahead and move into the OPTIONs section below...

Job Wizard

This OPTION belongs to the parent script of the project since parameters must accompany their respective "module" in order to be processed correctly. Useful for initially creating backup jobs, this parameter initiates a wizard that will walk you through all of the necessary steps. Starting this process is as simple as executing backup.lite --wizard at the command line.

Backup OPTION's

• --source specifies the path or network share that needs to be backed up. Currently this parameter can accept a local directory or networked path as its value. Please note that network paths indicate the network type, for instance '\\192.168.0.10\Share' indicates a Microsoft Windows share (SMB/CIFS) whereas '//192.168.0.11/Share' indicates a Linux share (NFS). When using this parameter, be sure to surround it using single quotes!

• --target can be accept the same values as the --source parameter, but represents the target location where the backup archives should be stored.

• --targetDir is a directory that is appended to the --target value to create a storage path so that separation and combination of the backup archives generated and maintained by the software can be determined by the backup administrator. For example, lets say that you are backing up two servers called "ServerA" and "ServerB" with each server having incremental backup jobs Monday through Thursday and a full backup on Friday. It might be desired to have all the archives of each backup job relating to the separate severs being grouped in the same directory. So for each backup job relating to each server, you could simply specify the same --target (e.g. /var/backups) and --targetDir values (e.g. ServerA) to have the archives grouped together (e.g. /var/backups/ServerA).

• --sourceUser provides the username to access the --source in the event that its value is a network share only! If no username and password are required to gain access to the --source network share, then simply do not use this parameter or the following two.

• --sourcePass provides the usernames corresponding password.

NOTE: later versions will use this to specify a password file for better security.

• --sourceDom provides the optional domain as part of the credentials when authenticating to a network share.

• --targetUser same as the --sourceUser parameter but for a --target network share.

• --targetPass same as the --sourcePass parameter but for a --target network share.

NOTE: later versions will use this to specify a password file for better security.

• --targetDom same as the --sourceDom parameter but for a --target network share.

• --compression specifies the type of compression to apply on the backup archive.

• --count regulates how many backup archives of the configured --type (e.g. daily, incremental, full, etc) are maintained in the --targetDir directory. As an example, if you have only one full backup job being stored in a unique --targetDir and set this value to "4", then only 4 full backup archives will be maintained.

• --include indicates the relative path and filename (e.g. ./dir/to/file/filename) that contains the specified data to backup. It should have one entry per line and should be relative to the --source value. For flexibility, the user may use wildcards and other allowed syntax via tar's --files-from parameters.

• --exclude same as the --include parameter, but for data that needs to be excluded during the backup process.

• --type specifies which type of backup should occur during the backup job. Each type backs up a different set of data which can be found here. If "custom" is used as the value here, the user must specify all of the tar options, with the exception of "-c -f -v --totals", as the value of the --params parameter below.

• --verify will verify the data integrity within the backup archive against the live data to help ensure errors in the backup process itself.

• --params allows administrators that require additional tar options the ability to pass them directly to tar with this parameter. Also see the --type parameter above. NOTE: invalid or incorrect values here can break the operation of this software - use at your own risk!

Restore OPTION's

• --source provides the local directory or network share containing the backup archives to restore data from. For notes on how to enter network shares, please see the "Backup OPTIONS's" above. Also see the --sourceDir parameter.

• --target indicates the local directory or network share that will receive the restored data. For notes on how to enter network shares, please see the "Backup OPTIONS's" above.

• --sourceDir defines the directory contained within the --source that's actually containing the backup archives. This is similar to --targetDir in the "Backup OPTION's".

• --sourceUser provides the username to access the --source in the event that its value is a network share only! If no username and password are required to gain access to the --source network share, then simply do not use this parameter or the following two.

• --sourcePass provides the usernames corresponding password.

NOTE: later versions will use this to specify a password file for better security.

• --sourceDom provides the optional domain as part of the credentials when authenticating to a network share.

• --targetUser same as the --sourceUser parameter but for a --target network share.

• --targetPass same as the --sourcePass parameter but for a --target network share.

NOTE: later versions will use this to specify a password file for better security.

• --targetDom same as the --sourceDom parameter but for a --target network share.

• --archive sets the path and filename of the (first) archive containing the data that the user wants restored. As an added benefit, this value can also contain just the prefix of a filename (e.g. '12345' instead of '12345.d.tgz') and the remaining extension will automatically be determined. Also see the --archives parameter below.

• --archives specifies how many consecutive archives' data should be restored after the --archive, based on each archives date. If this value isn't defined, it is assumed to restore all data provided by --include from each successive archive until the end is reached. This is useful, for example, when restoring a complete system using the last full backup and its following incremental archives. Also please note, if you wanted to restore from 3 archives total, this value should actually be 2 since --archive always counts as 1.

• --collapse removes the directory structure when restoring data with duplicate files being overwritten as restored.

• --data provides an alternative to --include and allows a user to enter a comma separated list, or optionally "*" for "all data", of files to restore from the --archive. This is useful when only a few files need to be restored from an archive and can easily be entered via the command line.

• --include indicates the relative path and filename (e.g. ./dir/to/file/filename) that contains the specified data to restore. It should have one entry per line and should be relative to the --source value. For flexibility, the user may use wildcards and other allowed syntax via tar's --files-from parameters.

• --exclude same as the --include parameter, but for data that needs to be excluded during the restore process.

• --overwrite specifies which existing files should be overwritten during the restore process.

• --params allows administrators that require additional tar options the ability to pass them directly to tar with this parameter. NOTE: invalid or incorrect values here can break the operation of this software - use at your own risk!

Sync OPTION's

• --port defines the port to communicate over when working with a remote node.

• --source is the directory or network share that needs to be duplicated. Unless you're performing a sync only job, or you require an alternative source directory for some reason, this value will inherit the value of the backup modules --source parameter.

• --target identifies the rsync daemon, host, or local directory needing to be updated. Like the --source parameter, if this value isn't specified, the value of the backup modules --target parameter is assumed.

NOTE: currently rsync daemons aren't supported, but will in future versions.

• --dir specifies the parent directory in the --target that requires sync'ing. This acts similar to the backup ACTION's --targetDir parameter.

• --tunnelUser provides the username to establish an SSH tunnel with the remote node. If a set of preshared keys have been exchanged, then simply do not use this parameter or the following one.

• --tunnelPass provides the usernames corresponding password.

• --params allows administrators that require additional rysnc options the ability to pass them directly to rsync with this parameter. NOTE: invalid or incorrect values here can break the operation of this software - use at your own risk!

Examples

Installation is a simple 2-step process...

$ cd /path/to/uncompressed/package
$ ./install

Backing up data only (no sync'ing)...

$ sudo ./backup.lite --noprompts --name=test backup --type=full --source='/tmp/temp' --target='/tmp/backup' --compression=gzip --include='/etc/backup/test.inc'

Beginning the "test" backup job @ Tue Mar 27 10:37:44 EDT 2012

Checking system environment...

  (i)   Directories...
           Configs: [checking] [exists] [writable] [success] [done]
           Temp: [checking] [exists] [writable] [success] [done]
           Mounts: [checking] [exists] [writable] [success] [done]
  (i)   Variables...
           --name: [done]
           ACTION: [done]

Beginning the 'backup' module...

  (i)   Processing the source and target...
           Source: [symlinking] [success] [done]
           Target: [local] [checking] [exists] [writable] [success] [symlinking] [success] [done]
           Storage: [checking] [exists] [done]
  (i)   Checking variables: [type] [include] [done]
  (i)   Processing a(n) "full" backup  -----------------------------[PROGRESS]--

./
./original/
./original/flash.img
./a/
./a/b/
./a/b/test.txt
./a/b/test.sh
./flash.tar.gz
Total bytes written: 656384000 (626MiB, 31MiB/s)

--------------------------------------------------------------------------------
  (i)   Creating a catalog of backed up data: [done]
  (i)   Cycling the backups...
           Archive: [deleting] [success] [done]
           Catalog: [absent] [done]
           Job Log: [absent] [done]
  (i)   Check archive count: [within limits] [done]
  (i)   Compressing the archive: [gzip]
--------------------------------------------------------------------[PROGRESS]--
 626MB 0:01:05 [9.56MB/s] [==================================>] 100%            
--------------------------------------------------------------------------------
           [deleting] [success] [done]

Calling exit routines for the modules...

  (i)   backup.lite script...
           Source: [checking] [unmounted] [deleting] [success] [done]
           Target: [checking] [unmounted] [deleting] [success] [done]
           Cleanup: [deleting] [success] [done]

The job has completed successfully @ Tue Mar 27 10:39:16 EDT 2012

Using the Web Interface

NOTE: this section applies for local and remote graphical user interfaces.

This section will discuss the options available to the user when interacting with various modules within the graphical interface, but before we get started, we'll need to explain how to make it available in the first place. Before you get discouraged, this process is nowhere near as complex as it may sound and can be accomplished with only one command, thanks to the clAPI framework! We'll now need to setup and/or start a web server that will enable the use of the GUI. If either of the following conditions are met, you can skip to the final paragraph of this section:

• If you're using a "XiniX Compatible" device
• If you installed this software using some type of graphical installer

• Typical Users The easiest way to accomplish this goal is by starting clAPI's built-in web server using a single command entered at the prompt within a terminal window:

Type the following:

$ clapi --noprompts --verbose shttpd

To produce the following:

Checking system environment...

  (i)   Directories...
           Conf: [checking] [exists] [writable] [success] [done]
           Logs: [checking] [exists] [writable] [success] [done]
           Temp: [checking] [exists] [writable] [success] [done]
  (i)   User Interface: [console] [done]

Beginning the 'shttpd' module...

  (i)   Initializing the web server on 127.0.0.1 over 8080: [serving]
--------------------------------------------------------------------[PROGRESS]--
Sat Jul  7 20:15:06 UTC 2012    INFO    Changing into the serving directory
Sat Jul  7 20:15:06 UTC 2012    INFO    Creating environment...
Sat Jul  7 20:15:06 UTC 2012    INFO    Starting UI dependencies...
Sat Jul  7 20:15:06 UTC 2012    INFO    Starting web server...

• Administrators For the admins that already have a web server installed such as paged, Apache, IIS, nginx, lighttpd, boa, or any other, you can simply point it to the directory that the 'GUI files' were installed to during the clAPI and backup.lite installation. Afterwards, restart your web server and you're good to go!

If you've made it this far, then you now have access the graphical interface of the software using any web browser. For users working with this software on a local device (e.g. personal computer, tablet, smart phone, etc), simply navigate to "http://127.0.0.1:8080", alternatively "http://localhost:8080", and after providing the authentication credentials for a valid system user account, you can interact with all of the installed modules. Users accessing this on a remote device (e.g. Backup server, NAS, etc) will need to use "http://x.x.x.x:8080" where "x.x.x.x" represents the IP address of the device. If this doesn't work for you, contact your network administrator or technical support company.

Backup Module

Overview Tab

• Archive Issues (button) This button will archive all the information that's presented in the "Issues that need Attention" section of the screen. This can be useful if that information will ever be required in the future.

• Scheduled Jobs As the name suggests, this box shows all the upcoming scheduled tasks along with their scheduled job number and planned execution time. There are no interactions possible in this area as all the content presented here is information only.

• Completed Jobs This area lists all of corresponding logs of each backup job that has completed succesfully. [NOTE: Please be aware that since these logs correspond to the currently stored backup archives, as they rotate, so do the logs!] This box does present the ability for the user to view any of those logs by simply clicking the log ID and then clicking the "Log" tab.

• Issues that need Attention In order to minimize the work needed to resolve encountered problems, this section was provided to show you the recorded errors along with one-click access to the logs (see above) for details.

Jobs Tab

• New (button)

• Rename (button)

• Copy (button)

• Delete (button)

• Save (button)

• Saved job

• Type & Compression

• Start date & time

• Frequency

• Data storage

• Miscellaneous

• Contact name

• Email address

• Job status

• Source backup path

• Connect as...

• Target storage path

• Connect as...

• Category/ID

• Sync URI & Port

• Remote directory

• Connect as...

Data Tab

• Save (button) Once you've got the "Include data" and "Exclude data" lists filled out according to the backup job, click this button to save them to disk.

• Directory listing (left-hand pane) This listing corresponds to the files and folders stored under the "Source backup path" from the "Jobs" tab. You can use this directory listing to add items to the "Included data" and "Excluded data" boxes by simply locating the desired file or folder and dragging it to intended listbox.

NOTE: currently this doesn't work if the "Source backup path" is a network share - we're working on correcting this in future versions.

• Included data By default this screen will have a value of "All data" indicating that all data shown in the "Directory Listing" will be backed up. You can, of course, remove this or any other entries by highlighting it in the list and clicking the associated "-" button. For flexibility, the ability to add manual entries, possibly containing optional wildcards, can be entered using the corresponding textbox and clicking the "+" button.

• Excluded data This box contains the same functionality as the "Included data" section, but instead, refers to files or folders that should not be included in the backup. This can work even if the "Included data" box contains "All data".

Log Tab

• Archive (button) This button will allow the user to archive the currently loaded log for manual deletion of their own accord.

NOTE: if you want to load a log in this screen, see the "Overview" tab options above.

Status Tab

• Clear Completed (button) Any backup jobs that are started from within this software using the "Run now" button on the "Jobs" tab will show their progress on this screen. Clicking this button will clear all of the completed jobs from the listing.

About Tab

This screen shows information related to the software that would be useful for troubleshooting any experienced problems. There's also several links on this page to hear from our users and community as well as finding help for the software itself.

Restore Module

Overview Tab

• Archive Issues (button) This button will archive all the information that's presented in the "Issues that need Attention" section of the screen. This can be useful if that information will ever be required in the future.

• Completed Jobs This area lists all of corresponding logs of each restore job that has completed succesfully. This box does present the ability for the user to view any of those logs by simply clicking the log ID and then clicking the "Log" tab.

• Issues that need Attention In order to minimize the work needed to resolve encountered problems, this section was provided to show you the recorded errors along with one-click access to the logs (see above) for details.

Jobs Tab

• Restore (button) Starts a restoration of designated data (from the "Data" tab) based on the currently viewed "Saved jobs" value using the configuration of the parameters accessible from within this screen.

• Saved jobs Lists all the current backup jobs, active or inactive, providing access to their archives' contained data for restoration. During the initial loading of this module or upon changing this value, the "Associated Archives" change dynamically to match all corresponding archives relating to this value.

• Restore options These two options control what data gets restored from the archive with the ability to retain or remove the directory structure containing your desired data for restoration. So, for example, if you wanted to restore a file called "abc.txt" contained within the directory "Documents/personal/abc.txt" to "/tmp/restore", and you removed the directory structure, instead of finding the restored document under "/tmp/restore/Documents/personal/abc.txt", it would be found under "/tmp/restore/abc.txt".

• Miscellaneous These two options were included for flexibility, but be careful when using them as it can break the functionality of this software! Incorporating these options will allow additional parameters to be passed directly to the underlying binaries (tar and rsync) during the restoration process. They should only include additional parameters, not full syntax of the related command.

• Associated archives

• Source storage path

• Connect as...

• Target restore path

• Connect as...

• Category/ID

• Sync URI & Port

• Remote directory

• Connect as...

• Tar parameters

• Rsync parameters

• Notes on the job

Data Tab

• Restore (button) This button performs the same tasks as the one on the "Jobs" tab.

• Directory listing (left-hand pane) This listing corresponds to the files and folders stored under the "Source storage path" from the "Jobs" tab. You can use this directory listing to add items to the "Included data" and "Excluded data" boxes by simply locating the desired file or folder and dragging it to intended listbox.

NOTE: currently this doesn't work if the "Source backup path" is a network share - we're working on correcting this in future versions.

• Included data By default this screen will have a value of "All data" indicating that all data shown in the "Directory Listing" will be backed up. You can, of course, remove this or any other entries by highlighting it in the list and clicking the associated "-" button. For flexibility, the ability to add manual entries, possibly containing optional wildcards, can be entered using the corresponding textbox and clicking the "+" button.

• Excluded data This box contains the same functionality as the "Included data" section, but instead, refers to files or folders that should not be included in the backup. This can work even if the "Included data" box contains "All data".

Log Tab

• Archive (button) This button will allow the user to archive the currently loaded log for manual deletion of their own accord.

NOTE: if you want to load a log in this screen, see the "Overview" tab options above.

Status Tab

• Clear Completed (button) Any backup jobs that are started from within this software using the "Run now" button on the "Jobs" tab will show their progress on this screen. Clicking this button will clear all of the completed jobs from the listing.

About Tab

This screen shows information related to the software that would be useful for troubleshooting any experienced problems. There's also several links on this page to hear from our users and community as well as finding help for the software itself.

Developers

Dave Henderson [dhenderson (at) cliquesoft (dot) org]