View source for backup.lite

This project allows a user to backup and/or sync their data between multiple locations including local directories, network shares, and remote nodes using SSH with optional email contact upon success and/or failure of job.  Not unlike our other bash-script-based software, this project relies on our clAPI framework for various functionality, so be sure this dependency is installed before using.  It's also worth mentioning, when using this software from the command prompt, that the placement of the OPTION's <b>must</b> follow their respective ACTION (or parent script) which can be determined via the <i>--help</i> output.  It might also help to read over the basics of [http://wiki.cliquesoft.org/index.php?title=clAPI clAPI] to get a better understanding when running software from the command line.


==Terms==
This projects' codebase is licensed under the AGPLv3 unless a valid CPL has been purchased.  More information about both of these licenses can be found under the "[http://www.cliquesoft.org/index.php?page=our_licenses Our Licenses]" link of our [http://www.cliquesoft.org/ homepage].




==Installation==
Installation couldn't be more easy!  Currently it's a simple 2-step process which can be accomplished via the following two commands.  After you execute the installation script, answer all the questions and your software will be ready for use!
<pre>
$ cd /path/to/uncompressed/package
$ ./install
</pre>




==Using the CLI==

===ACTION's===
Among our standard 'help', 'version', and 'update' command line ACTIONs, this project also contains several others that are available - 'backup', 'sync', and 'restore'.  It's probably pretty obvious what task each of those handles, so we'll go ahead and move into the OPTIONs section below...


====Job Wizard====
This OPTION belongs to the parent script of the project since parameters must accompany their respective "module" in order to be processed correctly.  Useful for initially creating backup jobs, this parameter initiates a wizard that will walk you through all of the necessary steps.  Starting this process is as simple as executing <i>backup.lite --wizard</i> at the command line.


====Backup OPTION's====

* <b>--source</b> specifies the path or network share that needs to be backed up. Currently this parameter can accept a local directory or networked path as its value.  Please note that network paths indicate the network type, for instance '\\192.168.0.10\Share' indicates a Microsoft Windows share (SMB/CIFS) whereas '//192.168.0.11/Share' indicates a Linux share (NFS). When using this parameter, be sure to surround it using <u>single quotes</u>!

* <b>--target</b> can be accept the same values as the --source parameter, but represents the target location where the backup archives should be stored.

* <b>--targetDir</b> is a directory that is appended to the --target value to create a storage path so that separation and combination of the backup archives generated and maintained by the software can be determined by the backup administrator. For example, lets say that you are backing up two servers called "ServerA" and "ServerB" with each server having incremental backup jobs Monday through Thursday and a full backup on Friday. It might be desired to have all the archives of each backup job relating to the separate severs being grouped in the same directory. So for each backup job relating to each server, you could simply specify the same --target (e.g. /var/backups) and --targetDir values (e.g. ServerA) to have the archives grouped together (e.g. /var/backups/ServerA).

* <b>--sourceUser</b> provides the username to access the --source in the event that its value is a network share <u>only</u>!  If no username and password are required to gain access to the --source network share, then simply do not use this parameter or the following two.

* <b>--sourcePass</b> provides the usernames corresponding password.
NOTE: later versions will use this to specify a password file for better security.

* <b>--sourceDom</b> provides the optional domain as part of the credentials when authenticating to a network share.

* <b>--targetUser</b> same as the --sourceUser parameter but for a --target network share.

* <b>--targetPass</b> same as the --sourcePass parameter but for a --target network share.
NOTE: later versions will use this to specify a password file for better security.

* <b>--targetDom</b> same as the --sourceDom parameter but for a --target network share.

* <b>--compression</b> specifies the type of compression to apply on the backup archive.

* <b>--count</b> regulates how many backup archives of the configured --type (e.g. daily, incremental, full, etc) are maintained in the --targetDir directory.  As an example, if you have only one full backup job being stored in a unique --targetDir and set this value to "4", then only 4 full backup archives will be maintained.

* <b>--include</b> indicates the relative path and filename (e.g. ./dir/to/file/filename) that contains the specified data to backup. It should have one entry per line and should be relative to the --source value. For flexibility, the user may use wildcards and other allowed syntax via tar's --files-from parameters.

* <b>--exclude</b> same as the --include parameter, but for data that needs to be excluded during the backup process.

* <b>--type</b> specifies which type of backup should occur during the backup job. Each type backs up a different set of data which can be found [https://en.wikipedia.org/wiki/Glossary_of_backup_terms here]. If "custom" is used as the value here, the user must specify <u>all</u> of the tar options, with the exception of "-c -f -v --totals", as the value of the --params parameter below.

* <b>--verify</b> will verify the data integrity within the backup archive against the live data to help ensure errors in the backup process itself.

* <b>--params</b> allows administrators that require additional tar options the ability to pass them directly to tar with this parameter.  Also see the --type parameter above. NOTE: invalid or incorrect values here can break the operation of this software - use at your own risk!


====Restore OPTION's====

* <b>--source</b> provides the local directory or network share containing the backup archives to restore data from. For notes on how to enter network shares, please see the "Backup OPTIONS's" above. Also see the --sourceDir parameter.

* <b>--target</b> indicates the local directory or network share that will receive the restored data. For notes on how to enter network shares, please see the "Backup OPTIONS's" above.

* <b>--sourceDir</b> defines the directory contained within the --source that's actually containing the backup archives. This is similar to --targetDir in the "Backup OPTION's".

* <b>--sourceUser</b> provides the username to access the --source in the event that its value is a network share <u>only</u>!  If no username and password are required to gain access to the --source network share, then simply do not use this parameter or the following two.

* <b>--sourcePass</b> provides the usernames corresponding password.
NOTE: later versions will use this to specify a password file for better security.

* <b>--sourceDom</b> provides the optional domain as part of the credentials when authenticating to a network share.

* <b>--targetUser</b> same as the --sourceUser parameter but for a --target network share.

* <b>--targetPass</b> same as the --sourcePass parameter but for a --target network share.
NOTE: later versions will use this to specify a password file for better security.

* <b>--targetDom</b> same as the --sourceDom parameter but for a --target network share.

* <b>--archive</b> sets the path and filename of the (first) archive containing the data that the user wants restored. As an added benefit, this value can also contain just the prefix of a filename (e.g. '12345' instead of '12345.d.tgz') and the remaining extension will automatically be determined. Also see the --archives parameter below.

* <b>--archives</b> specifies how many consecutive archives' data should be restored after the --archive, based on each archives date. If this value isn't defined, it is assumed to restore all data provided by --include from each successive archive until the end is reached. This is useful, for example, when restoring a complete system using the last full backup and its following incremental archives. Also please note, if you wanted to restore from 3 archives total, this value should actually be 2 since --archive always counts as 1.

* <b>--collapse</b> removes the directory structure when restoring data with duplicate files being overwritten as restored.

* <b>--data</b> provides an alternative to --include and allows a user to enter a comma separated list, or optionally "*" for "all data", of files to restore from the --archive.  This is useful when only a few files need to be restored from an archive and can easily be entered via the command line.

* <b>--include</b> indicates the relative path and filename (e.g. ./dir/to/file/filename) that contains the specified data to restore. It should have one entry per line and should be relative to the --source value. For flexibility, the user may use wildcards and other allowed syntax via tar's --files-from parameters.

* <b>--exclude</b> same as the --include parameter, but for data that needs to be excluded during the restore process.

* <b>--overwrite</b> specifies which <u>existing</u> files should be overwritten during the restore process.

* <b>--params</b> allows administrators that require additional tar options the ability to pass them directly to tar with this parameter. NOTE: invalid or incorrect values here can break the operation of this software - use at your own risk!


====Sync OPTION's====

* <b>--port</b> defines the port to communicate over when working with a remote node.

* <b>--source</b> is the directory or network share that needs to be duplicated. Unless you're performing a sync only job, or you require an alternative source directory for some reason, this value will inherit the value of the backup modules --source parameter.

* <b>--target</b> identifies the rsync daemon, host, or local directory needing to be updated. Like the --source parameter, if this value isn't specified, the value of the backup modules --target parameter is assumed.
NOTE: currently rsync daemons aren't supported, but will in future versions.

* <b>--dir</b> specifies the parent directory in the --target that requires sync'ing.  This acts similar to the backup ACTION's --targetDir parameter.

* <b>--tunnelUser</b> provides the username to establish an SSH tunnel with the remote node. If a set of preshared keys have been exchanged, then simply do not use this parameter or the following one.

* <b>--tunnelPass</b> provides the usernames corresponding password.

* <b>--params</b> allows administrators that require additional rysnc options the ability to pass them directly to rsync with this parameter. NOTE: invalid or incorrect values here can break the operation of this software - use at your own risk!




===Examples===
Backing up data only (no sync'ing)...
<pre>
$ sudo ./backup.lite --noprompts --name=test backup --type=full --source='/tmp/temp' --target='/tmp/backup' --compression=gzip --include='/etc/backup/test.inc'

Beginning the "test" backup job @ Tue Mar 27 10:37:44 EDT 2012

Checking system environment...

  (i)   Directories...
           Configs: [checking] [exists] [writable] [success] [done]
           Temp: [checking] [exists] [writable] [success] [done]
           Mounts: [checking] [exists] [writable] [success] [done]
  (i)   Variables...
           --name: [done]
           ACTION: [done]

Beginning the 'backup' module...

  (i)   Processing the source and target...
           Source: [symlinking] [success] [done]
           Target: [local] [checking] [exists] [writable] [success] [symlinking] [success] [done]
           Storage: [checking] [exists] [done]
  (i)   Checking variables: [type] [include] [done]
  (i)   Processing a(n) "full" backup  -----------------------------[PROGRESS]--

./
./original/
./original/flash.img
./a/
./a/b/
./a/b/test.txt
./a/b/test.sh
./flash.tar.gz
Total bytes written: 656384000 (626MiB, 31MiB/s)

--------------------------------------------------------------------------------
  (i)   Creating a catalog of backed up data: [done]
  (i)   Cycling the backups...
           Archive: [deleting] [success] [done]
           Catalog: [absent] [done]
           Job Log: [absent] [done]
  (i)   Check archive count: [within limits] [done]
  (i)   Compressing the archive: [gzip]
--------------------------------------------------------------------[PROGRESS]--
 626MB 0:01:05 [9.56MB/s] [==================================>] 100%            
--------------------------------------------------------------------------------
           [deleting] [success] [done]

Calling exit routines for the modules...

  (i)   backup.lite script...
           Source: [checking] [unmounted] [deleting] [success] [done]
           Target: [checking] [unmounted] [deleting] [success] [done]
           Cleanup: [deleting] [success] [done]

The job has completed successfully @ Tue Mar 27 10:39:16 EDT 2012
</pre>




==Using the Web Interface==
NOTE: this section applies for local and remote graphical user interfaces.

This section will discuss the options available to the user when interacting with various modules within the graphical interface, but before we get started, we'll need to explain how to make it available in the first place.  <b>Before</b> you get discouraged, this process is nowhere near as complex as it may sound and can be accomplished with only one command, thanks to the [http://wiki.cliquesoft.org/index.php?title=clAPI clAPI] framework!  We'll now need to setup and/or start a web server that will enable the use of the [https://en.wikipedia.org/wiki/Graphical_user_interface GUI]. If either of the following conditions are met, you can skip to the final paragraph of this section:

* If you're using a "XiniX Compatible" device
* If you installed this software using some type of graphical installer


* <b>Typical Users</b> The easiest way to accomplish this goal is by starting clAPI's built-in web server using a single command entered at the prompt within a terminal window:

Type the following:
<pre>
$ clapi --noprompts --verbose shttpd
</pre>
To produce the following:
<pre>
Checking system environment...

  (i)   Directories...
           Conf: [checking] [exists] [writable] [success] [done]
           Logs: [checking] [exists] [writable] [success] [done]
           Temp: [checking] [exists] [writable] [success] [done]
  (i)   User Interface: [console] [done]

Beginning the 'shttpd' module...

  (i)   Initializing the web server on 127.0.0.1 over 8080: [serving]
--------------------------------------------------------------------[PROGRESS]--
Sat Jul  7 20:15:06 UTC 2012    INFO    Changing into the serving directory
Sat Jul  7 20:15:06 UTC 2012    INFO    Creating environment...
Sat Jul  7 20:15:06 UTC 2012    INFO    Starting UI dependencies...
Sat Jul  7 20:15:06 UTC 2012    INFO    Starting web server...
</pre>

* <b>Administrators</b> For the admins that already have a web server installed such as [http://wiki.cliquesoft.org/index.php?title=paged paged], [http://httpd.apache.org/ Apache], [http://www.iis.net/ IIS], [http://www.nginx.org/ nginx], [http://www.lighttpd.net/ lighttpd], [http://www.boa.org/ boa], or any [https://en.wikipedia.org/wiki/Comparison_of_web_server_software other], you can simply point it to the directory that the 'GUI files' were installed to during the clAPI and backup.lite installation.  Afterwards, restart your web server and you're good to go!


If you've made it this far, then you now have access the graphical interface of the software using any web browser.  For users working with this software on a local device (e.g. personal computer, tablet, smart phone, etc), simply navigate to "http://127.0.0.1:8080", alternatively "http://localhost:8080", and after providing the authentication credentials for a valid system user account, you can interact with all of the installed modules.  Users accessing this on a remote device (e.g. Backup server, NAS, etc) will need to use "http://x.x.x.x:8080" where "x.x.x.x" represents the IP address of the device.  If this doesn't work for you, contact your network administrator or [http://www.digital-pipe.com/ technical support company].


===Backup Module===

====Overview Tab====

* <b>Archive Issues (button)</b> This button will archive all the information that's presented in the "Issues that need Attention" section of the screen.  This can be useful if that information will ever be required in the future.

* <b>Scheduled Jobs</b> As the name suggests, this box shows all the upcoming scheduled tasks along with their scheduled job number and planned execution time.  There are no interactions possible in this area as all the content presented here is information only.

* <b>Completed Jobs</b> This area lists all of corresponding logs of each backup job that has completed succesfully. [NOTE: Please be aware that since these logs correspond to the currently stored backup archives, as they rotate, <u>so do the logs</u>!]  This box does present the ability for the user to view any of those logs by simply clicking the log ID and then clicking the "Log" tab.

* <b>Issues that need Attention</b> In order to minimize the work needed to resolve encountered problems, this section was provided to show you the recorded errors along with one-click access to the logs (see above) for details.


====Jobs Tab====

* <b>New (button)</b> 

* <b>Rename (button)</b> 

* <b>Copy (button)</b> 

* <b>Delete (button)</b> 

* <b>Save (button)</b> 

* <b>Saved job</b>

* <b>Type & Compression</b>

* <b>Start date & time</b>

* <b>Frequency</b>

* <b>Data storage</b>

* <b>Miscellaneous</b>

* <b>Contact name</b>

* <b>Email address</b>

* <b>Job status</b>

* <b>Source backup path</b>

* <b>Connect as...</b>

* <b>Target storage path</b>

* <b>Connect as...</b>

* <b>Category/ID</b>

* <b>Sync URI & Port</b>

* <b>Remote directory</b>

* <b>Connect as...</b>


====Data Tab====

* <b>Save (button)</b> Once you've got the "Include data" and "Exclude data" lists filled out according to the backup job, click this button to save them to disk.

* <b>Directory listing (left-hand pane)</b> This listing corresponds to the files and folders stored under the "Source backup path" from the "Jobs" tab.  You can use this directory listing to add items to the "Included data" and "Excluded data" boxes by simply locating the desired file or folder and dragging it to intended listbox.
NOTE: currently this doesn't work if the "Source backup path" is a network share - we're working on correcting this in future versions.

* <b>Included data</b> By default this screen will have a value of "All data" indicating that all data shown in the "Directory Listing" will be backed up.  You can, of course, remove this or any other entries by highlighting it in the list and clicking the associated "-" button.  For flexibility, the ability to add manual entries, possibly containing optional wildcards, can be entered using the corresponding textbox and clicking the "+" button.

* <b>Excluded data</b> This box contains the same functionality as the "Included data" section, but instead, refers to files or folders that should <u>not</u> be included in the backup.  This can work even if the "Included data" box contains "All data".


====Log Tab====

* <b>Archive (button)</b> This button will allow the user to archive the currently loaded log for manual deletion of their own accord.

NOTE: if you want to load a log in this screen, see the "Overview" tab options above.


====Status Tab====

* <b>Clear Completed (button)</b> Any backup jobs that are started from within this software using the "Run now" button on the "Jobs" tab will show their progress on this screen.  Clicking this button will clear all of the completed jobs from the listing.


====About Tab====

This screen shows information related to the software that would be useful for troubleshooting any experienced problems.  There's also several links on this page to hear from our users and community as well as finding help for the software itself.



===Restore Module===

====Overview Tab====

* <b>Archive Issues (button)</b> This button will archive all the information that's presented in the "Issues that need Attention" section of the screen.  This can be useful if that information will ever be required in the future.

* <b>Completed Jobs</b> This area lists all of corresponding logs of each restore job that has completed succesfully.  This box does present the ability for the user to view any of those logs by simply clicking the log ID and then clicking the "Log" tab.

* <b>Issues that need Attention</b> In order to minimize the work needed to resolve encountered problems, this section was provided to show you the recorded errors along with one-click access to the logs (see above) for details.


====Jobs Tab====

* <b>Restore (button)</b> Starts a restoration of designated data (from the "Data" tab) based on the currently viewed "Saved jobs" value using the configuration of the parameters accessible from within this screen.

* <b>Saved jobs</b> Lists all the current backup jobs, active or inactive, providing access to their archives' contained data for restoration.  During the initial loading of this module or upon changing this value, the "Associated Archives" change dynamically to match all corresponding archives relating to this value.

* <b>Restore options</b> These two options control what data gets restored from the archive with the ability to retain or remove the directory structure containing your desired data for restoration.  So, for example, if you wanted to restore a file called "abc.txt" contained within the directory "Documents/personal/abc.txt" to "/tmp/restore", and you removed the directory structure, instead of finding the restored document under "/tmp/restore/Documents/personal/abc.txt", it would be found under "/tmp/restore/abc.txt".

* <b>Miscellaneous</b> These two options were included for flexibility, but are NOT required for a successful restoration.  Incorporating these options will allow additional parameters to be passed directly to the underlying binaries (tar and rsync) during the restoration process.  They should <u>only</u> include additional parameters, not full syntax of the related command.  Please note that using them can <u>break</u> the functionality of this software if you don't know what you're doing!

* <b>Associated archives</b> Here you'll find a list of all the current backup archives associated with the selected "Saved job".  Clicking on any in the list will load its contents in the "Data" tab so you can select what data you would like to restore.  There's also a complementing value when restoring entitled "Include X subsequent archives".  Its intended purpose is to speed up the restoration of selected data among various backup archives so the user doesn't have to restore from each one individually.  For example, lets say that you have a schedule of incremental backups Monday through Thursday and full backups on Friday. Wednesday your server crashes and you need to perform a full restoration of data.  You can simply load your last full backup (from Friday) and enter in '2' for the subsequent archive count (for Monday and Tuesdays backups) and all the selected data from Fridays archive will be restored as well as any matching data in Monday and Tuesdays backups, saving you the hassle of completing those steps yourself.  It's important to note that the subsequent archives that get processed are date oriented, not by the job!

* <b>Source storage path</b> This defines the source location where the backup archives are currently stored <u>without</u> the final directory known as the "Category/ID" (see below).  As a convenience to the user, this value and the target value are reversed in the restore module from the backup module since you would obviously be doing the reverse of a backup.  This value can, of course, be changed to suit your needs and currently can be a local directory (e.g. /home/someuser) or network share (e.g. \\1.2.3.4\windows\share or //1.2.3.4/linux/share).

* <b>Connect as...</b> These three boxes are the authentication for a network share as the "Source storage path" and consist of the username, password, and optional domain name.  Holding your mouse cursor over each box will show the function of each box.

* <b>Target restore path</b> Specifies the location where your data should be restored, but unlike the "Source storage path", this needs to contain the full path since the "Category/ID" (see below) doesn't pertain to this value.  Currently this value can consist of a local directory (e.g. /home/someuser) or network share (e.g. \\1.2.3.4\windows\share or //1.2.3.4/linux/share).

* <b>Connect as...</b> These three boxes are the authentication for a network share as the "Target restore path" and consist of the username, password, and optional domain name.  Holding your mouse cursor over each box will show the function of each box.

* <b>Category/ID</b> By default this value is provided by the settings of the original backup job and pertains to the final directory that gets appended to the "Source storage path" before restoration of data.  Unless the "Source storage path" is modified, this value can safely be left intact.

* <b>Sync URI & Port</b> Currently this value is disabled, but is intended for inclusion in later versions.

* <b>Remote directory</b> Currently this value is disabled, but is intended for inclusion in later versions.

* <b>Connect as...</b> Currently this value is disabled, but is intended for inclusion in later versions.

* <b>Tar parameters</b> Optional additional parameters to pass to "tar" when performing a restoration. Please note that values entered can <u>break</u> the operation of this software if entered incorrectly - use at your own risk!

* <b>Rsync parameters</b> Currently this value is disabled, but is intended for inclusion in later versions.

* <b>Notes on the job</b> Any entered job notes will be displayed here.


====Data Tab====

* <b>Restore (button)</b> This button performs the same tasks as the one on the "Jobs" tab.

* <b>Directory listing (left-hand pane)</b> This listing corresponds to the files and folders stored under the "Source storage path" from the "Jobs" tab.  You can use this directory listing to add items to the "Included data" and "Excluded data" boxes by simply locating the desired file or folder and dragging it to intended listbox.
NOTE: currently this doesn't work if the "Source backup path" is a network share - we're working on correcting this in future versions.

* <b>Included data</b> By default this screen will have a value of "All data" indicating that all data shown in the "Directory Listing" will be backed up.  You can, of course, remove this or any other entries by highlighting it in the list and clicking the associated "-" button.  For flexibility, the ability to add manual entries, possibly containing optional wildcards, can be entered using the corresponding textbox and clicking the "+" button.

* <b>Excluded data</b> This box contains the same functionality as the "Included data" section, but instead, refers to files or folders that should <u>not</u> be included in the backup.  This can work even if the "Included data" box contains "All data".


====Log Tab====

* <b>Archive (button)</b> This button will allow the user to archive the currently loaded log for manual deletion of their own accord.

NOTE: if you want to load a log in this screen, see the "Overview" tab options above.


====Status Tab====

* <b>Clear Completed (button)</b> Any backup jobs that are started from within this software using the "Run now" button on the "Jobs" tab will show their progress on this screen.  Clicking this button will clear all of the completed jobs from the listing.


====About Tab====

This screen shows information related to the software that would be useful for troubleshooting any experienced problems.  There's also several links on this page to hear from our users and community as well as finding help for the software itself.




==Developers==
Dave Henderson [dhenderson (at) cliquesoft (dot) org]