backup.lite

From Cliquesoft
Jump to: navigation, search

This project allows a user to backup and/or sync their data between multiple locations including local directories, network shares, and remote nodes using SSH with optional email contact upon success and/or failure of job. Not unlike our other bash-script-based software, this project relies on our clAPI framework for various functionality, so be sure this dependency is installed before using. It's also worth mentioning, when using this software from the command prompt, that the placement of the OPTION's must follow their respective ACTION (or parent script) which can be determined via the --help output. It might also help to read over the basics of clAPI to get a better understanding when running software from the command line.


License

This projects' codebase is licensed under the CPLv2 unless a valid CPLv1 license has been purchased. More information about both of these licenses can be found under the "Our Licenses" link of our homepage.



Installation

Installation couldn't be more easy! Currently it's a simple 2-step process which can be accomplished via the following two commands. After you execute the installation script, answer all the questions and your software will be ready for use! 

$ cd /path/to/uncompressed/package
$ ./install



Using the CLI

ACTION's

Among our standard 'help', 'version', and 'update' command line ACTIONs, this project also contains several others that are available - 'backup', 'sync', and 'restore'. It's probably pretty obvious what task each of those handles, so we'll go ahead and move into the OPTIONs section below...


Job Wizard

This OPTION belongs to the parent script of the project since parameters must accompany their respective "module" in order to be processed correctly. Useful for initially creating backup jobs, this parameter initiates a wizard that will walk you through all of the necessary steps. Starting this process is as simple as executing "backup.lite wizard" at the command line.


Backup OPTION's

  • --source specifies the path or network share that needs to be backed up. Currently this parameter can accept a local directory or networked path as its value. Please note that network paths indicate the network type, for instance '\\192.168.0.10\Share' indicates a Microsoft Windows share (SMB/CIFS) whereas '//192.168.0.11/Share' indicates a Linux share (NFS). When using this parameter, be sure to surround it using single quotes!
  • --target can be accept the same values as the --source parameter, but represents the target location where the backup archives should be stored.
  • --targetDir is a directory that is appended to the --target value so that separation or combination of the backup archives generated and maintained by the software can be determined by the backup administrator. For example, lets say that you are backing up two servers called "ServerA" and "ServerB" with each server having incremental backup jobs Monday through Thursday and a full backup on Friday. It might be desired to have all the archives of each backup job relating to the separate severs being grouped in the same directory. So for each backup job relating to each server, you could simply specify the same --target (e.g. /var/backups) and --targetDir values (e.g. ServerA) to have the archives grouped together (e.g. /var/backups/ServerA).
  • --sourceUser provides the username to access the --source in the event that its value is a network share only! If no username and password are required to gain access to the --source network share, then simply do not use this parameter or the following two.
  • --sourcePass provides the usernames corresponding password.

NOTE: later versions will use this to specify a password file for better security.

  • --sourceDom provides the optional domain as part of the credentials when authenticating to a network share.
  • --targetUser same as the --sourceUser parameter but for a --target network share.
  • --targetPass same as the --sourcePass parameter but for a --target network share.

NOTE: later versions will use this to specify a password file for better security.

  • --targetDom same as the --sourceDom parameter but for a --target network share.
  • --compression specifies the type of compression to apply on the backup archive.
  • --count regulates how many backup archives of the configured --type (e.g. daily, incremental, full, etc) are maintained in the --targetDir directory. As an example, if you have only one full backup job being stored in a unique --targetDir and set this value to "4", then only 4 full backup archives will be maintained.
  • --include indicates the relative path and filename (e.g. ./dir/to/file/filename) that contains the specified data to backup. It should have one entry per line and should be relative to the --source value. For flexibility, the user may use wildcards and other allowed syntax via tar's --files-from parameters.
  • --exclude same as the --include parameter, but for data that needs to be excluded during the backup process.
  • --type specifies which type of backup should occur during the backup job. Each type backs up a different set of data which can be found here. If "custom" is used as the value here, the user must specify all of the tar options, with the exception of "-c -f -v --totals", as the value of the --params parameter below.
  • --verify will verify the data integrity within the backup archive against the live data to help ensure errors in the backup process itself.
  • --params allows administrators that require additional tar options the ability to pass them directly to tar with this parameter. Also see the --type parameter above. NOTE: invalid or incorrect values here can break the operation of this software - use at your own risk!


Restore OPTION's

  • --source provides the local directory or network share containing the backup archives to restore data from. For notes on how to enter network shares, please see the "Backup OPTIONS's" above. Also see the --sourceDir parameter.
  • --target indicates the local directory or network share that will receive the restored data. For notes on how to enter network shares, please see the "Backup OPTIONS's" above.
  • --sourceDir defines the directory contained within the --source that's actually containing the backup archives. This is similar to --targetDir in the "Backup OPTION's".
  • --sourceUser provides the username to access the --source in the event that its value is a network share only! If no username and password are required to gain access to the --source network share, then simply do not use this parameter or the following two.
  • --sourcePass provides the usernames corresponding password.

NOTE: later versions will use this to specify a password file for better security.

  • --sourceDom provides the optional domain as part of the credentials when authenticating to a network share.
  • --targetUser same as the --sourceUser parameter but for a --target network share.
  • --targetPass same as the --sourcePass parameter but for a --target network share.

NOTE: later versions will use this to specify a password file for better security.

  • --targetDom same as the --sourceDom parameter but for a --target network share.
  • --archive sets the path and filename of the (first) archive containing the data that the user wants restored. As an added benefit, this value can also contain just the prefix of a filename (e.g. '12345' instead of '12345.d.tgz') and the remaining extension will automatically be determined. Also see the --archives parameter below.
  • --archives specifies how many consecutive archives' data should be restored after the --archive, based on each archives date. If this value isn't defined, it is assumed to restore all data provided by --include from each successive archive until the end is reached. This is useful, for example, when restoring a complete system using the last full backup and its following incremental archives. Also please note, if you wanted to restore from 3 archives total, this value should actually be 2 since --archive always counts as 1.
  • --collapse removes the directory structure when restoring data with duplicate files being overwritten as restored.
  • --data provides an alternative to --include and allows a user to enter a comma separated list, or optionally "*" for "all data", of files to restore from the --archive. This is useful when only a few files need to be restored from an archive and can easily be entered via the command line.
  • --include indicates the relative path and filename (e.g. ./dir/to/file/filename) that contains the specified data to restore. It should have one entry per line and should be relative to the --source value. For flexibility, the user may use wildcards and other allowed syntax via tar's --files-from parameters.
  • --exclude same as the --include parameter, but for data that needs to be excluded during the restore process.
  • --overwrite specifies which existing files should be overwritten during the restore process.
  • --params allows administrators that require additional tar options the ability to pass them directly to tar with this parameter. NOTE: invalid or incorrect values here can break the operation of this software - use at your own risk!


Sync OPTION's

  • --port defines the port to communicate over when working with a remote node.
  • --source is the directory or network share that needs to be duplicated. Unless you're performing a sync only job, or you require an alternative source directory for some reason, this value will inherit the value of the backup modules --source parameter.
  • --target identifies the rsync daemon, host, or local directory needing to be updated. Like the --source parameter, if this value isn't specified, the value of the backup modules --target parameter is assumed.

NOTE: currently rsync daemons aren't supported, but will in future versions.

  • --dir specifies the parent directory in the --target that requires sync'ing. This acts similar to the backup ACTION's --targetDir parameter.
  • --tunnelUser provides the username to establish an SSH tunnel with the remote node. If a set of preshared keys have been exchanged, then simply do not use this parameter or the following one.
  • --tunnelPass provides the usernames corresponding password.
  • --params allows administrators that require additional rysnc options the ability to pass them directly to rsync with this parameter. NOTE: invalid or incorrect values here can break the operation of this software - use at your own risk!



Examples

Backing up data only (no sync'ing)... 

$ sudo ./backup.lite --noprompts --name=test backup --type=full --source='/tmp/temp' --target='/tmp/backup' --compression=gzip --include='/etc/backup/test.inc'

Beginning the "test" backup job @ Tue Mar 27 10:37:44 EDT 2012

Checking system environment...

  (i)   Directories...
           Configs: [checking] [exists] [writable] [success] [done]
           Temp: [checking] [exists] [writable] [success] [done]
           Mounts: [checking] [exists] [writable] [success] [done]
  (i)   Variables...
           --name: [done]
           ACTION: [done]

Beginning the 'backup' module...

  (i)   Processing the source and target...
           Source: [symlinking] [success] [done]
           Target: [local] [checking] [exists] [writable] [success] [symlinking] [success] [done]
           Storage: [checking] [exists] [done]
  (i)   Checking variables: [type] [include] [done]
  (i)   Processing a(n) "full" backup  -----------------------------[PROGRESS]--

./
./original/
./original/flash.img
./a/
./a/b/
./a/b/test.txt
./a/b/test.sh
./flash.tar.gz
Total bytes written: 656384000 (626MiB, 31MiB/s)

--------------------------------------------------------------------------------
  (i)   Creating a catalog of backed up data: [done]
  (i)   Cycling the backups...
           Archive: [deleting] [success] [done]
           Catalog: [absent] [done]
           Job Log: [absent] [done]
  (i)   Check archive count: [within limits] [done]
  (i)   Compressing the archive: [gzip]
--------------------------------------------------------------------[PROGRESS]--
 626MB 0:01:05 [9.56MB/s] [==================================>] 100%            
--------------------------------------------------------------------------------
           [deleting] [success] [done]

Calling exit routines for the modules...

  (i)   backup.lite script...
           Source: [checking] [unmounted] [deleting] [success] [done]
           Target: [checking] [unmounted] [deleting] [success] [done]
           Cleanup: [deleting] [success] [done]

The job has completed successfully @ Tue Mar 27 10:39:16 EDT 2012



Using the Web Interface

NOTE: this section applies for local and remote graphical user interfaces.

This section will discuss the options available to the user when interacting with various modules within the graphical interface, but before we get started, we'll need to explain how to make it available in the first place. Before you get discouraged, this process is nowhere near as complex as it may sound and can be accomplished with only one command, thanks to the clAPI framework! We'll now need to setup and/or start a web server that will enable the use of the GUI. If either of the following conditions are met, you can skip to the final paragraph of this section:

  • If you're using a "XiniX Compatible" device
  • If you installed this software using some type of graphical installer


  • Typical Users The easiest way to accomplish this goal is by starting clAPI's built-in web server using a single command entered at the prompt within a terminal window:

Type the following: 

$ clapi --noprompts --verbose shttpd

To produce the following: 

Checking system environment...

  (i)   Directories...
           Conf: [checking] [exists] [writable] [success] [done]
           Logs: [checking] [exists] [writable] [success] [done]
           Temp: [checking] [exists] [writable] [success] [done]
  (i)   User Interface: [console] [done]

Beginning the 'shttpd' module...

  (i)   Initializing the web server on 127.0.0.1 over 8080: [serving]
--------------------------------------------------------------------[PROGRESS]--
Sat Jul  7 20:15:06 UTC 2012    INFO    Changing into the serving directory
Sat Jul  7 20:15:06 UTC 2012    INFO    Creating environment...
Sat Jul  7 20:15:06 UTC 2012    INFO    Starting UI dependencies...
Sat Jul  7 20:15:06 UTC 2012    INFO    Starting web server...
  • Administrators For the admins that already have a web server installed such as paged, Apache, IIS, nginx, lighttpd, boa, or any other, you can simply point it to the directory that the 'GUI files' were installed to during the clAPI and backup.lite installation. Afterwards, restart your web server and you're good to go!


If you've made it this far, then you now have access the graphical interface of the software using any web browser. For users working with this software on a local device (e.g. personal computer, tablet, smart phone, etc), simply navigate to "http://127.0.0.1:8080", alternatively "http://localhost:8080", and after providing the authentication credentials for a valid system user account, you can interact with all of the installed modules. Users accessing this on a remote device (e.g. Backup server, NAS, etc) will need to use "http://x.x.x.x:8080" where "x.x.x.x" represents the IP address of the device. If this doesn't work for you, contact your network administrator or technical support company.


Backup Module

Overview Tab

  • Archive Issues (button) This button will archive all the information that's presented in the "Issues that need Attention" section of the screen. This can be useful if that information will ever be required in the future.
  • Scheduled Jobs As the name suggests, this box shows all the upcoming scheduled tasks along with their scheduled job number and planned execution time. There are no interactions possible in this area as all the content presented here is information only.
  • Completed Jobs This area lists all of corresponding logs of each backup job that has completed succesfully. [NOTE: Please be aware that since these logs correspond to the currently stored backup archives, as they rotate, so do the logs!] This box does present the ability for the user to view any of those logs by simply clicking the log ID and then clicking the "Log" tab.
  • Issues that need Attention In order to minimize the work needed to resolve encountered problems, this section was provided to show you the recorded errors along with one-click access to the logs (see above) for details.


Jobs Tab

  • New (button) Used to create a new, blank backup job. After entering a unique name, the "Saved jobs" list is updated and you can begin to fill in the details for the job.
  • Rename (button) Allows you to rename an existing backup job. Note, renaming a backup job does NOT affect it's existing scheduled status.
  • Copy (button) This button creates a duplicate of the job currently selected in the "Saved jobs" list using the name entered. It's important to note that when jobs are copied, so are their scheduled settings.
  • Delete (button) Deletes the currently selected "Saved job".
  • Save (button) Saves the updated configuration for the selected "Saved job" based on the values on this page.
  • Saved jobs Lists all the current backup jobs, active or inactive, that are currently being maintained by the software.
  • Type & Compression Allows the user to specify the backup type and compression for the backup job. The backup types are as follows: Daily backs up all data that was created during the day; incremental backs up all the data since the last backup; full backups up all the data without restrictions; custom allows an administrator to fully define the tar syntax excluding "tar -cv --totals -f"; sync only prevents any type of backup, but simply sync's two locations whether they're local directories or remote nodes.
  • Start date & time These values allow you to schedule when a backup job will need to run. Clicking on the date box will present you with a calendar to select the exact day the job should execute. The time can be in 12 or 24 hour formats, however, the former will need to have an 'am' or 'pm' added (e.g. 9:00am).
  • Frequency If you would like to have a backup job run on at a reoccuring scheduled time, simply fill out the above two values along with this one. Values here should be the frequency between each backup (e.g. 4 hours, 7 days, etc).
  • Data storage Provides several options to maintain a collection a backup archives. The first option allows you to specify the number of archives that should be retained after the backup job completes. The second option allows you to keep however many archives that can fit within a specified amount of disk space. The final option allows administrators to manually manage their own backup archives.

NOTE: the second option is currently disabled, but will be available in future versions.

  • Miscellaneous Several miscellaneous options dealing with the backup. Verifying the data will ensure that the archives didn't have any issues with the data they are backing up. The other two options enable an administrator additional flexibility when creating a backup or sync'ing by passing additional options directly to the underlying binaries (tar and rsync). Please note that values entered can break the operation of this software if entered incorrectly - use at your own risk!
  • Contact name The contact name of the person that should receive any success or failure messages generated by the software.
  • Email address The corresponding value to the "Contact name". A value here enables automated emailing.
  • Job status Allows the user to enable or disable a backup job. This also will remove any job scheduling if currently configured.
  • Run now (button) Clicking this button will immediately start the selected backup job with its current progress being shown on the "Status" tab.
  • Source backup path This defines the source location that needs to be backed up. Currently this value can consist of a local directory (e.g. /home/someuser) or network share (e.g. \\1.2.3.4\windows\share or //1.2.3.4/linux/share).
  • Connect as... These three boxes are the authentication for a network share as the "Source backup path" and consist of the username, password, and optional domain name. Holding your mouse cursor over each box will show the function of each box.
  • Target storage path Specifies the location where your backup archives should be stored. Currently this value can consist of a local directory (e.g. /home/someuser) or network share (e.g. \\1.2.3.4\windows\share or //1.2.3.4/linux/share).
  • Connect as... These three boxes are the authentication for a network share as the "Source backup path" and consist of the username, password, and optional domain name. Holding your mouse cursor over each box will show the function of each box.
  • Category/ID Specifies a directory that is appended to the "Target storage path" so that separation or combination of the backup archives generated and maintained by the software can be determined by the backup administrator. For example, lets say that you are backing up two servers called "ServerA" and "ServerB" with each server having incremental backup jobs Monday through Thursday and a full backup on Friday. It might be desired to have all the archives of each backup job relating to the separate severs being grouped in the same directory. So for each backup job relating to each server, you could simply specify the same --target (e.g. /var/backups) and --targetDir values (e.g. ServerA) to have the archives grouped together (e.g. /var/backups/ServerA).
  • Sync URI & Port The rsync daemon, host, or local directory needing to be updated along with the port to communicate over when working with a remote node.

NOTE: currently rsync daemons aren't supported, but will in future versions.

  • Remote directory Specifies the parent directory at the "Sync URI" that requires sync'ing. This is similar to the "Category/ID" parameter.
  • Connect as... These fields specify the username and password file used to connect with a remote node.



Data Tab

  • Save (button) Once you've got the "Include data" and "Exclude data" lists filled out according to the backup job, click this button to save them to disk.
  • Directory listing (left-hand pane) This listing corresponds to the files and folders stored under the "Source backup path" from the "Jobs" tab. You can use this directory listing to add items to the "Included data" and "Excluded data" boxes by simply locating the desired file or folder and dragging it to intended listbox.

NOTE: currently this doesn't work if the "Source backup path" is a network share - we're working on correcting this in future versions.

  • Included data By default this screen will have a value of "All data" indicating that all data shown in the "Directory Listing" will be backed up. You can, of course, remove this or any other entries by highlighting it in the list and clicking the associated "-" button. For flexibility, the ability to add manual entries, possibly containing optional wildcards, can be entered using the corresponding textbox and clicking the "+" button.
  • Excluded data This box contains the same functionality as the "Included data" section, but instead, refers to files or folders that should not be included in the backup. This can work even if the "Included data" box contains "All data".


Log Tab

  • Archive (button) This button will allow the user to archive the currently loaded log for manual deletion of their own accord.

NOTE: if you want to load a log in this screen, see the "Overview" tab options above.


Status Tab

  • Clear Completed (button) Any backup jobs that are started from within this software using the "Run now" button on the "Jobs" tab will show their progress on this screen. Clicking this button will clear all of the completed jobs from the listing.


About Tab

This screen shows information related to the software that would be useful for troubleshooting any experienced problems. There's also several links on this page to hear from our users and community as well as finding help for the software itself.


Restore Module

Overview Tab

  • Archive Issues (button) This button will archive all the information that's presented in the "Issues that need Attention" section of the screen. This can be useful if that information will ever be required in the future.
  • Completed Jobs This area lists all of corresponding logs of each restore job that has completed succesfully. This box does present the ability for the user to view any of those logs by simply clicking the log ID and then clicking the "Log" tab.
  • Issues that need Attention In order to minimize the work needed to resolve encountered problems, this section was provided to show you the recorded errors along with one-click access to the logs (see above) for details.


Jobs Tab

  • Restore (button) Starts a restoration of designated data (from the "Data" tab) based on the currently viewed "Saved jobs" value using the configuration of the parameters accessible from within this screen.
  • Saved jobs Lists all the current backup jobs, active or inactive, providing access to their archives' contained data for restoration. During the initial loading of this module or upon changing this value, the "Associated Archives" change dynamically to match all corresponding archives relating to this value.
  • Restore options These two options control what data gets restored from the archive with the ability to retain or remove the directory structure containing your desired data for restoration. So, for example, if you wanted to restore a file called "abc.txt" contained within the directory "Documents/personal/abc.txt" to "/tmp/restore", and you removed the directory structure, instead of finding the restored document under "/tmp/restore/Documents/personal/abc.txt", it would be found under "/tmp/restore/abc.txt".
  • Miscellaneous These two options were included for flexibility, but are NOT required for a successful restoration. Incorporating these options will allow additional parameters to be passed directly to the underlying binaries (tar and rsync) during the restoration process. They should only include additional parameters, not full syntax of the related command. Please note that using them can break the functionality of this software if you don't know what you're doing!
  • Associated archives Here you'll find a list of all the current backup archives associated with the selected "Saved job". Clicking on any in the list will load its contents in the "Data" tab so you can select what data you would like to restore. There's also a complementing value when restoring entitled "Include X subsequent archives". Its intended purpose is to speed up the restoration of selected data among various backup archives so the user doesn't have to restore from each one individually. For example, lets say that you have a schedule of incremental backups Monday through Thursday and full backups on Friday. Wednesday your server crashes and you need to perform a full restoration of data. You can simply load your last full backup (from Friday) and enter in '2' for the subsequent archive count (for Monday and Tuesdays backups) and all the selected data from Fridays archive will be restored as well as any matching data in Monday and Tuesdays backups, saving you the hassle of completing those steps yourself. It's important to note that the subsequent archives that get processed are date oriented, not by the job!
  • Source storage path This defines the source location where the backup archives are currently stored without the final directory known as the "Category/ID" (see below). As a convenience to the user, this value and the target value are reversed in the restore module from the backup module since you would obviously be doing the reverse of a backup. This value can, of course, be changed to suit your needs and currently can be a local directory (e.g. /home/someuser) or network share (e.g. \\1.2.3.4\windows\share or //1.2.3.4/linux/share).
  • Connect as... These three boxes are the authentication for a network share as the "Source storage path" and consist of the username, password, and optional domain name. Holding your mouse cursor over each box will show the function of each box.
  • Target restore path Specifies the location where your data should be restored, but unlike the "Source storage path", this needs to contain the full path since the "Category/ID" (see below) doesn't pertain to this value. Currently this value can consist of a local directory (e.g. /home/someuser) or network share (e.g. \\1.2.3.4\windows\share or //1.2.3.4/linux/share).
  • Connect as... These three boxes are the authentication for a network share as the "Target restore path" and consist of the username, password, and optional domain name. Holding your mouse cursor over each box will show the function of each box.
  • Category/ID By default this value is provided by the settings of the original backup job and pertains to the final directory that gets appended to the "Source storage path" before restoration of data. Unless the "Source storage path" is modified, this value can safely be left intact.
  • Sync URI & Port Currently this value is disabled, but is intended for inclusion in later versions.
  • Remote directory Currently this value is disabled, but is intended for inclusion in later versions.
  • Connect as... Currently this value is disabled, but is intended for inclusion in later versions.
  • Tar parameters Optional additional parameters to pass to "tar" when performing a restoration. Please note that values entered can break the operation of this software if entered incorrectly - use at your own risk!
  • Rsync parameters Currently this value is disabled, but is intended for inclusion in later versions.
  • Notes on the job Any entered job notes will be displayed here.


Data Tab

  • Restore (button) This button performs the same tasks as the one on the "Jobs" tab.
  • Directory listing (left-hand pane) This listing corresponds to the files and folders stored under the "Source storage path" from the "Jobs" tab. You can use this directory listing to add items to the "Included data" and "Excluded data" boxes by simply locating the desired file or folder and dragging it to intended listbox.

NOTE: currently this doesn't work if the "Source backup path" is a network share - we're working on correcting this in future versions.

  • Included data By default this screen will have a value of "All data" indicating that all data shown in the "Directory Listing" will be backed up. You can, of course, remove this or any other entries by highlighting it in the list and clicking the associated "-" button. For flexibility, the ability to add manual entries, possibly containing optional wildcards, can be entered using the corresponding textbox and clicking the "+" button.
  • Excluded data This box contains the same functionality as the "Included data" section, but instead, refers to files or folders that should not be included in the backup. This can work even if the "Included data" box contains "All data".


Log Tab

  • Archive (button) This button will allow the user to archive the currently loaded log for manual deletion of their own accord.

NOTE: if you want to load a log in this screen, see the "Overview" tab options above.


Status Tab

  • Clear Completed (button) Any backup jobs that are started from within this software using the "Run now" button on the "Jobs" tab will show their progress on this screen. Clicking this button will clear all of the completed jobs from the listing.


About Tab

This screen shows information related to the software that would be useful for troubleshooting any experienced problems. There's also several links on this page to hear from our users and community as well as finding help for the software itself.



Developers

Dave Henderson [dhenderson (at) cliquesoft (dot) org]

Retrieved from "https://wiki.cliquesoft.org/index.php?title=backup.lite&oldid=5269"