rsync is a file copying utility for Linux which is similar to
scp (secure copy), but with more functionality. It keeps a file table of all transferred files, and only transfers those that have been modified (hence the sync bit).
rsync can copy/sync locally or across a network.
It has the following syntax:
$ rsync [options] source destination
The most popular options are (sorted alphabetically by short option):
|Short Option||Long Option||Description|
|Archive mode. This makes rsync also sync user and group settings for files and directories. Archive mode can be both extremely useful and extremely unhelpful. Archive mode will not work correctly if the source and destination systems do not have the same users and groups.|
|Do a trial run which doesn't actually make any changes. This is usually used in conjunction with -v to make sure you are doing it correctly before make any modifications.|
|Prints the progress to |
|Recursive, will sync all files in child directories also.|
|Preserves the modification times.|
|Verbose. Will prints out more information when process is run. You can add extra v's (e.g. |
|Compress data while doing transfer. Some files cannot be compressed, which includes |
Some of the most popular combinations of options are:
|rsync -arvz source destination||Good general purpose rsync use, without propagating deletions.|
|rsync -adrvz source destination||Good general purpose rsync use with deletion propagation (be careful!)|
rsync is a great tool for allowing you write code on a fully-fledged computer, and then transfer it to a RaspberryPi easily and quickly for running. Normally it only takes a matter to seconds to transfer after you have made modest code changes.
An interesting side-note is that you can replicate the capabilities of the Mac OS time machine with rsync using the following commands (taken from http://en.wikipedia.org/wiki/Rsync):
#date=`date "+%Y-%m-%dT%H:%M:%S"` date=`date "+%FT%T"` rsync -aP --link-dest=$HOME/Backups/current /path/to/important_files $HOME/Backups/back-$date ln -nfs $HOME/Backups/back-$date $HOME/Backups/current
Another interesting side-note for those interested is that at the core of the compare algorithm is a MD5 checksum and a rolling checksum. This speeds up the checking, rather than comparing the files bit-by-bit. However, this can lead to errors. The probability of rsync believing two files are in sync, but actually are not, requires both a collision (a collision is when different inputs product the same output) in the MD5 and rolling checksum, which sits around the 2^-160 mark (very unlikely).
How To Incrementally Copy A Large File From Server To Local Folder
If you want to copy a large file, you may want to do it incrementally because of disconnection issues, or to resume copying at a different time. rsync supports this with the
$ rsync username@ip_address:/directory_on_server/file_on_server.big --progress --append
--progress is useful as it continuously prints the percentage progress of the operation.
Including And Excluding Files
rsync supports a glob-like syntax for matching against files/directories you wish to include/exclude.
You can use the
--exclude-from flag to pass in a file containing a list of files/directories to exclude.
* matches any partial substring of a single directory.
** matches any substring of the path.
If you want to match against a
.git directory at the root level of your directory tree:
$ rsync --exclude=".git"
If you want to match against
__cache__ directories that could be at any level in your directory tree:
$ rsync --exclude="**/__cache__"
*** can be used with
--include so that when matching against directories, all sub-directoris and files below it will also be included.
An alternative to using
rsync’s include/exclude rules is to combine
rsync with the popular file-finding utility
Combining rsync With find
find source_dir -iname '*.jpg' -print0 | rsync -0 -v --files-from=- . destination_dir/