Copy thousands of files with rsync

Copying a lot of files across a network using native OS features can be slow due to the way the OS opens and closes files. When there are lots of small files, the overhead adds up.

If you have SSH enabled on both machines, you can use rsync to copy a directory from a remote machine to your local machine with compression which will speed up the process.

For example, to copy the directory ~/sound/wav from a remote machine to the ~/sound/wav location on your local machine, execute this command:

rsync -avz -e ssh user@192.168.1.2:~/sound/wav ~/sound/

Notice that you don’t specify the name of the target directory. It’ll create it for you.

Here’s what the options do:

• -a: Archive mode, which preserves permissions and timestamps.
• -v: Verbose mode, which gives you more information about the transfer so you can monitor it.
• -z: Enables compression during the transfer.
• -e ssh: Specifies to use SSH for the transfer. You don’t have to specify this as it’s the default, but by specifiying it you make it more clear.
• user@192.168.1.2:~/sound/wav: The source directory on the remote machine. Replace user with your username on the remote machine.
• ~/sound/: The destination directory on your local machine. rsync will create the wav directory inside this path.

To copy files in the reverse direction, change the source and target machines.

This method will work well for transferring large numbers of small files across your network. If you are transferring files that are compressed already, you won’t save as much time.