Today I needed to transfer contents between my Linux servers using a CAT5 LAN connection. The source server served files via HTTP and also had a FTP server running. I ruled out using FTP for transfer as I wanted to do the transfer via command line recursively so the natural candidate was wget. On the target Linux box I executed the following command to get a good transfer speed:
wget -r -nH –cut-dirs=2 –no-parent –reject=”index.html*” http://192.168.111.116/files/dump/
The speed received was something like this:
Downloaded: 2476 files, 27G in 45m 49s (10.0 MB/s)
This completed the job in a a short while with no residual HTML file being generated in target.