[kwlug-disc] Transferring large backup files over the network

[Paul Nijjar]

On Thu, May 23, 2019 at 04:19:20PM -0400, bob+kwlug at softscape.ca wrote:

> > Say I have a big (multi-terabyte) collection of backup files (which
> > tend to be large, in the tens of gigabytes) sitting on some server A.
> 
> Are the files themselves large or is it just a large collection? 
> 
> If they are large files, are they fresh each day or are they just partially modified each day?


They are large files, generated by an awful backup application I will
not mention here, for fear of polluting the list. There are many of
them, but only a few of them change each day. 

> > 
> > What FLOSS tools would be best for this use case and why?
> > If you know of good solutions that would play with NTFS, that is a
> > bonus. (rsync is not in that category, especially if Server A has to
> > run Windows.) But even if we soften the NTFS requirement, I am not
> > sure whether rsync is still the best solution here.
> 
> I have discovered program for Windows called MobaXterm.
> 
> It seems to bring a great deal of functionality to windows that I am accustomed to on a Linux CLI. (eg: Xwindow server, ssh client and server, rsync, cron!, ...)
> 
> I'm not sure if it is as free as you want/need it to be, but I think it has the base functionality to enable you to be able to rsync in the way you described.
> 
> Rsync is a great tool. Not sure if there are any nuances of your setup that would make it sub-optimal, but it's likely a good place to start.
> 
> And MobaXterm seems to have all the tools you'd need.
> 

Hrm. That brings to mind the (non-FLOSS) Windows Subsystem for Linux.
The problem with many of these "compile Linux tools on Windows" is
that they are often limited to 32-bit (although I am not sure whether
MobaXterm is). That actually has caused problems for me in my backups
when I have tried to use various Rsync for Windows tools in the past.

But if I install Linux in Windows using WSL, then the 64 bit versions
of these tools might be available, and then maybe NTFS is not going to
be the bottleneck any more. 


- Paul

-- 
Sign Up for the WatCamp Newsletter: https://da.gd/wtcmp-news
(Also http://pnijjar.freeshell.org still exists)