[NBLUG/talk] Synchronizing strategies

Omar Eljumaily omar at omnicode.com
Sat Nov 22 11:50:55 PST 2014


I think this is what I'm looking for.

http://manpages.ubuntu.com/manpages/trusty/man7/inotify.7.html

Thanks Zack and Jordan.  I think some combination of rsync, srcipts, and 
C++ code is what I'm going to use.  I mainly wanted to know if there was 
something like "just install this and it will solve your problem," which 
there seems not to be.

I guess the data size is not so much an issue as the number of files I'm 
talking about synching which I think is about 3 or 4 million files.  I 
know that Hostgator limits the number of files for inexpensive accounts 
to about 200k, although they claim unlimited data size (or used to).

Thanks,

Omar


On 11/22/2014 11:18 AM, Omar Eljumaily wrote:
> I think it would be cheaper to just get directory/file mod listings, 
> which I'm already paranoid would create undo overhead.  I know that 
> Samba plugins can tell you when files have been modified without 
> searching.  I'm wondering if there's such a thing for Linux file systems.
>
> Thanks,
>
> Omar
>
>
> On 11/22/2014 11:14 AM, Zack Gold wrote:
>>
>> You could write a clever script that stores md5sums of every file and 
>> check I'd it's changed.
>>
>> On Nov 22, 2014 2:13 PM, "Omar Eljumaily" <omar at omnicode.com 
>> <mailto:omar at omnicode.com>> wrote:
>>
>>     I'd like to have the knowledge of when a file changes.  I don't
>>     know how to get those notifications in linux.  Does anybody know?
>>
>>     Also, not all files get closed to the point where they can be
>>     copied every time they're modified.  Windows clients are
>>     notorious for keeping files open all day long without letting
>>     other processes copy them.
>>
>>     The thing with the could side is that I don't want to be the
>>     person responsible for running the cloud server, so I'm going to
>>     have to work within the cloud provider's framework.  The ones I'm
>>     looking at right now are Google and 1and1.com <http://1and1.com>
>>     (which seems to be pretty cheap, about $10/month for 1 terrabyte).
>>
>>     Thanks,
>>
>>     Omar
>>
>>
>>     On 11/22/2014 10:57 AM, Jordan Erickson wrote:
>>
>>         I use rsync+ssh+rsnapshot+cron for a (albeit daily) backup of
>>         multiple
>>         VMs in "the cloud" (my cloud) totaling about 1.5TB total to a
>>         local
>>         disk, then another rsnapshot to an external disk about every
>>         month. It's
>>         fast. rsync is designed to take large amounts of data and
>>         sync them
>>         efficiently.
>>
>>         Is there a requirement for "every hour" (and not something
>>         more frequent
>>         like, every time a file changes)? If not, maybe a trigger
>>         that whenever
>>         a file changes, it's sync'd... I'm sure multiple avenues are
>>         possible
>>         with some simple script-fu (and probably plenty of
>>         ready-to-go packages
>>         as well).
>>
>>
>>         Jordan Erickson (PGP: 0x78DD41CB)
>>         LNS: 707-636-5678 <tel:707-636-5678>,
>>         http://logicalnetworking.net
>>
>>
>>         On 11/22/2014 10:50 AM, steve wrote:
>>
>>             On 11/22/2014 10:43 AM, Zack Gold wrote:
>>
>>                 rsync + cron?
>>
>>                 On Nov 22, 2014 1:36 PM, "Omar Eljumaily"
>>                 <omar at omnicode.com <mailto:omar at omnicode.com>
>>                 <mailto:omar at omnicode.com
>>                 <mailto:omar at omnicode.com>>> wrote:
>>
>>                      Does anybody know about file synchronization
>>                 strategies, mainly for
>>                      synching from a local file server to a cloud?
>>                 What I'm trying to
>>                      accomplish is:
>>
>>             For a terebyte???
>>             _______________________________________________
>>             talk mailing list
>>             talk at nblug.org <mailto:talk at nblug.org>
>>             http://nblug.org/cgi-bin/mailman/listinfo/talk
>>
>>         _______________________________________________
>>         talk mailing list
>>         talk at nblug.org <mailto:talk at nblug.org>
>>         http://nblug.org/cgi-bin/mailman/listinfo/talk
>>
>>
>>     _______________________________________________
>>     talk mailing list
>>     talk at nblug.org <mailto:talk at nblug.org>
>>     http://nblug.org/cgi-bin/mailman/listinfo/talk
>>
>>
>>
>> _______________________________________________
>> talk mailing list
>> talk at nblug.org
>> http://nblug.org/cgi-bin/mailman/listinfo/talk
>
>
>
> _______________________________________________
> talk mailing list
> talk at nblug.org
> http://nblug.org/cgi-bin/mailman/listinfo/talk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://nblug.org/pipermail/talk/attachments/20141122/f0eebd92/attachment.html>


More information about the talk mailing list