copy if changed

January 6, 2012

I had written a bunch of versions of something like this for scripts over the years. Had to do it again, yesterday, and decided I’d do it ‘properly’ this time.

http://skvidal.fedorapeople.org/misc/copy_if_changed.py

It downloads from a url and copies to whatever you specify.

Summary:

copy_if_changed – copy file from a remote url to local destination. Return 0
if file has changed, 1 if it hasn’t and 2 if an error occurs

usage is: copy_if_changed url /some/local/destination

Really handy in shell scripts:


if copy_if_changed http://server/some/file /etc/some/file; then
run_something_here
fi

Other features:
– it makes a unique (timestamped) backup of the dest file before it copies it over.
– it uses sha256 for the checksumming

Not sure if it is useful to anyone else but I’m tired of rewriting the same thing.

About these ads

5 Responses to “copy if changed”

  1. denisarnaud Says:

    It is very useful, indeed!
    For instance, I will use it for updating local copies of Geonames data files (http://download.geonames.org/export/dump/).

  2. oxtan Says:

    you should be checking through the http protocol the last-modified header and comparing it to the if-modified-since header. You would be saving yourself and the webserver banwidth.

    There is a perfectly fine Perl script that does just this: lwp-mirror (you can get it if you install perl-libwww-perl in a rhel 6 system.

  3. skvidal Says:

    A couple of issues:
    1. http servers often lie.
    2. the url doesn’t have to be http, supports https, ftp, file:// too
    3. It might not be a file on an fs at all but content sucked from a cgi.

    I’ll pass on getting fancy with the http protocol if only b/c I’ve had so much experience with servers being full of crap. :)

  4. oxtan Says:

    I will admit point 3 as the only reason why you would want to do this. However, in my 15 years working as a sysadmin I have yet not come accross this situation.

    The lwp library supports http, https, gopher, ftp, news, file, and mailto resources (from their documentation page), so point 2 does not apply.

    As to point 1, well, that is your personal opinion and you probably have written your script for a reason ;-). I have used the lwp-mirror script often enough to know it works well for exactly the same purpose you have written yours. I have not had any problems with any webservers yet while using it. Maybe just lucky.

    I just wanted to point there is a perfectly fine solution out there for this kind of problem so one does not have to invent their own wheel.

    Thanks for sharing it!

  5. Miloslav Trmač Says:

    The program is useful and all, but I have a pet peeve :)

    To compare two files, don’t compute and compare hashes, just compare the file contents.

    The program has to read all of the data anyway, so computing the hash only adds CPU load and adds the risk of breakage caused by a hash collision.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: