copy if changed
January 6, 2012
I had written a bunch of versions of something like this for scripts over the years. Had to do it again, yesterday, and decided I’d do it ‘properly’ this time.
http://skvidal.fedorapeople.org/misc/copy_if_changed.py
It downloads from a url and copies to whatever you specify.
Summary:
copy_if_changed – copy file from a remote url to local destination. Return 0
if file has changed, 1 if it hasn’t and 2 if an error occurs
usage is: copy_if_changed url /some/local/destination
Really handy in shell scripts:
if copy_if_changed http://server/some/file /etc/some/file; then
run_something_here
fi
Other features:
- it makes a unique (timestamped) backup of the dest file before it copies it over.
- it uses sha256 for the checksumming
Not sure if it is useful to anyone else but I’m tired of rewriting the same thing.
January 6, 2012 at 10:17 pm
It is very useful, indeed!
For instance, I will use it for updating local copies of Geonames data files (http://download.geonames.org/export/dump/).
January 6, 2012 at 11:31 pm
you should be checking through the http protocol the last-modified header and comparing it to the if-modified-since header. You would be saving yourself and the webserver banwidth.
There is a perfectly fine Perl script that does just this: lwp-mirror (you can get it if you install perl-libwww-perl in a rhel 6 system.
January 7, 2012 at 1:22 am
A couple of issues:
1. http servers often lie.
2. the url doesn’t have to be http, supports https, ftp, file:// too
3. It might not be a file on an fs at all but content sucked from a cgi.
I’ll pass on getting fancy with the http protocol if only b/c I’ve had so much experience with servers being full of crap.
January 7, 2012 at 10:08 am
I will admit point 3 as the only reason why you would want to do this. However, in my 15 years working as a sysadmin I have yet not come accross this situation.
The lwp library supports http, https, gopher, ftp, news, file, and mailto resources (from their documentation page), so point 2 does not apply.
As to point 1, well, that is your personal opinion and you probably have written your script for a reason
. I have used the lwp-mirror script often enough to know it works well for exactly the same purpose you have written yours. I have not had any problems with any webservers yet while using it. Maybe just lucky.
I just wanted to point there is a perfectly fine solution out there for this kind of problem so one does not have to invent their own wheel.
Thanks for sharing it!
January 8, 2012 at 3:54 am
The program is useful and all, but I have a pet peeve
To compare two files, don’t compute and compare hashes, just compare the file contents.
The program has to read all of the data anyway, so computing the hash only adds CPU load and adds the risk of breakage caused by a hash collision.