I need to create a cron job to mirror my websites but I'm clueless as to how to do macros (or whatever they are called).


This is what I think I need:

1. tar -cpzf domainname.tar.gz *
2. Open ssh to Mirrored Site
3. wget domainIP/domainname.tar.gz
4. wait xxx amount of time for xfer to finish or just execute the next command with the prompted?
5. tar -xzf domainname.tar.gz
6. Close ssh session


In case I got the above all wrong and you can't tell what I'm trying to do, let me go line by line.

1. Create a ZIP of my site.
2. Secure Telnet to my mirrored site.
3. Transfer the ZIP file.
4. Not sure if this is needed - a wait long enough to let the xfer finish before the next command runs.
5. unZIP the ZIP
6. Close the Secure Telnet session.

I'd like this to run 2 times a day (for web page changes - not that they make any) and I'll have another job that just updates the email and accounts every hour.

Your help is greatly appreciated!

Tron Of Borg

When you put that all in your script, you don't need to worry about the time delay. The next command won't execute until the first one finishes unless you put the first one in to the background.

But instead of running an ssh session, use scp instead. It's much better suited. Otherwise, your logic is sound. Did you want someone to write this script for you?

I would be very thankful if someone would do this.

My hosting company has a link for me that creates the cron job. I think I just need to upload the script to my site and then tell this http page when to run the script and where the script is.

As for the scp vs ssh, I'll have to check to see if my host has scp installed. Yep! I SSHed in and ran scp and got the usage info for scp.

BTW, the prompt (user interface/shell/whatever its called) is bash-2.05 if that makes any difference.

Thanks again,
Tron Of Borg

Hi,
What's it worth to you? :P

I'll write you one... I think that a Cisco Aironet 350 card would be fair trade :=) :P.

--Bovine

I really like the line "BTW the shell/prompt whatever is bash2." Hehehe, good stuff.

Here, try this:

First, do all of this:
ssh implements the RSA authentication protocol automatically. The user
creates his/her RSA key pair by running ssh-keygen(1). This stores the
private key in $HOME/.ssh/identity and the public key in
$HOME/.ssh/identity.pub in the user's home directory. The user should
then copy the identity.pub to $HOME/.ssh/authorized_keys in his/her home
directory on the remote machine (the authorized_keys file corresponds to
the conventional $HOME/.rhosts file, and has one key per line, though the
lines can be very long). After this, the user can log in without giving
the password. RSA authentication is much more secure than rhosts authen-
tication.

You'll also have to use ssh-agent to add that key and it's password. "man ssh-agent" to get more info on that. Though a simple `ssh-agent` when the system boots should suffice.


Then use the following script. you need to do the above so that ssh will automagically login otherwise your script will hang waiting for a password.

#begin transfer script
#written by shawn on 3/9/02 to avoid art history homework
#for BACC-Core

domain = ;
remote_user = ;
remote_host = ;
#local_user = ;
#local_host = ;

tar -cpzf $domain.tar.gz *;
scp $domain.tar.gz $remote_user@$remote_host:$domain.tar.gz;

ssh $remote_user@$remote_host;
tar -zxf $domain.tar.gz;
exit;
#end script

That should work. If it doesn't lmk.

Can I review to make sure I 'get' it?
Line:
5 - 9: Are these variables? Do I just type in the values here? With quotes or no quotes around text?
8-9: Do I leave these remarked out (and blank) since the script is run from the system that would reference these?
11: does this line create the secure telnet AND transfer the file?
12: is this line needed if line 11 is used? Looks like it just creates another secure telnet connection.
13: This decompresses the compressed file. If line 11 doesn't transfer the file an line 12 just creates the telnet connection, don't I need to transfer the file before this line?
14: terminates the script
15: tells us WINDOZE users that this is the last line of the script! LOL


Once again,
Thank you!!!

Tron Of Borg

--------------SCRIPT-----------------
1. #begin transfer script
2. #written by shawn on 3/9/02 to avoid art history 3. homework
4. #for BACC-Core

5. domain = ;
6. remote_user = ;
7. remote_host = ;
8. #local_user = ;
9. #local_host = ;

10. tar -cpzf $domain.tar.gz *;
11. scp $domain.tar.gz $remote_user@$remote_host:$domain.tar.gz;

12. ssh $remote_user@$remote_host;
13. tar -zxf $domain.tar.gz;
14. exit;
15. #end script

Hi,
If your using RSA keys then that implies that your are using SSHv1. There are known weaknesses with this version of the protocol. e.g. It's possible to perform a MiM attack on an SSHv1 connection and then there's the CRC32 compensation attack (though this affects the SSH Daemon and not the user directly I suppose).

Most people (sane people that is) should have disabled support for v1 of the SSH protocol when configuring thier SSH daemons (most vendors have been affected by the CRC32 Compensation attack). Heh, ideal world situation I suppose.

Tron, if your hosting company allows SSHv1 connections then they may need to be shot in the head :-). (I'd bet they're running a vulnerable SSH Daemon as well if this is the case - most hosting companies are muppets).

You really want to use SSHv2 along with DSA keys. (ssh-keygen -t dsa)

>domain = ;
>remote_user = ;
>remote_host = ;
>#local_user = ;
>#local_host = ;
>
>tar -cpzf $domain.tar.gz *;
>scp $domain.tar.gz $remote_user@$remote_host:$domain.tar.gz;

Um, I take it this is supposed to be some sort of perl script? If your using the Bourne shell (Bash is derived from Bourne) you'd assign variables without using spaces.. e.g -

FOO=value

No need for the semi-colon either.


>ssh $remote_user@$remote_host;
>tar -zxf $domain.tar.gz;
>exit;
>#end script

Won't work, you'd SSH into the remote host and just idle, the next command in the script wouldn't execute until you disconnected and it would be executed on the localhost.

You'd want to do -

ssh -l $REMOTE_USER $REMOTE_HOST tar zxf $DOMAIN.tar.gz

Which would SSH into the remote host, execute the tar command on that host and then exit.

You'd also want to perform some sanity checking e.g. check the exit codes of the commands you execute, possibly mail out if it fails, remove the tarball after it's been transferred (or archive it somewhere perhaps).

So, a version of this script that might be useful.

-----

#!/bin/sh
# Assumes quite a bit, might work though.
DOMAIN="whatever"
REMOTE_USER="borgman"
REMOTE_HOST="foobarwibblemoo"

# Assumes the script is running in the correct directory
tar cpzf $DOMAIN.tar.gz *

# Assumes keys are setup correctly, without passphrases
scp $DOMAIN.tar.gz $REMOTE_USER@$REMOTE_HOST:

# Clean up the local tarball, if not needed...
rm -f $DOMAIN.tar.gz

# Untar/gunzip it on the remote host, assumes your home directory on the remote host is where you want the files...
ssh -l $REMOTE_USER $REMOTE_HOST tar zxf $DOMAIN.tar.gz

# Remove it from the remote host, if not needed...
ssh -l $REMOTE_USER $REMOTE_HOST rm -f $DOMAIN.tar.gz

# This is the end of the script!

---

I'd either write a better script based on the above, or use something like rsync to do the job.

--Bovine

Yeah, i was in a hurry. No you don't need the quotes when you enter in variables unless they have spaces.

remote_user=shawn will work fine.

But use bovines method. That'll work better.
you don't need keys with empty passphrases either if you use ssh-agent. But I suppose that depends on the machine.


Also:

SSH protocol version 2

When a user connects using the protocol version 2 different authentica-
tion methods are available. Using the default values for
PreferredAuthentications, the client will try to authenticate first using
the hostbased method; if this method fails public key authentication is
attempted, and finally if this method fails keyboard-interactive and
password authentication are tried.

The public key method is similar to RSA authentication described in the
previous section and allows the RSA or DSA algorithm to be used: The
client uses his private key, $HOME/.ssh/id_dsa or $HOME/.ssh/id_rsa, to
sign the session identifier and sends the result to the server. The
server checks whether the matching public key is listed in
$HOME/.ssh/authorized_keys and grants access if both the key is found and
the signature is correct. The session identifier is derived from a
shared Diffie-Hellman value and is only known to the client and the serv-
er.

If public key authentication fails or is not available a password can be
sent encrypted to the remote host for proving the user's identity.

Additionally, ssh supports hostbased or challenge response authentica-
tion.

Protocol 2 provides additional mechanisms for confidentiality (the traf-
fic is encrypted using 3DES, Blowfish, CAST128 or Arcfour) and integrity
(hmac-md5, hmac-sha1). Note that protocol 1 lacks a strong mechanism for
ensuring the integrity of the connection.


So no, an RSA key does *not* imply SSHv1.

I'm stepping through the scripts line-by-line and finding problems. Nothing wrong with the scripts...just things on the server or my lack of knowledge.

For instance, the TARing of the site is running into "permission denied" and erroring out.

Also, I did the ssh-keygen and generated the 2 keys. However, they were dumped in my home directory. I took both keys and put them in the same location of my mirror site and tried to scp and ssh in and I was prompted for the password for each connection. I guess I need to "find" or create the location I need to put those files in. Or, can I pass the password along like the username?

Tron Of Borg

Hi,

Tron:

On the remote server you'll need to add your PUBLIC key to $HOME/.ssh/authorized_keys. Keep the private key on the source server, this and the public key should be stored in $HOME/.ssh/identity & $HOME/.ssh/identity.pub if using RSA keys or $HOME/.ssh/id_dsa & $HOME/.ssh/id_dsa.pub if using DSA keys.

It could be that your provider has disabled the use of keys (doubt it). If you can look at the sshd configuration file then you may be able to figure it out look at /etc/sshd/sshd_config (could be in a different location though). Have a search for it -

find / -type f -name sshd_config -print

Are you chroot'd to your home directory or can you 'roam' about the system?

Another thing to consider would be the SSH version in use, is it OpenSSH or a Commercial version? There are differences in the way they store keys unfortunatley :/.

Try running -

ssh -V

to get the version. The above method of using the keys works for OpenSSH, the commercial version (ssh.com) is only slightly different.

When you run the script, where is it run from? Is it running as a cron job now or are you running it manually? Are you sure it's erroring while trying to execute the tar command - i.e. have you made the script executable (is it erroring because of this)?

Also, as no paths are specified in the script, you would need to execute it in the directory you want to archive. You need write permissions here, otherwise the tar file can't be created... (I'd assume that as they are your sites you should have write access :P).

Shawn:

Fair point about the keys, you'd generally want to use DSA keys with version 2 of the SSH protocol however. There are diffences between RSA keys generated for v1 or v2 of the SSH protocol also.

Nice copying and pasting from the man pages btw :P.

--Bovine.

Yeah, i thought you would dig that.

Tron: Feel free to post any errors you get here on the board. It'll make it easier for us to help you with.

I sent your suggestions to my hosting to help with the SSH stuff and they explained that by the time the DNS entries changed, they'd probably have the problem fixed.

Now, if I have 4 DNS entries with the first 2 going to my main site and the second 2 going to the mirror, how does this really effect things?

Won't the first 2 just time out and then the 3rd will kick in? Maybe I should do 1, 3, 2, 4 for the order of NS servers?

Tron Of Borg

Hi,
I take it your using some sort of Global DNS load balancing system - or is it plain old RRDNS?

Need more info :/

--Bovine.

I simply have all my customer's Domains Name Server info pointing to 2 of my DNS servers. From there, the DNS server reports the IP of their domain. The registrar (sp?) for most of them is NetSol.

Tron Of Borg

The 'right' way to handle this problem is to use RSYNC with SSH, setting a trusted key on the receiving side that only permits the execution of the RSYNC command.

This method will only copy files that are new/changed, and gives an increased security, as the key cannot be used to open a shell session.

http://rsync.samba.org/

Best of all, everything you need to do can be done as a single command in a cron job, no need to rebuild the wheel with a convoluted shell script and waste cycles on tar/zip/etc.

I've done this for multi-gigabyte web sites, with a half-dozen or more mirror servers pulling updates. It's fast, reliable, low-maintenance, and when nothing changes, almost no network traffic is used.