Power Outage

A couple of weekends ago, after a day trip with the family, we came home to a dark house after a strong storm. The electricity came back a couple hours later, so I started bringing up the servers. Unfortunately, the file server wasn't cooperating.

I've written before that this server will not hold the BIOS settings through a power off - despite changing the battery. Since it can likely be fixed only by replacing the motherboard, I've decided to live with it. Unfortunately, when I brought the box up, I forgot to reset the clock back to the current, so Linux decided to force a disk check. After about 25%, it froze. I gave it half an hour before giving up and bouncing the box.

After that, when it rebooted, it couldn't find a boot disk. Arrgh!

It was a week before I came back to it.

Since it still wasn't booting, I decided to reinstall grub as I have done several times before on different machines.

sudo grub
find /boot/grub/stage1 (this will return the drive name - mine is hd0,1)
root (hd?,?) (use the drive name returned in the previous step)
setup (hd0)
quit

Surprisingly, this worked! I'm back in business.

A side note, during my down week, I found the need to recover several files from the backup server, and I'm happy to report that all were successful. It's nice to know that the restore works not only in a test, but also when it is actually needed.

Now, here's the rub. When I thought the hard drive was toast and I was going to be tasked with recovering the entire thing, I looked through the BackupPC documentation and many Google searches, but I found no examples of someone who has done that. I'm tempted to setup a test box, back it up, and then try to recover it just to see if I can. Should be a fun weekend project.

Backup Check

After a little over a week, a quick check of the backup server shows that it's working just fine.

New Backup Server - Part 7

In Part 6, the full backup of the file server had started. Observant readers may note that the last post was made two days ago. That's because the backup finished after 44 hours! We backed up 143GB with a compression average of 27%.

I think I'm setup. I'll try a quick recovery, but I don't expect problems with it. Let's call this project done.

New Backup Server - Part 6

Part 5 saw the localhost backup run successfully. Time to move on to the big prize, the file server.

Before I can backup my file server, I need to setup the public keys required for rsync to login to the server as root without a password. For this part of the task, the new guide is pretty limited and apparently missing some steps, so I'm reverting back to the HowToForge Guide on Page 4. The instructions are clear and complete, and it worked last time.

As I went through these steps, everything went well. The commands made sense, and the responses came back as expected. Except for the path to the home directory on the backup server. The guide said to expect /var/lib/backuppc, but instead I saw /home/backuppc. I wasn't able to change it, so I went with it.

Everything was great, until the last command which tested the connection, as it still asked for the password. I changed one thing: on the file server, I removed what I had added to the authorized_keys2 file - from = "gentry2" - and the ssh command worked.

However, my attempt at a backup still failed. What finally made it work was this: instead of using the IP address in the ssh test, I used the hostname of the server. I was asked to confirm the RSA key fingerprint once again. Then, the backup worked. Yeah!

I hope to better understand this stuff someday, but I'm just happy that it's working. (I wonder how long this full backup will take.)

Read the exciting conclusion in Part 7.

New Backup Server - Part 4

As I mentioned in Part 3, I found a new guide for the installation and configuration of BackupPC - BackupPC Setup at TakSuyama.com. I'll give it a try.

The first step is to install the pre-requisites:

sudo apt-get update
sudo apt-get install ssh samba smbfs rsync apache2
sudo apt-get install php5
sudo apt-get install libapache2-mod-php5
sudo /etc/init.d/apache2 restart

So far, so good. The restart of apache2 completed without incident and I can see a default web page. Just like inPart 2, I'm receiving a hostname error from apache. I'll try to fix it the same way.

That worked well - no problems.

Next, the guide says it is time to install BackupPC.

sudo useradd backuppc
sudo passwd backuppc
sudo apt-get install backuppc

OK, the web interface for BackupPC is up. (I wish I could take a snapshot of this point, so it would be easy to go back in case of problems.)

Next, I'll change the password.

htpasswd /etc/backuppc/htpasswd backuppc

Using the Edit Config tab, I changed the TopDir to /backup, and restart the web server again - just to be sure things are still OK. And they are. I'm feeling pretty good about this.

Unfortunately, as I attempted to restart BackupPC, problems came up. Here's the error:

* Restarting backuppc...
No process in pidfile `/var/run/backuppc/BackupPC.pid' found running; none killed.
grep: /etc/backuppc/config.pl: Permission denied
BackupPC cannot be started because important parameters are missing from config.pl.
If you just upgraded BackupPC, please update /etc/backuppc/config.pl.

Which is a pretty poor error message. Why not tell me what needs to be updated?

After thinking about it a little, I looked at access settings on the file. I changed the security levels on the config.pl file with sudo chmod a+r config.pl, and I was able to get a little further. Now the error on BackupPC restart is start-stop-daemon: Unable to set gid to 1001 (Operation not permitted). Slow progress!

UPDATE: Sometimes I get worked up and miss the easy stuff. The restart of BackupPC should be performed by root, so the correct command is:

sudo /etc/init.d/backuppc restart

More in Part 5.

New Backup Server - Part 2

(This is the third time I've tried this)

In Part 1, I installed the base linux software and made a few tweaks. In this exciting episode, I'll install the backuppc software.

Just like last time, I'll follow this tutorial to install the software.

(for this version (3.0.0) at least, be sure to check 'apache 2' as the web server of choice)

During the installation, as apache was starting, I got this error:

apache2: Could not reliably determine the server's fully qualified domain name

Several sites, including this one recommend that I modify /etc/apache2/httpd.conf to include the line:

ServerName hostname

I used the same name assigned as the hostname during the Ubuntu setup.

As before, I modified the topdir setting, but I used the Edit Config option to do it.

When I attempted to restart apache2, I received some serious looking errors:

* Restarting web server apache2 httpd (pid 13462?) not running
install: cannot change owner and/or group of `/var/lock/apache2': Operation not permitted
(13)Permission denied: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs

I think it has something to do with the apache2 server running as root rather than backuppc.

Enough for tonight. More in Part 3.

New Backup Server - Part 1

I'm starting with an eMachines 600MHz box with 256MB of RAM. There is a 10GB harddive, and I just installed the 250GB drive which will hold the backups. The operating system of choice is Ubuntu 7.10 Server.

Right of the bat, I can't seem to use the USB keyboard in the installation program. No worries, as I have a PS/2 model around here somewhere...

I'm up to the partitioning step, and I've formated the entire 250GB drive as 1 ext3 partition which is set to mount as /backup. On the 10GB drive, there was already a swap area, with the rest as ext3. I set that to mount as root, and erased the partition contents. Odd, the drives are showing up as SCSI, while they are actually PATA.

Partitioning done, the base system is installing.

It's been stuck at 83% for at least 10 minutes. Can't be good. The hard drive is blinking occasionally. Patience, Dan ... it finally moved.

At the screen asking if I want to install any server software, I chose only the OpenSSH option.

And it's reboot time.

Next, I'll do the usual stuff. (Plus, I installed lynx - my favorite browser)

Looks like in this version, unlike 6.10, the Universe repositories are already enabled.

No problems. Ready for Part 2

New Backup Server - Preamble

OK, so it's been about a month since I installed the new drive in the file server, and things have been working great since. The next task is to put the other new 250GB drive in the backup server so that I can backup all of the data on the file server - rather than excluding half of it. My goal is to be able to restore the entire system if needed.

I didn't really mention before that the old Compaq machine that I assigned to the backup role has always given me trouble. At power up, I had about a 50% chance of a successful boot. A couple of weeks ago, I installed the new drive, and it wouldn't boot for anything. It wouldn't even POST. I disconnected the new drive, and it booted once, but never again. After monkeying with it for an evening, I removed the new drive, and chucked the rest of it in the trash! If I can't get it to boot without lots of TLC, I probably shouldn't rely on it for my backups.

So, now it's time to start over. I've got another machine, which was once my daughter's Xubuntu machine, but she's moved on to my old desktop, so that box is now available.

I'll start with Part 1.

BackupPC #7 - Epilogue

In part 6, I had backups running successfully from each machine - one Linux file server and three Windows clients. Before I call this project finished, there's one more thing I have to do.

I was looking at my Windows desktop, and noticed that I still have the 99GB Backup folder on the local drive. This is a backup of the file server, plus a few older files. But, before I delete this backup, I want to make sure that the new backup is working. So, it's time for a restore.

I had placed a file on the server earlier with the intention of deleting, then restoring it. I tried this tonight, and it worked without a hitch. Still, it will probably take me a little while to build up the nerve to actually delete the old Backup folder.

My only negative issue with BackupPC is that it seems that I have to store clear text passwords on the server, and keep shares open on the clients. It would be nice if the windows machines had an option such as rsync on Linux that would be more secure. (There probably is a better way to do it, but I don't know what it would be. Those with more experience are welcome to leave suggestions and hints.)

BackupPC #6 - Actual Backups Completed

When we last heard from our hero in part 5, he had finally figured out how to connect 2 Linux hosts together.

Since then, I've backed up the file server (22 hours), and 2 of the 3 desktop PCs. One problem that has not gone away is the fact that I can't connect to the other DHCP hosts. I tried modifying the DHCP Address Range parameter like this:

$Conf{DHCPAddressRanges} = [{ ipAddrBase => '192.168.254',first => 13, last => 253}];

But still no luck.

Just to get things going, I started the backups from the target machines so that the program would know where to look. Now that I've done this, I can start future jobs from the server or from my machine (assuming the DHCP assigned address doesn't change). I'm thinking it might be easier if I just use static IPs for the desktops!

Anyway, the backup of my daughter's machine, as well as my desktop, went great. My wife's is giving me a generic 'backup failed' error. I'm sure I'll figure it out.

The file server backup took 22 hours, but that was only on the 2nd try. The first was interrupted after 6 hours by a power outage here at the LSD Ranch. Apparently it starts from scratch after being interrupted.

Learn the exiting conclusion of this story in part 7.

BackupPC #5 - Unlocking the Keys

In part 4, I still hadn't resolved the public/private key issue. The problem was that the scp command called for to move the public key from one server to another kept failing. I was typing in the correct password for root, but it wouldn't take it.

Just a couple of minutes ago, I had a 'DUH!' moment. Way back when I was configuring ssh, I disallowed login by root. Yet this was what I was trying to do. After changing that option back to Yes, it worked great.

The keys are merged, and the full backup of the file server has started. I have no idea how long it will take to backup around 50GB, but I'm not going to stay up to find out. Turn to part 6 for the answer.

BackupPC #4 - Configuration Tweaks

While I wait for the desktop machine backup started in part 3 to finish (it's really taking a lot of processor time on the desktop machine), I was reading through the configuration file and decided to change a few things.

First, to limit when the backups run, I changed the wakeup schedule, which was originally set to once an hour, to only run at 2:00 AM.

$Conf{WakeupSchedule} = [2];

Next, I adjusted the number of simultaneous backups allowed so that I wouldn't overload the backup machine's old, slow processor.

$Conf{MaxBackups} = 2;

Finally, I noticed that files in the initial backup weren't compressed. I found that the compression level was set to 0, which means no compression. I changed it to the recommended value of 3.

$Conf{CompressLevel} = 3;

The docs say that a perl library called Compress::Zlib is required for compression to work. I'll have to see if it is loaded.

UPDATE: Apparently, the compression library is available, because a fresh full backup of the backup server achieved 73% compression. While that was running, I added two more desktops to the list. I'm going to refrain from starting the first backups manually, and instead let the auto job pick them up at 2:00. I'll check the status in the AM.

I found another tutorial on the Debian Help site that doesn't mention using public/private keys for rsync use. I'll give it a read and maybe try another configuration for the file server backup.

I can't wait until part 5!

BackupPC #3 - Backing Up a Windows Machine

I'm going to take a break from the public/private key issues as described in part 2, and instead focus on backing up my Windows machine.

We keep all of our documents, photos, music, and other important files on the server, so I only need to backup machine specific things. In my case, I'm mainly interested in my Thunderbird address book. After a little digging around, I've decided to grab the Application Data and My Documents folders. I'm not looking at the entire Documents and Settings folder because I don't want to backup all of the temporary Internet files, nor will I include the Desktop folder, since it is where I store downloaded TV shows and other temporary things (that add up to lots of GBs and would fill up the backup server quickly).

As instructed on page 6 of our tutorial, I started by setting up a shared folder on the desktop machine. I added a line to /etc/backuppc/hosts for this machine, and this time I used a '1' in the 2nd column to indicate a DHCP assigned address.

Next, I created a configuration file for the desktop and copied the suggested lines from the tutorial and updated the information for my situation. I don't really like storing my password in text like that. I'll have to secure the file to keep prying eyes out.

I also included the following line to limit the backup to the two desired directories.

$Conf{BackupFilesOnly} = ['/Application Data', '/My Documents'];

I restarted the daemon, and then started the backup from the web page. It's going and seems to be OK.

In part #4, I'll return to the file server problems.

But at Least the Email Works

This message arrived in my Inbox today:

Dear Dan,

Your PC (hostname) has never been successfully backed up by our
PC backup software. PC backups should occur automatically
when your PC is connected to the network. You should contact
computer support if:

- Your PC has been regularly connected to the network, meaning
there is some configuration or setup problem preventing
backups from occurring.

- You don't want your PC backed up and you want these email
messages to stop.

Otherwise, please make sure your PC is connected to the network
next time you are in the office.

Regards,
BackupPC Genie
http://backuppc.sourceforge.net

BackupPC #2

In part 1, we left our hero with BackupPC installed but with nothing to back up. Or so he thought....

When I came downstairs this morning, I refreshed the BackupPC Server Status page, and found that the file pool now contained 928 files! From where? It seems that the default setup will backup the localhost machine. Now, this box only has a few hundred MB of files on it - Ubuntu and the BackupPC files. Apparently, a full backup was done during the night, and I was greeted with a successful backup status. That means it works!

Now it's time to setup the main file server for backup. I started tonight by adding the server to the hosts file - /etc/backuppc/hosts. I added the following line:

hostname 0 username

(You can insert your hostname and username instead of my italicized placeholders.)

And I used this command to setup the user in BackupPC:

htpasswd /etc/backuppc/htpasswd username

For this next part, I'm going to deviate a little from the instructions. The config.pl file is setup to backup localhost (as I learned this morning), and I don't want to change that. So instead, I will create a new config file for the file server. It will be called:

/etc/backuppc/hostname.pl

I only need to enter the settings that are different from the main config file. Here's what I came up with (sorry for the extra line breaks):

# BackupPC config file for hostname

# Set the file transfer method
$Conf{XferMethod} = 'rsync';

# Use $hostIP for the client path
$Conf{RsyncClientCmd} = '$sshPath -q -x -l root $hostIP $rsyncPath $argList+';
$Conf{RsyncClientRestoreCmd} = '$sshPath -q -x -l root $hostIP $rsyncPath $argL$

# Name the partition
$Conf{RsyncShareName} = '/';

# Limit backup to the /export directory
$Conf{BackupFilesOnly} = '/export';

# Skip the music files and podcasts
$Conf{BackupFilesExclude} = ['/export/music', '/export/Podcasts'];

We'll see what happens.

After restarting the daemon, it's time to configure SSN and rsync. I followed the instructions on page 4 of the tutorial.

... 30 Minutes Later ...
I'm getting totally hosed on the public/private key stuff. I wonder if it is because I've already generated keys before for openssh access. I'm going to pack it in for the night, so stay tuned for part 3.

BackupPC #1

I started the BackupPC installation tonight. As a guide, I'm using this tutorial from HowToForge.

Starting with the very first step, my installation is different than documented. The mail program, exim, is asking questions I don't know how to answer. I guessed that I should use a smarthost to send mail with no local delivery. I set the domain and SMTP machine to my domian and my hosting company's mail server respectively. We'll find out later if those choices were correct.

Also, the setup never asked for my workgroup name.

The web interface came up without incident.

I should have noticed earlier that the data is stored in /var/lib/backuppc, so I'm going to have to setup my 80GB drive with that path. I've been looking through the config docs to see if I can alter that path - and I think that I should be able to, since it refers to a variable called $topdir that references the path - but I can't find anything. ...... You know, I'm going to look a little more before using that default. Hold on...

... OK, so I'm glad I checked. I found a note on this page that tells us that the TOPDIR setting is in this file:

/usr/share/backuppc/lib/BackupPC/Lib.pm

I changed the setting to /backup, the path I set when I partitioned the drives.

I restarted the daemon with /etc/init.d/backuppc reload, and now the web interface won't come up.

Error: Unable to connect to BackupPC server
I moved the directories from /var/lib/backuppc to /backup and restarted the daemon, but no luck.

I know this is a newbie move, but sometimes all can be set straight by a quick reboot. (It's my years of Windows training that made me do it!)

That's enough for tonight. More soon in part 2.

The Usual Stuff

Like the last several times, I'm performing the basic server setup using Ubuntu 6.10 Server Edition. It's a good basis for what I'm doing. Like before, I'm:

Assigning a Static IP
Configuring the Repositories
Setting Up SSH
Performing Software Updates
Making SSH Configuration Changes

And configuring ntpdate as outlined here.

UPDATE: 30 minutes later, and these steps are done. That's all for tonight.

Building the Backup Server

Well, it's been a week since I switched to the new server, and there have been no problems. I had backed up the contents to my desktop hard drive just before the switch, so I'm ready to re-purpose the old server. It's going to become the backup server for everything else.

Long time readers will remember that the server only has an 80GB hard drive (plus a 7GB for the OS). How can I use that to back up 72GB on the file server, plus other files from the desktops, let alone save room for future growth on the 250GB drive? Well, I can't - at least not forever. But, I have a plan to get started, and I'll upgrade when I have to.

First, I'm not going to backup music files or podcasts. I have some of the music on CDs at work, and the rest I don't really care about that much. If I lose that Ted Nugent album, I'll live. The podcast directory is a temp folder at best. I move the files from the server to my MP3 player, and I only listen once. Cutting those two folders saves 20GB off the top!

Second, the software, backuppc, uses a compression algorithm. I don't expect much savings on the JPGs, but there may be a little on the rest of the files (maybe 20GB down to 5GB).

From the desktop PCs, I only plan to backup important configuration files, such as Thunderbird profiles. Anything else should be saved on the server, and the family knows that.

I'm starting by installing Ubuntu 6.10 Server Edition. I'll run through most of the same steps used on the file server, except for the Samba stuff. (We'll see.)

I'll report back after the initial setup is done - before the backuppc work starts.