People who are otherwise quite used to computers, are quite surprised to find out that we can do administration tasks on their Linux servers without leaving the office. (People who don't use computers at all find this even more surprising.) This is what we call remote support, and it is one of the things we do.

This page contains a number of glowing accounts written by ourselves about how we fixed problems without moving off the chair we were warming.

If you enjoy the sound of this page, you may also enjoy the sound of us blowing our own trumpet (Ogg Vorbis encoded, 24kb) (particularly if you have an Ogg player).

The tale of the forbidden web site

Their Mandrake Linux web server and mail server had been moved from one ISP to another. In the process it stopped working, and whenever they browsed to their web site it said "403 Forbidden". They were hoping we could go over to the ISP's premises and have a look at it to fix the problem.

As it turned out, we were able to fix it without leaving the room, saving costs on travelling time and who knows what else. (The VirtualHost configuration in httpd.conf didn't match the NameVirtualHost declaration.) (15 minutes)

For good measure, our health check script showed up that there was a problem with their mail service too which we were able to fix (their previous ISP didn't answer their DNS queries anymore) (10 minutes, although we only thought to check the health after half an hour).

The case of the dysfunctional network card

A SuSE Linux computer that sends and receives mail via a dial-up connection was misbehaving. The network connection would occasionally become unreliable, but it would work fine for a while if they rebooted the computer.

The particular network card the computer was using was an Intel EtherExpress 100. There are two different Linux kernel modules which support this card - e100 and eepro100. The problem seemed to be that the card didn't quite agree with the kernel driver it was using.

To fix this problem, you have to get in your car with a spare network card and drive out to Isando. That's not how we fixed it.

We logged in via the internet through the dial-up connection, removed the problematic kernel module, and loaded the better kernel module. This solved the problem - and we didn't have to reboot. The customer was quite surprised to see how Linux hardware problems can be resolved.

POP goes the server

Another Linux guy had been fiddling with their SuSE Linux dial-up machine for two hours the previous day, and couldn't get the mail server going. They wanted to bring it into the office for us to fix. We connected to it remotely, and fixed up the pop3 server in the time it took to finish the sentence (xinetd had to be started). For good measure we fixed the SMTP server (which was harder) and installed a nice firewall script. Not bad for an hour's work.

The sad sad tale of the dial-on-demand server
This was indeed a sad sad tale. The people are still crying, because the compassionate phone company Telkom billed them around R22,000.00 because their phone line was dialled out for a month or two. We're glad we were not involved in the saga. It is not a happy day when you have configured dial on demand and then later add a machine that fetches mail ... but has no reverse DNS for its IP address. 'Twas indeed an incredibly sad sad tale.

  • New machine - Hi! It's me - where's my mail?
  • Server - I wonder who that is ... let's dial the internet and see if we can find out.
  • Internet DNS servers - Never heard of him.
  • Server - I guess that's not so serious - he can have his mail anyhow.
  • (Repeat every 5 minutes, until the accounting department notices) (Yewowch!)
We managed to diagnose the situation in an hour (and a bit). There was also the standard spyware on a few of the computers. Another complication was that dial on demand needs silence on the connection before it hangs up - and silence is really scarce on the internet these days with the proliferation of automated MS-worms (we didn't really fix that part of the problem - we just reduced the timeout - the server was running Redhat 7.1, which does not support pppd's active-filter addition which can solve it nicely).

This is the kind of problem that makes you pull your hair out. If you are paying the phone bill, it makes you pull other people's hair out. We are glad we didn't have to leave the comfort of the office and explain in detail to the agitated individuals.

Change hard disks on the fly

These guys had a Linux server in Rosebank, and it was behaving badly - the hard disk was giving up the ghost. They could log in remotely, but could not make changes, because kernel had stopped write access to the disk because of the errors (quite a sensible measure). There was a second disk in the machine which they were using for backups. We copied the system and data from the buggy disk to the working disk (it happened to fit), ran chroot, ran lilo /dev/hda (after editing fstab and lilo.conf), and told the machine to reboot. Two minutes later it finished booting up off the working disk and not the buggy one. Nobody went to Rosebank. (We don't know if they have taken out the buggy disk yet...)