I have a rsync job running right now that’s going to take a while to complete. It’s bandwidth limited and as I sat staring at the stream of text flying across my terminal I also took stock of what I’m doing.

I’m currently migrating all the apps, content and processes from one server to another. It’s part of a hardware upgrade that also includes going to a new version of Redhat (EL5 to EL6). I’ve also taken stock of how a couple apps were built on the old server and reimplementing them in a new way that will be easier to maintain.

I’m on day 3 of this process now, and the key thing I’ve done is documented everything as I’ve done it. One nice thing about working with Linux is about 90% of my documentation can be run as a set of shell scripts, but that’s probably better explained in another blog post I may do later.

For systems administrators, as you move up in your career you’ll need to learn to do documentation. As you work your way towards senior you’ll get experience in troubleshooting which is a core part of your required skill set. Another thing you need to learn is to be able to teach and help more junior systems administrators. You also need to be able to communicate processes to your peers. The idea being if you walk out and get hit by bus someone should be able to come behind you and complete your remaining tasks without having to do all your research over again. That goes from high level descriptions of what you’re doing down to getting the required find, rsync and other commands right on their first pass.

Documenting as you go helps with this. Taking a look at the current process, there are risks. The server I’m copying stuff to is new, that means a RAID controller, disk, or any other physical component in the server could die. If I wasn’t documenting every step of the process as I did, I’d likely have to research parts of it all over again if I had to start over. Now, I don’t. I can follow the script I’ve put together and likely have the last 3 days worth of work done in 1.

Some steps will need to be done again the day of the actual swap over. Part of what I’m moving is a large svn and trac install. The day of the move I’ll have to make sure everything is current on the new server. Already I’ve identified I’ll be better off making a dump of svn and trac and copying them over rather than trying svnsync. Using Trac on the new server which is an upgraded version required syncing it to the repositories which broke the svnsync I had set up. Plus, svnsync for an enterprise repository several years old was extremely slow. I was able to change the documentation for day of to use dumps instead of svnsync really quickly because of the documentation I’ve been doing.

It’s also good practice. You’re going to need to know how to do good documentation if you’re going to advance as a systems administrator. So documenting at every opportunity improves your ability to provide good documentation, experience is key for getting better at anything.

In 4-5 years we’re going to need to do this again. When that time comes the documentation will be there to make the job that much easier. I’ve also got a task to look at configuration management applications like Puppet this year. The documentation I’m building now can potentially assist in setting up Puppet scripts in the future to keep a lot of the configuration I’m currently managing by hand in one place.

As a Systems Administrator/Engineer you’ll find over the years you’ll forget a lot as you learn new things. High level concepts will stick around, but the details will slip away. If you document them, you’ve basically just upgraded your personal (and enterprise) memory capacity.

2 key things to remember.

- You can never have too much documentation.
- The correct answer to the question “How do you know when documentation is complete?” is “It’s never complete.”