Ah Gnome3, KDE 4.x, Unity… how the GUI desktop landscape has changed over the past year or so for Linux. Honestly, I haven’t been a fan of any of them.

KDE, it’s nice but experience has been it just seems slow. I’m not sure how to explain it, but just basic interaction with desktop has always left me with this feeling of it being sluggish. Once upon a time I was a huge KDE fan, likely from my years of using Slackware for everything. I actually did just install it at work, the ATI card I have there doesn’t play nice with Gnome3, therefore Cinnamon as well.

Gnome3, just feels like a mess. A vanilla Gnome3 install just isn’t very intuitive. Gnome has a trend of trying radical things that are just frustrating. Remember when Nautilus went spatial? Ugh. Drove me nuts a time I was actually turning into a huge fan of Dropline Gnome. With Gnome3 was it really necessary to for ctrl-alt up/down instead of left/right for desktop switching? I only have over a decade of muscle memory there.

Unity… it’s interesting. I think a big reason I tend to stay away from it is my negative experiences with it when it first came out. It was glitchy on my hardware, locking up at random times. In all fairness with 11.10 it worked fine. I think I just had developed a bad taste for it by then. Unity has no problem running on an ATI card though *nudge Gnome3*.

Now Cinnamon. The only fault I can find with it is it’s Gnome3 roots means it has the same hardware issues. My laptop with an Nvidia card runs it great though. Cinnamon just seems to be what Gnome3 should have been.

Cinnamon has lower panel with the application menu. I like the fact you can add apps to it real easy, acting like a dock. I have a Macbook I use primarily for working from home, I’m used to having a dock (and at the bottom of the screen, Unity). However it builds on Gnome3’s always having one empty virtual desktop. Over all I find the virtual desktop management which is pretty much all Gnome3 to be an improvement. It’s real easy to move applications from desktop to desktop. When visually managing the desktops it’s vertival, however Cinnamon kept the left/right key bindings and visual transitions. I don’t have to relearn anything.

Over all, it’s appearance is a little rough, but the functionality has been a perfect fit. About the only improvement I’d make is a top panel that functions like on Unity and my Mac, holding the current application context menus and such. I just prefer that approach.

If you are running Mint and haven’t tried it yet, sudo apt-get install cinnamon-session and give it a go. It’s been the most intuitively easy to use desktop released this year.

So for my current project ChatFor.Us I’m using asyncmongo with tornado. I’ve started messing around with tornado.gen to try and get rid of the spaghetti callbacks I was building up. First of all, I love tornado.gen, it’s great stuff.

asyncmongo is a special case, in the it doesn’t return just one item to callbacks. Rather, it returns {response, error=None}, or error can be an object if there is an actual error. This gets a little tricky. Turns out I wasn’t the first person to run into this.

https://github.com/facebook/tornado/issues/351

As you can see, the issue has been resolved, but no one really explained how it works anywhere. What Ben did is basically build an Arguments object which is what gets returned. You can access it various ways, and here’s how I’ve found is the simplest for me.

I guess at this point code explains it best.


# execute the query, expecting args and kwargs which is a tuple, dict response
response, error = yield tornado.gen.Task(db.users.find_one, {"id", id})
# result is actually the first item in response which is a tuple
result = response[0]


I have an idea for a modification to make this a bit simpler, and will try to put together a pull request soon.

This is a follow up to my last post. Without actually writing any code I figured out a way to possibly pull of the idea in my previous post. It was a little complicated and had room to fail. It also didn’t cover keeping people logged in long term. Basically the whole “Remember me for x days” would be difficult to implement. It wouldn’t be impossible, but was complicated enough to make me think that maybe there is a better way.

Actually there is a better way, https. The problem I have with https is it costs money. Yea, not a lot, but still there is a cost associated with it and I like the whole idea of the free web. The nice thing about https is that it helps ensure you’re talking to who you think you’re talking to. That’s the whole point of ssl certificates. It’s not perfect, there have been problems in the past. However it’s a pretty good system that gets the job done.

I’m not proposing to change that. My idea is for when you need less secure data transmissions. httpe or http encrypted.

The basic principle is that requests and responses would be encrypted using a shared key between the client and server. The basic negotiation would be:

Client sends an unencrypted request, which consists of randomly generated key.
The Server responds with a response encrypted with the key sent by the browser. The payload being a new key.
The Client and Server could then begin encrypted conversations using the server generated key.

You could then set cookie headers and such as you like, for example to maintain persistent across sessions.

The risk of someone hijacking a session is limited to someone sniffing the original request. There would be some overhead. The server would need to maintain the key for at least the duration of the session. Then again, it already needs to do this for the session token any way.

At the worst this would allow people to set up and get basic sessions and encrypted traffic running for their prototype and early beta ideas without forking over the money for an ssl cert to start. Meanwhile the way we’ve been doing http sessions for years doesn’t really have to change.

Edit: HTTP 1.1 Upgrade header looks like a similar approach that’s probably a lot better than the one I came up with while drinking coffee. http://en.wikipedia.org/wiki/HTTP/1.1_Upgrade_header

First, please excuse any typos or weird formatting. Writing this post on my Android while waiting on my car to be serviced. Been here 4 hours so far, doing this also to kill boredom.

This is a post I have been wanting to do for a while. I have now written 2 sessions libraries. Gaeutilities and asyncmongo sessions. So I feel I have a pretty good grasp of the issues.

With both libraries I introduced some security concepts that affect performance. The main security issue with http sessions is unless you are using ssl it’s too simple to sniff and replay data to hijacking a users session.

My implementation both times to address this has basically been a kludge. I constantly change the token stored as a cookie in the browser to make hijacking more difficult. I also suggest people implementing the library always tie the user account to an active session.

In an ideal all this token rotation would not be necessary. With HTTP 1.1 we can keep a persistent connection open. What would be nice is if we could start using that connection as a session and also have a way to recover if that connection is lost and reestablished.

We would also need a way for load balancers to interact with backends to manage that session id. That way the lb could manage the session itself. There already exist headers for proxies to pass client up information, persistent connection sessions could key off of that.

Cookies would still be useful. Sessions are really only necessary when identity must be confirmed.

Sessions are a problem that have been solved a lot of ways by a lot of frameworks. However as we move to browsers and http servers ( or at least proxies ) supporting persistent connections I think it can be done in a way that provides more security.

This is probably a little vague. I may see if I can a hack a model together that demonstrates the concepts. The main question is how to handle the connection being lost and not losing identity when the client reconnects. I don’t think moving to requiring a constant connection only is the right answer, that feels like a step backwards as well.

This is response to this post - I’ll Probably Never Hire Another Pure SysAdmin - and others like it I’ve seen lately.

Now I started out in IT back in the late 90’s, out of highschool. The internet wasn’t like it is now, with the wealth of information and thousands of blogs. We had Slashdot and that’s about it for high profile tech news and discussion sites. These days I imagine people interested in Systems Administration have a much broader range of articles to read and I imagine a lot stick with tech focused sites. That makes articles like this dangerous, because quite frankly, they’re wrong.

Sure, technology focused small companies may have less need of a Sysadmin. Of course, technology companies are going to be started by people who have at least a passing interest in technology in the first place. They probably have an idea and opinions about many facets of technology, from computer languages to operating systems. They’re comfortable getting Rackspace cloud machines or spinning up Amazon instances. They also are not a majority of businesses out there.

I’ve never worked for a technology company. I’ve always supported businesses that focus on non-technology endeavors. What I can tell you is that these companies are going to want to Sysadmins for some time to come. Some may try to do without them, but eventually they’re going to need them.

Now, I’m not sure what a Pure Sysadmin is. By nature of the profession I don’t believe there is a such thing. You have devops guys who are passable programmers. You have Sysadmins who come from the networking stack. You have the operating system gurus. You have the virtualization guys. There’s a lot to systems administration, and the areas you become good at often depends on your background and the needs of your current employer.

So, I’m going to dive more into business needs for Sysadmins, rather than focus on a specific type. Business has an interesting take on technology. You have the people who have no interest in technology. It’s a tool to get a job done. Don’t make the mistake of thinking these are dumb people who can’t learn technology. These are more often than not incredibly intelligent people who are too busy getting a job done to have the time to learn new technology. Executives, scientists, marketing geniuses and the like. These are the guys you’re supporting an Exchange environment for, because even though Zimbra or Google Apps seems like an obvious no brainer to you, they’re not interested in it. Heck, just updating Outlook 2007 to Outlook 2010 is a major migration for your company of 3000 employees, that costs more money than you care to think about in training and lost productivity.

Cloud seems great. In fact you’re going to have business users who do keep up with latest trends bringing it up all the time. However, with the cloud the business first loses dedicated support staff. Instead they’re sharing support staff with all the clouds other customers. Also, there’s information the business is just not comfortable hosting outside it’s walls. Accountability is big too. Also, when business is big enough, it needs to be able to schedule maintenance around the business, not vice versa. Google Apps is going down for maintenance that day you need the product up for a major trade show? Too bad. Better have a local copy you can demo and promise the clients you can show them the real thing later. I know my department has had to schedule around a conference before.

Business often dictates technology to you. This is actually one of the frustrating things you learn about being a Sysadmin. Yes, from a pure technical sense you probably are the most qualified to make the choice on what software the company should use. The reality is, you provide a service to that company and they’re going to tell you what software they want. It’s your job to make it work, no matter how frustrating it can be.

When you look at the pure dollars and cents, if the could was going to knock out the Sysadmin profession, it would have done it already. In a lot of cases it probably is cheaper to run in the cloud than host your own datacenter. IT departments are a cost. If the business isn’t technology based you spend lots of money while providing no revenue. Yet, we’re all still here. Heck, my department is over worked and we’d love to expand the budget to bring in a couple more people. Yet, most people aren’t using the cloud after years of it being available. Business needs infrastructure that fits it’s model. There’s no cookie cutter cloud solution that fits the core goals of a company. Lots have tried. Email gets outsourced, cloud provider goes down, suddenly they consider bring it back inhouse. Why? Well look, it’s easier to keep email up and running for a few hundred to few thousand accounts than it is to keep it running for a few million.

I’m a web guy, a large part of my work has been supporting web sites. I have yet to see where anything cookie cutter can support a company website of any decent size. Either the website is a major product offering, or a major marketing tool Either way, the business has specific features they want that website to offer. You may deploy a CMS, you’ll be customizing that CMS. You’ll also be doing a lot of web server tuning to make it work like they want it to work.

Businesses by now understand they need a solid infrastructure to run on. They need Sysadmins whom they should never see. Everything needs to just work, and when it’s not working they need to know that someone is fixing it immediately. Those someones should know it’s down before the business does and hopefully have it up before the problem is noticed. They need people who understand N+1 for redundancy and availability. They those guys who spend hours writing bash/perl/python/whatever monitoring scripts. They need the people setting up a WSUS server and scheduling patching for Tuesday, Wednesday, Thursday and Friday and spending each of those mornings running tests on every machine to make sure Black Tuesday patches didn’t knock out a mail store or to be sure the Blackberry Server doesn’t need yet another reboot at 5am.

They need people who have an entire share of documentation and regularly have discussions about is it good enough that if someone had to replace them if it would be manageable or not. They need people who discuss that eventually most servers are going to need to switch to postfix because Redhat has moved to it and in a few years people are going to be looking for jobs knowing that instead of sendmail.

They also need people who can sit in on meetings and have intelligent conversations about business topics and how they relate to existing organization infrastructure and methods. As I said, I’m not sure what a Pure Sysadmin is. But I do know that Sysadmins aren’t going anywhere any time soon. So, if you have an interest in technology and business and are thinking Systems Administration is the way for you to go to enjoy your career, do it. Don’t hesitate. Heck, when I started down my path I thought I was going to be a developer. Then they started outsourcing those jobs and I became a Sysadmin with the though that they couldn’t outsource cause they need someone in house to manage those servers. Well, look around. The developer jobs came back. If there ever is a trend to move to the cloud for non-tech business, it won’t last half as long as the attempt to outsource development.

Simply put, the cloud can’t offer business the expertise it needs to be successful. The most the cloud will be is another tool in the Sysadmin arsenal to provide to best level of service to it’s customer, which is the business professional. That’s what we’ve been doing for decades and what we’ll continue to do.

One of the key secrets to being a good sysadmin is knowing how to be lazy. I’m one of those guys who will spend an hour writing a script to do something I could have knocked out in 10 minutes. Wait.. that’s not being lazy? It is if I have to perform that task more than 6 times.

My job requires a lot of ssh. My workstation might as well be a thin client. Working from home I use ssh even more. My VPN connection doesn’t give me as much access to certain areas of our network like I do when logged in locally. So I’ll do a lot of

ssh -t servericanget.to ssh servericantget.to

So what I’ve done is started playing with my ssh config.

The most time saving thing is enabling connection sharing. No I can log into servericanget.to once and keep that tab open. Then when I need to hop through it using ssh -t to get to another server, I don’t get prompted for a password.

# Enable connection sharing
ControlMaster auto
ControlPath /tmp/ssh_mux_%h_%p_%r

So, after that, what else can I tweak. Not only am I sysadmin, I’m a car junky too. Love American Muscle and huge fan of Subaru. Little turbo boxer with AWD… oh man I miss my WRX. Anyway… as you might imagine, I like performance too. So, I’ve tweaked out the ssh config adding things like compression, preference for faster ciphers, etc etc.

# Sometimes speeds up initial handshake
GSSAPIAuthentication no
# Prefer protocol 2
Protocol 2,1
# Enable Compression
Compression yes
# Update Ciphers
Ciphers blowfish-cbc,aes128-cbc,3des-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc

Last, I tweak some default host settings. Pretty standard stuff. I rarely need X11 fowarded, but I’ve had a few times where I’ve had to disconnect and reconnect with -X. I don’t see any reason not to have it on by default.

# Global defaults for hosts
Host *
ForwardX11 yes

You might note that I don’t have ForwardAgent included. Technically there is a risk in doing that if you can’t trust intermediate hosts. 99.9% of the time this won’t be an issue, but I never know when I might need to jump into a host somewhere some way that it might be a problem. So, might as well just not enable it. I haven’t actually had a case where I needed/wanted it anyway.

So, the final version can be found here - https://github.com/joerussbowman/Mystuff/blob/master/desktop/ssh/config

If I make changes, I’ll keep that repository up to date, so that’s the best place to go grab it from rather than just me copy/pasting something in this blog post that might get outdated.

If you have any suggestions for other ssh features you use, use the comments to share.

There’s a blog post being passed around various circles right now titled Node.js is Cancer.

The quick summary is the author doesn’t like node.js, thinks it’s bad for IT in general, and comes up with several sort of thought out reasons to back up his stance.

I’m not going to debate the entire blog post. I’m pretty agnostic about node.js, I’ve tried to use it a few times. I just generally prefer Python/Tornado. I do take some umbrage to one paragraph in the blog post thought.

If you’re using Node, there’s a 99% probability that you are both the developer and the system administrator, because any system administrator would have talked you out of using Node in the first place. So you, the developer, must face the punishment of setting up this HTTP proxying orgy if you want to put a real web server in front of Node for things like serving statics, query rewriting, rate limiting, load balancing, SSL, or any of the other futuristic things that modern HTTP servers can do. That, and it’s another layer of health checks that your system will need.

No, as a sysadmin I wouldn’t try to talk a developer out of using node.js, well, much anyway. My reasons for being hesitant to go down that path wouldn’t be for the reasons the author of the blog post goes into though. My primary reasons to ask if we should go that direction are:

1) It’s a new stack to support. So more work keeping it up to date. I don’t care what you bring to the table, this is always something to consider if the value of the new technology is worth the additional effort to maintain it.

2) Do we have enough developers to support it? Otherwise, am I going to to have to learn how to program server side javascript because a developer gets hit by a bus? Where I’ve been, it’s always fallen to the sysadmins to support the technology when the developers can’t.

As for the reasons the blog author thought a sysadmin would resist… well if that’s the case then I don’t think Java would have ever made it. I mean you want me to support a VM running ontop of an OS with ugly cryptic error messages? Yea right. Python? Umm hello, Perl works, why don’t you silly developers stick with that?

In case you missed it, the last couple sentences were sarcasm. I, and most systems administrators like technology. I like load balancers, any day I get to play with the F5’s is a good one. I’m going to write lots of health checks anyway. Even if that new app is written with Python using wsgi, I’m going to have to write health checks for it anyway to know it’s up and serving appropriate content.

Ted Dziuba, I get it. You don’t like node.js. Don’t bring us sysadmins into your argument though. We’re here to support our organizations and make sure people can get their jobs done in the way that’s best for them to do it. We support a lot of technology we don’t care far. We know of lots of technology that if people would just use it, our jobs and theirs would be easier. We also know the world doesn’t really work like that. We’re problem solvers, whether it’s fixing something that’s broken or providing engineering experience to get something working. We don’t dictate technology, we make it work. So seriously, leave us out of your battle. We’re not interested.

* Disclaimer: I’m blaming all typographical errors on the fact I haven’t finished my coffee yet. I’ll also blame the slightly grumpy tone of this blog post on that as well.

I’m playing with 2 different project ideas right now. One it was a given I’d go with Tornado. It’s basically a very similar implementation to what I’m doing with unscatter.com. The other though is a chat server, and I really thought I’d try some other technologies.

I had a few other technologies I want to play with. Google Go, Erlang and Brubeck were topping the list. However, life is busy for me right now. My job has a lot going on. I have a family, with 2 very young kids. It’s that time of the year where HOV traffic is actually kind of light, so my commutes aren’t as long. Holidays are coming up…

So, I’m back to using Tornado for everything. There are 2 main reasons why.

1) I know Tornado. When you use a technology long enough, it’s easier to just stick with it.

2) Tornado is extremely quick to get started with on a new machine. Much easier than just about anything else.

It’s like this.

If you’re developing on Linux or a Mac, most likely you have Python install.

So, here’s a quick script to get a Tornado project going.

mkdir ~/git
cd git
git clone https://github.com/facebook/tornado.git
mkdir my_app
cp -Rp tornado/tornado my_app/
cd my_app
vi my_app.py
# start coding

No make. No installing extra support libraries. Just get tornado, start coding.

One of the things I do on unscatter.com is update timestamps for /facebook and /twitter results every 60 seconds. I like having the “Just Now” or “5 minutes ago” timestamps, but always want them to be representative of the actual time of the status message. This came from when I had a working Twitter/Facebook/Myspace client built into Unscatter.com. Something that will be built again by the way.

However, that operation can be a bit expensive depending on how many status messages are on the page. There’s also no point to doing it if the user isn’t looking at the page. So today, while working on the site I came up with a new way to do this. The idea is to only perform the operation if the window is in focus. At first I considered updating my updateTimestamps() function to check this. Then I realized setting up a tick that fires an event I can subscribe to would be even better. That way if I come up with other operations I want to do every 60 seconds, I can plug them in right away.

So, here’s the few lines of code necessary to make this happen with YUI3.


// ticker
var windowActive = 1;

win = Y.one(“window”);

win.on(“focus”, function(e) {
windowActive = 1;
});
win.on(“blur”, function(e) {
windowActive = 0;
});

tick = function() {
if (windowActive > 0) {
Y.Global.fire(“tick”);
}
}
setInterval( “tick()”, 60000 );

Then to update the timstamps every 60 seconds, I just need to call this.


Y.Global.on("tick", updateTimestamps);


Want to change the wallpaper every 60 seconds on the crazy Bing like page you’re working on?


Y.Global.on("tick", updateWallpaper);


Or check Twitter for your latest status message… whatever.

I’ll probably expand this in the future. Thinking about including the number of the minute it’s running on and firing an event for that as well. I’ll see about turning to into a proper synthetic event and submit it to the YUI Gallery if there’s interest.

I have a rsync job running right now that’s going to take a while to complete. It’s bandwidth limited and as I sat staring at the stream of text flying across my terminal I also took stock of what I’m doing.

I’m currently migrating all the apps, content and processes from one server to another. It’s part of a hardware upgrade that also includes going to a new version of Redhat (EL5 to EL6). I’ve also taken stock of how a couple apps were built on the old server and reimplementing them in a new way that will be easier to maintain.

I’m on day 3 of this process now, and the key thing I’ve done is documented everything as I’ve done it. One nice thing about working with Linux is about 90% of my documentation can be run as a set of shell scripts, but that’s probably better explained in another blog post I may do later.

For systems administrators, as you move up in your career you’ll need to learn to do documentation. As you work your way towards senior you’ll get experience in troubleshooting which is a core part of your required skill set. Another thing you need to learn is to be able to teach and help more junior systems administrators. You also need to be able to communicate processes to your peers. The idea being if you walk out and get hit by bus someone should be able to come behind you and complete your remaining tasks without having to do all your research over again. That goes from high level descriptions of what you’re doing down to getting the required find, rsync and other commands right on their first pass.

Documenting as you go helps with this. Taking a look at the current process, there are risks. The server I’m copying stuff to is new, that means a RAID controller, disk, or any other physical component in the server could die. If I wasn’t documenting every step of the process as I did, I’d likely have to research parts of it all over again if I had to start over. Now, I don’t. I can follow the script I’ve put together and likely have the last 3 days worth of work done in 1.

Some steps will need to be done again the day of the actual swap over. Part of what I’m moving is a large svn and trac install. The day of the move I’ll have to make sure everything is current on the new server. Already I’ve identified I’ll be better off making a dump of svn and trac and copying them over rather than trying svnsync. Using Trac on the new server which is an upgraded version required syncing it to the repositories which broke the svnsync I had set up. Plus, svnsync for an enterprise repository several years old was extremely slow. I was able to change the documentation for day of to use dumps instead of svnsync really quickly because of the documentation I’ve been doing.

It’s also good practice. You’re going to need to know how to do good documentation if you’re going to advance as a systems administrator. So documenting at every opportunity improves your ability to provide good documentation, experience is key for getting better at anything.

In 4-5 years we’re going to need to do this again. When that time comes the documentation will be there to make the job that much easier. I’ve also got a task to look at configuration management applications like Puppet this year. The documentation I’m building now can potentially assist in setting up Puppet scripts in the future to keep a lot of the configuration I’m currently managing by hand in one place.

As a Systems Administrator/Engineer you’ll find over the years you’ll forget a lot as you learn new things. High level concepts will stick around, but the details will slip away. If you document them, you’ve basically just upgraded your personal (and enterprise) memory capacity.

2 key things to remember.

- You can never have too much documentation.
- The correct answer to the question “How do you know when documentation is complete?” is “It’s never complete.”