Trygve.Com > Diary > "The bleary-eyed Nyx project"
Even Gretta refers to man pages sometimes

Trygve's digital diary
special Nyx rebuild edition -- fall 1999:

"The bleary-eyed Nyx project"

In a directory, plagued by corruption...files were mysteriously vanishing....

Okay, wait, I think I've heard the trailer too many times and it's affected my brain...let's start that over again....

'Twas the night before diskcrash
and all through the house
Not an admin was working
or using a mouse....

Nope, that's not right...take three....

The Sparcs at Nyx HQ before the rebuild
trygve logo
what's new

Better homedirs and garbage, anyway, I was at Rob's Rapplean's [ ] housewarming party (that's Rob over there; he's the one on the right in this picture).

Though it was really more of a garage-warming party. You see, Rob's garage is bigger than his house and, um, since there's not actually a driveway or anything functionally similar leading up to it, it's not really suited for parking cars inside, or at least getting cars in and out of, anyway. But he can fit a big party in there, so that's what he was up to that night.

Darlene with management tool

...meanwhile, Darlene, intrepid Nyx director by day, slightly less intrepid Nyx director by night, was heading up to Nyx HQ, hoping to restart Arachne (the webserver) which had crashed and was refusing to respond to any attempts to restart it remotely (sometimes these machines just want a little personal attention).

While Darlene was on her way, one of our volunteer admins was working on a kernel upgrade to Nyx's news server, Iris, and starting the new kernel left Iris also sitting there idly consuming electric power and refusing to boot up.

Unfortunately, neither Arachne nor Iris would respond to Darlene's ministrations--and, instead, Nyx10, the main login machine and currently the box housing everybody's home directories, also crashed and wouldn't come back up.

to put it simply, Nyx was on the fritz...

Did somebody order a pizza?

whew! and so I spent most of Rob's party holed up in his basement trying to talk Darlene through her attempts to tame a roomful of rebellious Unix machines...and after a few hours of this, I decided to pull rank and make the executive order to take Nyx down and have her bring the machines down here to the treehouse.

This isn't quite as bad as it sounds, because I had all the Sun monitors, keyboards, terminals, and networking hardware that we needed already, so it was just a matter of bringing the CPUs and external drive boxes down, but even so, we're still talking about a pretty full car by the time she'd gotten everything into it. (The Exabyte 8mm tape jukebox stayed at Nyx HQ; if we really needed to do anything with 8mm tapes, I figured I could change the tapes manually.)

Better homedirs and garbage

And so, Darlene filled her car with computers and drive enclosures and just enough room for a small Nyx director in the driver's seat (proper driver installation is always very important when computers are involved) and headed down to Rob's party, where we moved everything into my car so that I could get on to the exciting task of setting up an ISP in my conference room, leaving Darlene to take on the arduous task of partying the night through. least after I'd stopped at the grocery store and stocked up on diet Coke; seemed like there was a good chance this was going to be a long night.

Better homedirs and garbage

I think I could have patched everything back together and gotten it working in about twelve hours, but with just about everything already offline, I thought it made a good excuse to fix a lot of things at once that would have meant taking the system down anyway. I hadn't actually intended it to take four straight days to rebuild everything, but I think it was well worth it.

Invoking the telnet daemon

Having everything down here made it much easier to deal with--I'd been putting together machines before this grand system crash with the intent of upgrading Nyx, but configuring and testing them down here without benefit of the rest of an ISP isn't so effective when it comes to tracking down the little problems and incompatibilities that have a habit of cropping up--and then when they go up to Nyx HQ, those problems and incompatibilities like to crop up in the middle of the night, and even when I do go up there in the wee hours, it almost always seems like there's some important part that I have down here that didn't make it up with me. Out here at the treehouse, I figure I currently have at least enough equipment and parts to whip up another three complete ISPs but, unfortunately, I can't just throw the whole warehouse into the back of the car (not that I haven't tried on occasion).

rubber rodents for free internet access

Once everything was set up down here, it was great; I had the tools, I had the technology, and most of all, I could kick the hardware when necessary. Besides tracking down a lot of long-time problems, it gave me the chance to replace every hard drive on the Nyx machines, something that had to be a positive change with some of those drives having been spinning continuously for nearly a decade. (Those 100meg drives were pretty much state-of-the-art back then. Life's different now.)

typing furiously away

Everything's mostly back where it was, except for some minor configuration twiddling and new hard drives. Arachne had the most problems--its longstanding stability problems turning out to have a lot to do with a partition table that allowed the swap partition to overlap a portion of the filesystem mounted on /lib (oops), but persistent memory errors convinced me to build a replacement webserver from scratch. Arachne had been a Sparc 1+ upgraded to an Sparc 5 equivalent with an Axil drop-in upgrade board, so switching over to a real Sparc 5 didn't require much reconfiguration.

Is that a bug or a feature?

I don't have a version of Linux that runs on this platform (at least not one that's bug-free) but this guy caught my eye this morning. I know, it's just a grasshopper, and you can't really tell it from this photo, but this grasshopper was really big. Not big for a bus or even for an apple tree, but it was the biggest grasshopper I'd seen around these parts in the past several years (almost as big as my index finger). Not that this really has anything to do with Nyx, but I don't have any cats to put pictures of on the web, so I think I can get away with putting up pictures of rocks and bugs instead.

Office A on a Monday night

But, um, getting back to our saga, after a general system overhaul, the machines that make up Nyx all checked out okay and headed back home to Nyx HQ, hopefully feeling rested and refreshed after ther little vacation (it's not every ISP that gets to travel, donchaknow). Back here at the treehouse, I'm sure the computers that make up my lttle home network miss getting to hang around with their Nyxish cousins, but at least they can still send packets to each other when they feel like it.

Postscript: in April of 2000, after years of attempting to get at least a few T1 lines out to the treehouse, USWest (now Qwest) finally laid new fibre lines between here and downtown and the treehouse was on the net. Nyx was then finally relocated to the treehouse and no longer had to be operated by remote control and this has also made it much easier to maintain and upgrade.

Getting the server room set up in the basement also helps keep the machines cool and means that I don't have to keep them in my converence room any more.

but will he see the light?
Trygve's Digital Diary
the base of the tree