I don't know about wisdom, I'd rather call it experience, I'm also not an engineer, I never had any formal training. I'm a veteran System Admin, I've written some articles and taught to other SysAdmins. I've been working with Unix almost exclusively since the early 80's.

What's amazing about Unix is that code I wrote more than 25 years ago still compiles and works unchanged today.

"Amazing stories": Here's a couple.

1

I once went to a customer's (big lawyer firm) during a week-end to do a big upgrade on a server. It had one big disk (60 MB I think it was) that I had to remove from the old server and install in the new one and copy the data to the new disk, which was a bit bigger. These were large 5.25 inch MFM disks. New server is installed and up, I connect the old disk and start copying: It's slow as mollasses, one 10KB file could take 4-5 minutes. So I quickly wrote a script which used dd to read and write one sector at a time and show me progress. Reads would hang for minutes at a time every 27 sectors.

So I called the firm's SysAdmin at his house and ask him about backups (he'd never done any) and if he'd noticed anything strange about the server. He says "Yeah, it's been horrendously slow for a week, so we're happy about the upgrade". I hang up.

On a napkin I figure that the copy will take upwards of ten days at this rate, and the client expects the server up Monday morning. This is bad.

I clearly recall seeing the heads' stepper motor shaft on the side of the disk, so I gently put the tip of my finger on it to see if I can sense it trying to seek repeatedly while reading. It does, but I notice that when I apply a little bit of pressure it stops, and on the console I see my program outputting successful sector numbers non-stop.

I spent the next 20 hours with my fingertips on a shaft. Data transferred successfully, customer happy, and I couldn't type for the next 2 days.

2

As part of a new systems installation at the northern tip of Canada, the only problem reported was that some terminals were showing the wrong time. Some were local, some connected through DATAPAC, some through satellite links. After 30 minutes of head scratching I realized: This server has terminals in three time-zones. Easy TZ fix.

3

At a software development company, we made a full Library management system which used at its core a home-made database manager. Records in this database were tunable so that they fit in sector lengths exactly, which is fine. A lot of our customers were on MIPS systems which used 8K sectors, and a few of our very large customers had terrible performance problems. We were close to being sued for unfulfilled marketing promises. The whole development team were having their butts kicked non-stop and ordered to "fix it."

After 3 weeks of headless chickens running around I overstepped the bounds of my responsibilities and take a look at the problem. In 20 minutes I discover that yes, records are 8K, sectors are 8K but at the very beginning of every DB files is a 1K DB description record. I demonstrate to the Senior programmer that ever DB read incurs a 2-sector physical read and any write incurs 2 reads and two writes because every record is overlapping two sectors. I ask him to grow the DB descriptor by 7 kilo-bytes of filler. We install a small update at the customers, re-structure the DB files and hey presto: system load goes from awful to very responsive. Everyone happy all around and I saved the company millions from a sloppy software design.

4

At an aeronautics company the development team are writing their own code to simulate heat-exchange in jet engines. The data sets are massive, the number cruncher servers are overloaded and are swapping non-stop for days at a time. I look at the situation with a senior dev.; they open a humongous file, read it in memory, crunch the data, then write it back to disk. I show him how mmap() works: Hey presto, 400% performance increase and the code is simpler. Millions saved. (mmap extends the system's virtual memory with a file's contents, an amazing concept.)

5

I went to a customer to do a server upgrade and replacement. These machines were NCR Towers made by the famous cash register company. They were about 4 feet high, 6 feet long, 8 inch thick and weighed about 300 pounds. Upgrade went swimmingly and as I get ready to depart the nice customer says to me: "Hey what do I do with the old machine?" I says "I dunno." He says "You want it?"

For years afterward I ended up using an NCR Tower 450 with 4 megabytes of ram, a 120 MB disk, 16 serial ports and two serial terminals at my house. I could compile a "Hello World!" program in about 1.5 minutes, I didn't need to heat two rooms during winter time. The system was valued at about $22,000 back then. Before PCs were popular, I had a Unix server at home and I was happy as can be. :) It was a bitch to move around though.

I've got plenty more but this is TL. :)