Your Linux Guy .com

I’ll spare you all of what you can read all over the Internet about the specifics of the latest Opensuse 11.3 release (and there are some good ones, like All About Linux, and OStatic to name a couple) …  And instead I’ll just provide some rambling anecdotes for you…

But I do want to say from the outset:  It really is beautiful.

I had been running the 32 bit version of the “Release Candidate 1″ on a Lenovo R60 for a week or so, and was blown away.  When it was officially released, I put the 64 bit version on a Lenovo T500 and am even more amazed.

For the most part, it has the bones and muscles of Opensuse 11.2 underneath…  But at the surface of the skin, everything seems smoother, faster, quicker, prettier, easier… just better.

You see, I’ve been running Opensuse for years as a laptop OS, and my laptop is a mission-critical tool to my work as a consultant.  And pretty much on the day of each new release, I download and load up the Opensuse OS, try to figure out how to make iFolder and Jungledisk work, pull back my data, and then try to make the rest of my favorite apps work.  This time, the iFolder3.8 client (part of the distro these days) and Jungledisk installed and worked so easily, I was up in no time flat.

It looks like VMware Server 2x is going to be more of a problem though, given the new kernel version 2.6.34.x….  Maybe we can get Radu interested, although maybe not, since he seems pretty fed up with having to support all his awesome previous work.  Well, since it looks like VMware doesn’t seem to be releasing updates to VMware Server 2x anymore (or at least that’s the rumors I hear), maybe it’s time for me to move over to KVM for good this time…

As you might have noted in my previous posts, I do not like to visit server consoles for upgrades.  So I want to thank Novell for having outlined a handy little instruction sheet here:  How to upgrade to SLES/SLED 11 SP1 (TID 7005410) on how to do various remote upgrades from SLES 11 to SLES 11 sp1, including remotely via shell session.  I’ve used these steps outlined in the section called “Update to SP1 via patches” –> “2) Update by using zypper” with great success so far.  However, I’d like to toss one or two little tips back to the community…

In these steps outlined by Novell, you are required to manually retrieve the values from /etc/products.d/*.prod and individually install the special upgrade/migration packages for each listed product definition.  And that’s fine in a lab, but with a farm of servers and potentially more than one upgrade/migration type each (in my situation they were cluster/HA servers, so each had two items), this can be a bit of a bother.  So for added convenience, I wrote a little “for” loop tweak to take care of those manual steps.

Without further ado, here are the modified steps for the upgrade to SLES11 sp1 using zypper with my tweak (thanks again Novell, and please see the caveats below):

zypper ref -s
zypper up -t patch
zypper up -t patch
for item in `cat /etc/products.d/*.prod|grep '<product>'|sed s/\<*.product\>//g |sed s/" "//g`; do zypper in -t product $item;done
suse_register -d 2 -L /root/.suse_register.log
zypper ref -s
zypper lr
zypper dup

…and there are a few very important points to be made about the above steps:

• Do not run those steps as a continuous script;  step through it line-by-line, watching the results at each step.
• Make sure your registration keys are all up to date and your repositories are in good shape before starting!  This is especially important and complex if you are using the HA extension or other custom add-on products that require registrations and repositories.
• In my experience, I sometimes needed to re-check and repair my repositories after the first or second “patch” statement, and sometimes before the “dup” statement; I do not know why.  Even Novell’s doc suggests that there may be complications.
• Afterward, you might want to go in and run the yast customer center registration tool, and allow it to clean itself up.

I hope that helps!

As a sysadmin (probably like you), sometimes I need to whip up special-purpose mini-scripts to do simple monitoring tasks that I don’t want to bother our main monitoring system with… This is one of them…

I needed to monitor a network device (an appliance, actually) constantly and rapidly, to see if the interface was flapping up / down.  So, we settled on one ping per second, with a one-time email notification when it drops, and a one-time notification when it returns.

Below is that script;  all you have to do is copy the contents into a text file (watch out for line wrapping), change the “PINGDEST” and “EMAILADDRESSES” variables, flag it as executable (chmod 700 <scriptname>) , and you should be good to go!

#!/bin/bash

PINGDEST="192.168.1.12"
EMAILADDRESSES="jpavlov@yourLinuxGuy.com,questions@yourLinuxGuy.com"
BITFLAG=0
SENTMAIL=0

while true
do
   /bin/ping -q -c1 -W1 $PINGDEST > /dev/null 2>&1
   PINGRESULT="$?"
   if [ "$PINGRESULT" != "0" ]; then
      BITFLAG=1
   else
      BITFLAG=0
      sleep 1
   fi
   if [ "$BITFLAG" == "1" ] && [ "$SENTMAIL" == "0" ]; then
      echo "Ping to $PINGDEST FAILED! Generated by $0 on $HOSTNAME" |mailx -s "Ping failure to $PINGDEST"  $EMAILADDRESSES
      SENTMAIL=1
   elif [ "$BITFLAG" == "0" ] && [ "$SENTMAIL" == "1" ]; then
      echo "Ping to $PINGDEST resumed. Generated by $0 on $HOSTNAME" |mailx -s "Ping responding again on $PINGDEST"  $EMAILADDRESSES
      SENTMAIL=0
   fi
done

A goofy, single-purpose script, to be sure; but you never know, maybe it will help someone…

This is just a simplified adaptation of a community post by author sakila here:
http://www.novell.com/communities/node/9298/patching-and-upgrading-oes2-sp1-oes2-sp2
…all due credit (and thanks) to the author for putting that together!!  Please read that doc first to get a feel for what I’m after here, and take in all the warnings and caveats.  Maybe even post a comment, and thank the author…

I don’t know about you, but the last thing I want to have to do during a server OS upgrade is drive all around the state to go hit “enter” on a console or something.  If I can’t do it remotely, I don’t want to do it!  So my intention here is just to document a possible remote upgrade method, clean up some typos, and so forth.  Please note that I have followed this procedure a good handful of times, and have had great success with it so far.  If you follow this procedure, make sure to drop a comment and let me know how it went.

And for heaven’s sake, if you don’t know what you’re doing, don’t do this!  Read it all through first, maybe test on a demo machine, etc…  Also, I hope it goes without saying that you’ll probably want to make use of the “screen” utility to do all this work…  More on screen another time…

Anyway, back to the task at hand…

First, patch your OES2-SP1 system completely up-to-date with this:

rug up -t patch -g security -g recommended OES2-SP1-Updates SLES10-SP2-Updates

…but watch out for being prompted for the move-to-sles10-sp3 issue !!  Reboot as needed.

Now a quick aside or two, right in the midst of things… If you have a crazy, complicated admin password, change it temporarily for this process.  Set it to “changeme”, or “temp123″, or anything without special characters.  You’ll thank me for it later, and it will work better when you create the “answer” file.  Also, please remember that if you have an SMT server, make sure to prepare the new channels you’ll need.  Oh, and if you have custom installation sources (like and ftp server or webserver with ISOs mounted on it or something), they’ll be useless in a few minutes anyway, so you may want to delete them now…

Then, assuming all patches are up to date, your server codes are valid, etc., etc., the first step of the process is to get the “update” patch in place:

rug in -t patch move-to-oes2-sp2

…you should not need to reboot here.

Now the box is mid-way between versions.  We can see that it needs its versioning and control packages (those that define what version it is to be) with this command:

rug lu -t patch

So now we need to continue to turn it into an extremely un-patched OES2sp2 server, with this command:

rug up -t patch -g recommended

…and DO NOT REBOOT when prompted!  Because if you reboot now, you will need to physically be at the console screen to enter the eDirectory tree Admin password when prompted.  We don’t want that (unless you have HP Lights-out boards or something).  So, in order to create an answer file to automatically stuff the the Admin password in (remember, you changed it to something simple, right?), do the following:  In a shell session, as root,

cd /root

…to ensure the answer file is away from other users, since it will be created in your working directory.  Now create the answer file like this:

yast create-answer-file [adminpassword]

…of course, replacing “[adminpassword]” with your eDirectory Admin password.  Unfortunately, you must enter the password on the command line, then it re-displays it to you, so:

1. make sure there are no eyes on the monitor other than yours while doing this, and
2. I strongly recommend that when you finish this session, you log back in and vi /root/.bash_history and remove the line with the admin password (so other users don’t see it later!)

Now, copy it to where it needs to be:

cp answer /opt/novell/oes-install/

Now reboot:

init 6

When the server is back up and running, it will complete the update of all the OES-related services; this takes some time.  Once that’s done, you need to do a final round of patching with our brand-new updated channels:

rug up -t patch -g security -g recommended OES2-SP2-Updates SLES10-SP3-Updates

…and reboot if needed.  Oh, and some cleanup…  Delete the temp answer file:

rm /opt/novell/oes-install/answer

…And don’t forget to remove the answer file password line from your /root/.bash_history…  And change your Admin password again…

And as a final step, clean up your installation sources (yast inst_source) if you had custom ones. And you’ll also need to re-add your “Add-On” source for the OES software;  it’s a requirement.  So one way to do it would be to run “yast add-on” , and walk through the steps of adding the URL to the source, ensuring all the patterns are listed (they should all be set to “Reconfigure is Disabled”).

And that’s it!  If all went well, you should now have a fully upgraded server.  Enjoy!

On May 4th of this year, we at Cavalier had a visit from a group of students from Lapeer County Intermediate School District, specializing in networking technology.  They had come for a tour of our technology, and related discussions.

Now I know that the first image in your mind when you read that is a blend of disaffected/bored and hyper/reckless teens running through the hall, or lighting stuff on fire, etc., right?  Wrong.  These students were well-behaved and attentive, and more importantly, inquisitive.  Of course, the bummer of the visit was that it was very brief (we *do* try to earn a living here );  and even though they were watched and guarded closely, we could not take them to where the really cool stuff is, due to security concerns.

Unfortunately, I could not attend the entire mini-tour since I was busy on other things.  But I managed to join the group at the end when summaries were being given and questions were being taken.  And to me, that was probably the coolest part anyway;  because once the students relaxed a bit and starting asking questions, we all started to get the sense that this group was really absorbing what they saw, and was truly interested in what they are doing.

They asked questions that ran the gamut; about the validity of technical certifications, resume writing, the prevalence of older technology still in the workplace as tech generations change, honesty, human networking, volunteering, and so on.  The small group of Cavalier employees clearly got a kick out of it, and it was over all-too-soon.

After they all left, I was left with a strange feeling…  I clearly did not realize how much I would enjoy interacting with students that are interested, and I was left thinking I’d like to do it more.  Hmm.  Maybe volunteer to be a guest lecturer in school technical classes?  Maybe get involved in community technology groups?  Hmm.  Hmm……

I want to thank Kelly R. and Terri W.  for bringing the kids… er… young adults… out for the visit.  And I want to thank the students for interacting, and making it fun;  remember gang, we all started somewhere, and you’ll all be well on your way if you apply focus and diligence.

It was a kick, and I really enjoyed it.

If you have an iFolder 3.8 (and some previous versions) server, and you use the local database (instead of LDAP or what-not), then you may have run into a situation where you have to change a password for a regular user when that current password is not known.

Of course, if the user *knows* his or her own password, he or she can use the “settings” link in the iFolder Web Access page, or the “Security” menu item in the client interface.

But if the password is UNknown, then you really have no choice but to do it on the command line of the iFolder server.  The problem with that, however, is that in order to do that on the command line, you’ll need to enter both the admin credentials and the user’s new credentials as well… which is never a good idea to do, since that will sit in the command history by default, etc.  Besides, who wants to memorize that crazy string anyway?

So, here’s a tiny little favor for you…  I stuffed it all into a tiny little helper script that you can have.  Just paste these contents into a script, do a chmod +x to make it executable, and away you go.  it will prompt you for admin password, username to change, and the new password for that user.

I hope it helps!  Here you go…

#!/bin/bash
clear
echo ""
echo "This is the user password change tool for iF3..."
echo ""
echo "Please enter the admin password: "
echo ""
read ADMINPW
clear
echo ""
echo "Please enter the username for which you are changing the password: "
echo ""
read USERNAME
clear
echo ""
echo "Please enter the new password for $USERNAME (careful with crazy special characters): "
echo ""
read USERPW
clear
echo ""
echo "Processing..."
echo ""
# For 32 bit...
/usr/bin/mono /usr/lib/simias/bin/UserCmd.exe setpwd --url http://localhost --admin-name admin --admin-password $ADMINPW --user $USERNAME --password $USERPW
# For 64 bit...
#/usr/bin/mono /usr/lib64/simias/bin/UserCmd.exe setpwd --url http://localhost --admin-name admin --admin-password $ADMINPW --user $USERNAME --password $USERPW
echo ""
# I know the exit codes from mono are of no real value, but oh well...
if [ "$?" -eq "0" ];then
echo "If the exit message reads: \"Failed - Invalid admin credentials\", the password for $USERNAME was not changed. "
echo "If the exit message reads: \"SetPassord for $USERNAME - False\", then $USERNAME might not exist in the system. "
echo "If the exit message reads: \"SetPassord for $USERNAME - True\", then the password for $USERNAME is now changed!"
else
echo "...The script encountered a problem! Exiting..."
exit 0
fi
echo ""
echo "...Done!"

I really wanted to get the LPI-3 certification (note the past tense).  It really looked like fun;  it’s got a whole bunch of hard-core stuff about OpenLDAP and such, which I love and I’m way in to.  Unfortunately, a great disincentive has appeared.  First, some background…

I achieved the LPI Level 1 certification on 2004-01-30.

When I took the LPI-1 tests back in early 2004, they had a promise of perpetual certification.  They were the only certification I had heard of doing that.  Perhaps they later thought better of it, because they eventually removed that standard, and instead set time limits on all new certifications, and imposed an expiry (of 5 years after last certification) on all existing ones.  Of course, I do not begrudge them any of those later actions;  it is arguable that the highly relaxed initial standard was more than a bit short-sighted, and the result of an eager new organization.

I achieved the LPI Level 2 on 2005-04-09.

The LPI-3 didn’t exist back when I got the LPI-2…  back when I had lots of time on my hands.  When the LPI-3 was formulated, I watched it evolve, monitoring the news when I could, trying to hatch a plan when I could go get it.  It was released to the public in 2007.  I had heard the news that they were going to remove the “lifetime” certification, but then time just went by too fast for me to prepare for the LPI-3.  Life happens.  I’m not looking for sympathy, my life is great;  busy, and great…

During that window, it would not have made logical sense (in fact, it seems wasteful) for me to have re-taken one of the tests for an LPI level for which I was already certified in order to re-certify.  I figured there would be plenty of time.  But as the window was closing, I looked much more closely into the requirements, and realized I was un-prepared to take the LPI-3 test.  And due to the limited re-take policy, my limited resources, and the fact that I’m not the type to do things un-prepared, I declined to take the LPI-3 test(s) as the last days of my window passed.

My LPI-1 and LPI-2 certifications have been moved to an “inactive” state on 2010-04-09.

This is because of that aforementioned expiry, and the fact that I have taken not taken any LPI test (to re-certify) since I achieved my LPI-2 certification in 2005.  Again, I can’t blame them a bit.  It’s been a while, and they would like to maintain current, fresh standards.

But here’s the problem:  By rule (and yes, I confirmed with an LPI official), because I didn’t get completely certified as an LPI-3 before the threatened expiry, in order to achieve the the LPI-3 at this point I must re-certify at ALL LEVELS of the LPI.  This means that in order to be permitted to take and pass the tests for the LPI-3, I must re-take and re-pass all the tests necessary to re-acquire the LPI-1 (two passed tests) and the LPI-2 (two more passed tests), just to arrive right back where I was before 2010-04-09.

There is no tiered “grandfather” type clause.  There’s no “refresher”.  There is no “catch-up” or “make-up”.  Apparently, there is no sympathy for the additional effort, just time from the last certification…

As a result, it seems to me that the more time and money you invest in the LPI, the more you stand to lose in the event of expiry.

That’s not just dis-incentivization, it’s a smack-down.

Now I think I’m glad I didn’t get the LPI-3, because since there’s no LPI-4 now, my LPI-3 would be expiring shortly anyway, then I’d just have to do it all over again if they ever did make the LPI-4…

8|

I got this question the other day from a Centos administrator: “The certwatch tool has been sending the me an email warning me that I need to renew the SSL Certificate. What do I do?”

The email message read like the following (names have been changed):

################# SSL Certificate Warning ################

Certificate for hostname 'www.yourlinuxguy.com', in file:
/etc/pki/tls/certs/www.yourlinuxguy.com.cert

The certificate needs to be renewed; this can be done
using the 'genkey' program.

Browsers will not be able to correctly connect to this
web site using SSL until the certificate is renewed.

##########################################################
Generated by certwatch(1)

The administrator had searched the Internet, and found mostly impolite messages from youngsters about reading the docs. Accurate, perhaps, but not too helpful.  This person’s server was not using a public Certificate Authority (CA); just a local one on the box itself (as is common practice in development environments).  If you don’t understand all that, it will make more sense as we go on…

First I want to clarify a couple things.  When you run the genkey tool, you are actually doing what the name suggests;  generating a new key pair (public/private), from which the certificate is formulated.  Technically, you are not *renewing* the certificate as the certwatch warning message implies, but that’s okay (it *is* possible in some situations to “renew” a certificate based on an existing key pair, but that’s not important right now). The genkey tool makes it so easy and convenient that it is just easier this way. Remember that a certificate is nothing but a public key that is “stamped” with approval by a CA.  In this case, the CA is you, too.  Not exactly a trusted hierarchy, but there you go…

So to cut to the chase, it’s really easy.  Here’s basically what you need to do:  1.) run the tool, and  2.) validate your SSL settings in your httpd.conf.

Run The Tool

At this point, you may or may not want to make a backup of your cert files…  This is up to you…  You can do that with something like this command (of course, your mileage may vary):

cp -av /etc/pki/tls /etc/pki/tls.bak

If you just run the genkey tool without specifying the certificate lifetime, it defaults to something like 30 days.  Let’s try something a little longer; like 4 years.  Now just specify your hostname on the command line:

genkey --days 1460 www.yourlinuxguy.com

…this will launch an interactive tool to do things like generate the random data, make the key pair, and walk you through specifying the content of the certificate (Country, Location, etc.).  It will place the new stuff in some default location, and at the end of the process, tell you where it all is.  You should take note of the location, but it will likely be what I mentioned above for the backup.

Validate Your SSL Settings

Now, theoretically, you should be able to restart your Apache daemon.  However, you might have used custom names or locations for your certificate files in the past, so you might want to check to be sure they match the SSL settings in your Apache config files.  Of course it’s hard for me to tell you where those settings are, since it’s so easy to customize Apache; but here’s a good way to find the two most important values:

cd /etc/httpd
egrep -R -e "SSLCertificateKeyFile" -e "SSLCertificateFile" *

…and that will likely return results from a file called “ssl.conf” or something like that.  Edit the config file if necessary; just make sure the values match the place that the genkey tool placed the new private key and cert file, and you should be good.

Now, you can restart Apache…

/etc/init.d/httpd restart

…and you’re done!  I hope that helps…

Here’s a fun one for you.  I had a situation recently where the customer had some of his windows PCs properly being populated in his reverse DNS zone (in an AD-controlled environment), and some that were not.  He asked me to quickly scan the network and figure out which were which (before we started to look into the “why” part).  Note that for the examples that follow, the customer’s fictitious network address is good ol’ 192.168.1.0/24, and the commands were tested with the default packages on Opensuse 11.1 and 11.2.

So first, I had to find those that are in the reverse DNS zone… So I simply made sure my PC points to the correct internal DNS server that is authoritative for the forward and reverse DNS zones, and then ran a sweep with some filtering:

nmap -sP 192.168.1.0/24 |grep -v "^Host 192\.168\.1"|grep "^Host"|awk '{ print $2,$3 }'

Then, in order to get the local netbios name for Windows computers that aren’t in reverse dns , I came up with this:

for item in `nmap -sP 192.168.1.0/24|awk '{ print $2 }'|grep "^192\.168\.1"`;do thingy=`nmblookup -A $item|grep "<20>"|awk '{ print $1 }'`;echo $thingy \($item\);done

…of course, non-Windows computers and devices without netbios will simply not respond to the query. It might be neater to dump the output to file and such if you wish, but I didn’t bother. I’m just keeping it simple here.

Note that the above does not take in to account if there are multiple reverse DNS entries per PC. If you need to know that, you could do something like this:

for item in `nmap -sP 192.168.1.0/24 |grep -v "^Host 192\.168\.1" |grep "^Host"|awk '{ print $3 }'|sed s/[\(,\)]//g`; do nslookup $item|grep "name =";echo "";done

…and so on.  You could really go crazy with this sort of thing… It can also be handy if you are in an environment where you don’t have access to the DNS/DHCP administration tools, but want to see what’s out there…. Now wasn’t that fun?

Here’s a quick cheat-sheet on how to do fast backup of your eDirectory tree from an OES Linux server.  While we’re at it, we’ll do a quick restore of an object.

First, let’s back it up!  Let’s grab the whole tree:

ndsbackup cvf /tmp/ndsbackup-treeRoot.bak [Root]

…note that you will be prompted for the admin user credentials, and by specifying the “[Root]” object, it will grab every single object in the tree and put it in the specified file.

Now, let’s get a little fancier:

ndsbackup cvf /tmp/ndsbackup.`date +%Y%m%d%H%M`.bak -a admin.corp [Root]

…here we are adding a date stamp to the output file, and giving it the admin username.  It will still prompt for the admin password, of course.  There are tricks to handle that, too, but this is good enough to get the point across.

If you want to see what’s in your backup file:

ndsbackup tf /tmp/ndsbackup.201003051421.bak

Make sure to copy that backup file off the server somewhere, somehow.  In reality, you might actually be writing the result directly to an nfs or samba share or something anyway, but the point is to get the data off the box for safety.

So let’s imagine a week goes by, and whoops… you deleted someone accidentally.  Don’t worry!  Because all you have to do to restore that user object is to put the file back on the filesystem where you can reach it, and do something like this…

To restore just that user you deleted:

ndsbackup wxvf /tmp/ndsbackup.201003051421.bak -a admin.corp cn=duser.ou=Users.o=corp

Or, plow in the whole container if you accidentally wiped it all out:

ndsbackup wxvf /tmp/ndsbackup.201003051421.bak -a admin.corp ou=Users.o=corp

…If you wish, you can remove the “w” from the above restore examples if you don’t want to be prompted to confirm for each object.  However, I tend to think the “w” is a good idea…

That’s all for now.  Of course, there are tons more options for ndsbackup if you look in the man page, but I’m not going in to that, because after all, this *is* a cheat sheet…