{"id":470,"date":"2010-01-26T15:01:11","date_gmt":"2010-01-26T20:01:11","guid":{"rendered":"http:\/\/yourlinuxguy.com\/?p=470"},"modified":"2010-01-28T07:37:29","modified_gmt":"2010-01-28T12:37:29","slug":"how-i-fixed-my-openfiler-2-3-server-after-updates-broke-it","status":"publish","type":"post","link":"https:\/\/yourLinuxGuy.com\/?p=470","title":{"rendered":"How I fixed my Openfiler 2.3 server, after updates broke it…"},"content":{"rendered":"

This article is written in a story-like retrospective fashion.\u00a0 (Update, 20100128, morning: Now with a happy ending! See below…<\/span>)\u00a0 It’s a chain of events that took me from a happy Openfiler user, to and extremely angry Openfiler user, to a cautious Openfiler user.\u00a0 So let’s begin.\u00a0 “It was a dark and stormy night…”<\/em><\/p>\n

It had been almost a year since I patched my Openfiler 2.3 server, and some significant updates had been released in that time.\u00a0 So after planning an outage window, I set out to apply the queued batch of updates (including mkinitrd<\/code> and a kernel<\/code> or two, more on that later).\u00a0 It seemed such a simple task….<\/p>\n

After the patches were applied, the Openfiler server would not boot.\u00a0 Instead, it spewed out more errors than I could read to the console, ending with a kernel panic.<\/p>\n

So I booted to the Openfiler install cdrom, and entered rescue mode with “linux rescue<\/code>“.\u00a0 Fortunately, it discovered and mounted all my partitions, and I was able to run with networking in order to reach the package sources or other sources\/destinations if I should need them.<\/p>\n

I noticed that the errors I had seen on the console pointed me toward missing drivers in the initrd<\/code>, so I started looking there.\u00a0 While poking around, I think I noticed that the latest update bundle had re-compiled all my initrd<\/code>‘s in \/boot<\/code>. I’m not actually 100% sure about that now in hindsight, but it’s not important now.\u00a0 Anyway, I de-constructed one of the initrd<\/code>‘s this way to see what was up:<\/p>\n

<\/code><\/p>\n

mkdir \/tmp\/temp\r\ncd \/tmp\/temp\r\ncp \/boot\/initrd-2.6.29.6-0.15.smp.gcc3.4.x86_64.img initrd-2.6.29.6-0.15.smp.gcc3.4.x86_64.img.gz\r\ngunzip initrd-2.6.29.6-0.15.smp.gcc3.4.x86_64.img.gz\r\ncpio -i --make-directories < initrd-2.6.29.6-0.15.smp.gcc3.4.x86_64.img\r\nls -l\r\nls -l lib\/<\/pre>\n
Hmm.\u00a0 No drivers there.\u00a0 Should be a bunch of *.ko<\/code> files…\u00a0 Hmm…<\/p>\n
I decided to roll back;\u00a0 so I ran conary<\/code> to revert to the state before the update\u00a0 (Run “conary rblist<\/code>” to figure out the number of the point-in-time to which to roll back, then “conary rollback (number)<\/code>” to actually do it.\u00a0 See this wiki page<\/a> for details).<\/p>\n
For some reason, this did not repair all my initrd<\/code>‘s, but it did seem to give me a working mkinitrd<\/code> script (in hindsight, I did not think to analyze it or keep a copy of it… drat).\u00a0 So I was now able to hand-build a working initrd<\/code> for one of the kernels<\/code> (the most recent before the update) with this command:<\/p>\n
 <\/code><\/p>\n
mv \/boot\/initrd-2.6.29.3-0.3.smp.gcc3.4.x86_64.img initrd-2.6.29.3-0.3.smp.gcc3.4.x86_64.img.bad\r\nmkinitrd \/boot\/initrd-2.6.29.3-0.3.smp.gcc3.4.x86_64.img 2.6.29.3-0.3.smp.gcc3.4.x86_64<\/pre>\n
And now, I was able to reboot into this kernel<\/code> with my new working initrd<\/code>…. and this time I made backups of it in my \/home<\/code> directory!<\/p>\n
So I proceeded to re-apply the patches bit by bit.\u00a0 I quickly realized that the dreaded kernel<\/code> update was a requirement as part of the major Openfiler update, so it came back with the re-run of the last few updates anyway.\u00a0 But this time, I was ready.\u00a0 I modified the \/boot\/grub\/menu.lst<\/code> to use my old trusty kernel<\/code> and initrd<\/code>, (default=1<\/code>), and make sure my hand-made initrd<\/code> was in place.\u00a0 Ok, good.\u00a0 Reboot.\u00a0 Right.<\/p>\n
Back up in my older kernel<\/code> (phew!), but with my newer kernel<\/code> installed, I troubleshot…\u00a0 I tried to build my own initrd<\/code> against the new kernel<\/code>, but I kept getting errors like this for each needed module:<\/p>\n
 <\/code><\/p>\n
\/usr\/bin\/strip: \/lib\/modules\/2.6.29.6-0.15.smp.gcc3.4.x86_64\/.\/kernel\/drivers\/rtc\/rtc-lib.ko: File format not recognized<\/pre>\n
I dove in to the \/sbin\/mkinitrd<\/code> and found the cause.\u00a0 There are some lines in an “if<\/code>” block that optionally call “strip<\/code>” if present, and that is where it seems to err out.\u00a0 If “strip<\/code>” is not present, it just copies the module, like I want.\u00a0 Hmm.\u00a0 I don’t know if strip<\/code> was there before or not, or if the mkinitrd<\/code> suddenly has this new “if<\/code>” block, and I don’t care.\u00a0 I commented it out to look like this:<\/p>\n
 <\/code><\/p>\n
for MODULE in $MODULES; do\r\n#\u00a0\u00a0\u00a0 if [ -x \/usr\/bin\/strip ]; then\r\n#\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \/usr\/bin\/strip -g $verbose \/lib\/modules\/$kernel\/$MODULE -o $MNTIMAGE\/lib\/$(basename $MODULE)\r\n#\u00a0\u00a0\u00a0 else\r\ncp $verbose -a \/lib\/modules\/$kernel\/$MODULE $MNTIMAGE\/lib\r\n#\u00a0\u00a0\u00a0 fi\r\ndone<\/pre>\n
…and re-built my initrd<\/code> for the latest kernel like this:<\/p>\n
 <\/code><\/p>\n
mv \/boot\/initrd-2.6.29.6-0.15.smp.gcc3.4.x86_64.img initrd-2.6.29.6-0.15.smp.gcc3.4.x86_64.img.bad\r\nmkinitrd -f -v \/boot\/initrd-2.6.29.6-0.15.smp.gcc3.4.x86_64.img 2.6.29.6-0.15.smp.gcc3.4.x86_64<\/pre>\n
…and no more errors!\u00a0 Yay!\u00a0 I modified the \/boot\/grub\/menu.lst<\/code> back to use my new kernel<\/code> and initrd<\/code>, (default=0<\/code>), and I re-booted into it with no kernel panic! Yay!<\/p>\n
And that’s my story.\u00a0 I really hope this helps someone out there…<\/p>\n
UPDATE, 20100127, evening<\/span>:\u00a0 On the Openfiler forums<\/a>, there is talk of a fix.\u00a0 To quote a Rafiu, “This was due to a strange situation where 64-bit version of binutils was  not built and the group update succeeded regardless.<\/em><\/p>\n
We have  now resolved this issue. Apologies to all that were affected by it. “<\/em><\/p>\n
I have not yet tested this, but I do see that the latest binutils is in the update list now.\u00a0 More to come…<\/p>\n
UPDATE, 20100128, morning<\/span>:\u00a0 Yes.\u00a0 The latest set of updates at this time, *including* the correct set of binutils (2.17.50.0.6-7-0.0.2) for 64-bit, worked perfectly, and my server is still running as smoothly as ever after rebooting.\u00a0 Hooray.\u00a0 Though I will never get that Saturday night back that I spent chasing my initrd all over the place, but there you go…<\/p>\n
\ud83d\ude09<\/p>\n","protected":false},"excerpt":{"rendered":"
This article is written in a story-like retrospective fashion.\u00a0 (Update, 20100128, morning: Now with a happy ending! See below…)\u00a0 It’s a chain of events that took me from a happy Openfiler user, to and extremely angry Openfiler user, to a…
Read more<\/a><\/p>\n","protected":false},"author":5,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[12,88,87],"tags":[],"class_list":["post-470","post","type-post","status-publish","format-standard","hentry","category-advanced","category-mkinitrd","category-openfiler"],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/pnjn1-7A","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/yourLinuxGuy.com\/index.php?rest_route=\/wp\/v2\/posts\/470","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/yourLinuxGuy.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/yourLinuxGuy.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/yourLinuxGuy.com\/index.php?rest_route=\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/yourLinuxGuy.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=470"}],"version-history":[{"count":7,"href":"https:\/\/yourLinuxGuy.com\/index.php?rest_route=\/wp\/v2\/posts\/470\/revisions"}],"predecessor-version":[{"id":474,"href":"https:\/\/yourLinuxGuy.com\/index.php?rest_route=\/wp\/v2\/posts\/470\/revisions\/474"}],"wp:attachment":[{"href":"https:\/\/yourLinuxGuy.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=470"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/yourLinuxGuy.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=470"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/yourLinuxGuy.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=470"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}