Replace a failed Hard Drive in a Solaris Box

cfgadm is your friend when you need to replace some hardware.

cfgadm -al   will show you the state of the various pieces of hardware

cfgadm -c configure sata2/6     will reconfigure the disk once you replace it

Ruby on Rails Application

I am looking at the PowerDns on Rails front end for a Power DNS backend. The PowerDNS on Rails requires Ruby on Rails so I started looking into how to get it running on CentOS 5.3.

After downloading and installing the app, I tested it via running:

ruby script/server

which starts up a small web browser on port 3000.

I also had to delete the index.html file so the application was accessible from the root of the webserver and all the links worked.

Once it was tested, I wanted to make the installation more robust and permanent. I found this great tutorial to help wrap Mongrel Cluster and Apache around the app.

Solaris Verbose Booting

When you are sitting around waiting on Solaris to boot it is helpful to know what it is doing instead of staring at a blinking cursor.

I found it helpful to turn on some verbose debugging options during boot. I also added the -k for the debugger which takes almost no resources and can be helpful when there is a hang.


You can either add these to /boot/grub/menu.1st, or edit the grub entry when it boots.

Here is the new multiboot line:

Serial:
kernel /platform/i86pc/multiboot -B console=ttya -k -v -m verbose

VGA/Text:
kernel /platform/i86pc/multiboot -B console=text -k -v -m verbose

In 10u6 it displays a nice countdown of ZFS filesystems getting mounted which is a HUGE win!

Be ready though, it will throw a lot of information at you, in particular, it will dump a lot of information about each disk.

A new game

So, we are sitting here waiting for a solaris box to finish booting and decided to come up with a game.  What kinds of things can you do while waiting for Solaris to boot:

• Update your resume.


• Come up with more reasons not to stick with Solaris.


• Re-write ZFS.


• Write a new filesystem.


• Ponder "Why".


• Come up with ways to torture people that ask "How much longer?".


• Come up with ways to torture people that ask "So why are things set up that way?".


• Read War and Peace.


• Re-Read War and Peace.


• Go jogging. Around the city.


• Take up painting, with one hair at a time.


• Count how many times you can hit yourself in the head with a hammer before you pass out.


• Take up a new hobby, like knitting, starting with raising the sheep.


• Count how many dots are in the ceiling tiles, with your eyes closed.


• Come up with new things the Mythbusters can prove or disprove.

Sun x4500 and Harddrive firmware patches

If you have a x4500 ( Thumper ) with the 500g disks, make sure you install patch number 140576-03.

140576-03 sata firmware patch for 500gb Hitachi Drives

It only affects the following drives:

Disk Drive HDS725050KLA360, 500.0GB (Sun model number HDS7250SASUN500G) without disk firmware revision AV0A

The patch requires a reboot where it will go through disk by disk and install the firmware. With 48 disks, expect it to take a while.


Also install the latest Solaris patches and firmware for the x4500 which is 1.6 as of today.

Sun x4500 and firmware

I ran across a number of issues with 10u6 and the x4500 ( Thumper ) especially with their Marvell drivers. In particular you have to make sure you are running the latest and greatest firmware and have all of the latest kernel and driver patches. When you install the latest firmware Sun recommends doing a clean install of Solaris so that the disks get numbered correctly. What you will find is that c3's can become c5's and c5's can become c3's. It only happens on the "boot drives" in the x4500's. I did a Live Upgrade flar install with the new firmware and Solaris figured out the drive ordering with no problems.

Solaris Upgrade using Live Upgrade

One of the neatest things I have found about Solaris is their Live Upgrade package. In a nutshell, you can upgrade one disk while leaving your original environment untouched and then boot into the upgraded OS. If it doesn't work, simply reboot into your original install.

There are a few key things to keep in mind when you are doing this.

• You need two disks.


• Live Upgrade can auto magically break meta devices, etc.


• You will need the CD or ISO if you want to do a full install.


• You can use a flar for an install.

NOTE: I haven't played with ZFS roots yet but it is on my short list. Everything below is based on UFS.

Here are the steps I used to do a new install of 10u6 on the first disk {c5t0d0 } of an x4500 with no meta devices and 10u5 on the second disk { c5t4do }.

First, lets set up the disk partitions. You can either manually do them via format, or like in my case, I just want to copy the layout from the first disk:

$ prtvtoc /dev/rdsk/c5t4d0s2 | fmthard -s - /dev/rdsk/c5t0d0s2

Let's mount up the DVD ISO for 10u6 so we can install the Live Upgrade packages and use it for the flar install later:

$ mkdir /mnt/cdrom
$ lofiadm -a /export/home/sol-10-u6-ga1-x86-dvd.iso
$ mount -F hsfs -o ro /dev/lofi/1 /mnt/cdrom


Now we can install the Live Upgrade Packages:

$ cd /mnt/cdrom/Solaris_10/Product
$ pkgadd -d . SUNWlur SUNWluu


You have to tell Live Upgrade to set up the environment. Here I have told it that the name of my new environment is 10u6 ( you can name your original with the -c option ), I've told it to make a ufs / on c5t0d0s0, a swap partition on c5t0d0s1, a ufs /var on c5r0d0s3, and a ufs /export/home on c5t0d0s7. Take a look at the man page for lucreate for more options and settings.

$ lucreate -s - -n 10u6 -m /:/dev/dsk/c5t0d0s0:ufs -m -:/dev/dsk/c5t0d0s1:swap -m /var:/dev/dsk/c5t0d0s3:ufs -m /export/home:/dev/dsk/c5t0d0s7:ufs


Oops, the lucreate failed. What I realized was that I had an old 10u3 environment still in play.
lustatus showed it:

$ lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
10u3 yes no no yes -
10u5 yes yes yes no -

So now I want to delete the 10u3 since it is on the disk that I want to install 10u6 on.
$ ludelete 10u3

Well, that didn't work either. It is hung up on not being able to find the old metadevices which don't exist any more. Live Upgrade is rather picky about how things are working and if anything changes without it knowing about it things go bad fast.

So, now let's start cleaning things up manually:

Edit /etc/lutab
I deleted the 10u3 lines and then updated the 10u5 lines with a 1 at the beginning instead of a 2.

The ICF files show the disk partitions for an LU environment. I wanted to get rid of 10u3 and make 10u5 the first and only entry.
$ mv /etc/lu/ICF.2 /etc/lu/ICF.1

Edit /etc/lu/BRUB_backup_menu
Remove 10u3 section

Let's delete, or move, the Grub slice file
$ mv /etc/lu/GRUB_slice /etc/lu/GRUB_slice.bak

Now that all of the 10u3 things are cleared, try the lucreate again:

$ lucreate -s - -n 10u6 -m /:/dev/dsk/c5t0d0s0:ufs -m -:/dev/dsk/c5t0d0s1:swap -m /var:/dev/dsk/c5t0d0s3:ufs -m /export/home:/dev/dsk/c5t0d0s7:ufs

And all went well.

Sometimes it will complain and ask you to run lux86menu_propagate
Since you already have the ISO mounted, it is easy:

$ /usr/lib/lu/lux86menu_propagate /mnt/cdrom

And finally, run the luupgrade command to build your new environment.
Here I am building an empty shell and using my 10u6 flar to install. You can also tell it to make a copy of your existing environment which you can later patch with lupatch

$ luupgrade -f -n 10u6 -s /mnt/cdrom -a { path to flar }

Time to go get a drink, read a book, etc because it can take quite a while.

Here are a few things you can use when your new environment is built:
lustatus
lumount ( Mount up the new environment so you can make changes to it before you reboot. )

When you are ready to boot into your new environment, run luactivate:

$ luactivate 10u6

It will then tell you how to recover should you need it and some other info about rebooting, etc. Make sure you read it.

When you reboot, at least on the x86 platform, you will have the new entries in Grub and your new environment should be the default.

Enjoy!

DNS and Bind

I keep running into this problem where people want static DHCP assignments, and their own hostname, but don't want to wait on OPS to give it to them.  So, one would think that ddns is perfect for this, and in a way I agree, but it seems to cause some issues with caching, the jnl files, and hand editing the zone files. 

So, here are a few commands I found that help alleviate some of the issues:

Hand editing the zone file:

$ rndc freeze zonefile
Edit the zone file making sure to update the serial number
$ rndc thaw zonefile
$ rndc reload

I'm looking at moving to a powerdns setup with a mysql backend, but I'm still looking for a nice web interface to handle both DNS and DHCP entries that will allow regular users to request a hostname/IP given their MAC.

Tags: bind, dhcp, ddns, linux, solaris

Ad's

I'm not a big fan at all of ads, but hey, in this day and age, what can a little extra income hurt?  Might be enough to take the wife out to eat.  So, happy clicking and I'll tell my wife I'm counting on you!

First Post

Everyone has to have a first post, and I guess it sets the tone, but I am just using it to get the ball rolling. My plan is to take random ideas that I have, both complete and incomplete, and get them somewhere. I guess a blog is better than nothing. So, here goes, if there is anyone out there, enjoy.