[
"joy, it has its own justice
and dreams are languid and lawless
and everything bows to beauty
when it is fierce
and when it is flawless"

- Ani Difranco, Fierce Flawless
]

truk Rambles - Read truk's Blog! RSS Feed


Breaking a Linux Software RAID 1 for Import using VMware Converter
2014-08-12 10:25:21

I rarely post super-geeky stuff on here, but since I couldn't find any good instructions to do this important and critical activity on the Internet, and my co-workers had to piece together a set of steps that worked for us, I wanted to share what we learned, hoping to save the next person all of the work.

Linux has supported the idea of software RAID, particularly RAID 1 (or disk mirroring) for a long time. Disk mirroring is a great way to gain some insurance against a single disk failure bringing down a critical system, as everything rewritten to one disk is also written to the other disk. Many servers use hardware RAID, which mirrors the disks at a lower level than the operating system can see, making it easier to gain this redundancy. However, hardware RAID has always been more expensive than software RAID, and so there are quite a few servers out there using software RAID to protect their data.

This week, we had one of our last physical (non-virtual) server (RedHat Linux 4 AS) that needed to be virtualized. Due to the size of the data stored on that system and how it uses an external disk array, it was important that we virtualize it in place, using excellent VMware's Converter Standalone to import the running machine, so that there was no downtime while importing the data. However, the Converter Standalone will not import Linux systems using software RAID, due to problems accessing the underlying data structures of the disk through the metadevices presented by the software RAID. (You know you are having this problem when the Converter complains about not being able to access the /boot partition.) The best solution was to break the mirrored software RAID and boot the system off of one disk, so that all of the necessary partitions could be imported and the system could be virtualized.

Unfortunately, as important and seemingly common as breaking a mirrored software RAID is in Linux, I couldn't find any good, comprehensive, working instructions on how to do it, and breaking a software RAID is a tricky business. It is very, very easy to end up with a non-booting system and no easy way to repair it. So, to help out the next person that runs into it, I'm posting the steps that we did to break the mirrored software RAID and set the system to boot off of only one disk, so that VMware's Converter Standalone would work on it.

First, get an idea of what you are dealing with. Logged in as root, inspect the system:
[root@cr2 cr]# df -h

Filesystem Size Used Avail Use% Mounted on
/dev/md2 456G 45G 389G 11% /
/dev/md0 487M 35M 427M 8% /boot
none 4.0G 0 4.0G 0% /dev/shm
/dev/sda1 2.0T 560G 1.4T 30% /archive

[root@cr2 cr]# more /etc/fstab

# This file is edited by fstab-sync - see 'man fstab-sync' for details
/dev/md2 / ext3 defaults 1 1
/dev/md0 /boot ext3 defaults 1 2
none /dev/pts devpts gid=5,mode=620 0 0
none /dev/shm tmpfs defaults 0 0
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0
/dev/md1 swap swap defaults 0 0
/dev/sda1 /archive ext3 defaults 1 0
/dev/hda /media/cdrecorder auto pamconsole,exec,noauto,managed 0 0

[root@cr2 cr]# more /etc/mtab

/dev/md2 / ext3 rw 0 0
none /proc proc rw 0 0
none /sys sysfs rw 0 0
none /dev/pts devpts rw,gid=5,mode=620 0 0
usbfs /proc/bus/usb usbfs rw 0 0
/dev/md0 /boot ext3 rw 0 0
none /dev/shm tmpfs rw 0 0
/dev/sda1 /archive ext3 rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0

In this case, we have 3 metadevices (md0 as /boot, md1 as swap, and md2 as root /). You can get further details about your RAID configuration using the mdadm tool, as well as mdstat:
[root@cr2 cr]# cat /proc/mdstat

Personalities : [raid1]
md1 : active raid1 sdc2[1] sdb2[0]
2200832 blocks [2/2] [UU]
md2 : active raid1 sdc3[1] sdb3[0]
485668928 blocks [2/2] [UU]
md0 : active raid1 sdc1[1] sdb1[0]
513984 blocks [2/2] [UU]
unused devices: <none>

You can also run these commands:
cat /proc/mdstat

mdadm --detail /dev/md2     (to inquire about the disk members of the /dev/md2 metadevice)

When you are finally ready to do this, make sure you have a full backup of your system, and then stop all running services, especially any that would possibly write data, such as databases.

Next, we need to use mdadm to set one of the disks (we chose /dev/sdc) to be set as "failed" and removed from the RAID array. To do this, we ran these commands:
mdadm --fail /dev/md2 /dev/sdc3

mdadm --remove /dev/md2 /dev/sdc3

mdadm --zero-superblock /dev/sdc3

mdadm --fail /dev/md1 /dev/sdc2

mdadm --remove /dev/md1 /dev/sdc2

mdadm --zero-superblock /dev/sdc2

mdadm --fail /dev/md0 /dev/sdc1

mdadm --remove /dev/md0 /dev/sdc1

mdadm --zero-superblock /dev/sdc1

At this point, the software RAID still exists, but the /dev/sdc disk has been removed from it. All of the data on /dev/sdc is set as standalone.

Next, we need to modify the partition table on /dev/sdc to change it from software RAID to standard Linux partitions.
fdisk /dev/sdc

Select "p" to change the partition table, then "t" to change the type of partition. Select the partition number from the list. We changed /boot and / to be standard ext3 partitions, which is code 83, and the swap partition was changed to 82. Be sure to select "w" at the end to write all of these changes to the /dev/sdc disk when you are done.

Next, we need to mount the / and /boot partitions of /dev/sdc so that we can change files on that filesystem, so that a reboot on /dev/sdc is possible.
mkdir /mntboot

mkdir /mntroot

mount /dev/sdc3 /mntroot/

mount /dev/sdc1 /mntboot/

vi /mntroot/etc/fstab

Change fstab to so that /dev/sdc partitions will be automounted upon boot, rather than the /dev/md devices. Also, move the mdadm.conf file on /dev/sdc out of the way, so that it cannot be used when booting /dev/sdc.
mv /mntroot/etc/mdadm.conf /mntroot/etc/mdadm.bak

Now, we need to modify grub so that the bootloader will load Linux using /dev/sdc and not the /dev/md device. Notice that I will be doing this on the existing /dev/md running filesystem.
vi /etc/grub.conf

Replace the /dev/md2 (or whatever your root partition is) references with /dev/sdc3 (in our case). Save the file and close it.

Also change /mntroot/boot/grub/grub.conf with the same information.

Next, we need to run mkinitrd to use grub to update the bootloader, so that /dev/sdc will be used on boot. To do this look at the kernel you are booting from in /etc/grub.conf. For us, the mdkinitrd command looked like this:
mkinitrd -f -v /boot/initrd-2.6.9-103.ELsmp.img 2.6.9-103.ELsmp

After you run that, you will the bootloader being reconfigured. When it is complete, you are ready to reboot the server. You should boot up on /dev/sdc and be ready to do the VMware Converter Standalone importer.

View more truk blog entries





Lakehouse Moments
2014-07-19 19:02:22

image



image



image



image



image



Thanks to the Parkers for another wonderful lakehouse weekend! Cool July weather, good friends, and a beautiful sunset. What more could you want?

View more truk blog entries





Katherine and Sadie
2014-07-13 12:40:14

Katherine and Sadie, the 9.5 week old Border Collie.
image

image

image

View more truk blog entries





One Second Everyday - April to June, 2014
2014-07-03 08:46:41



This video covers events from a trip to St. Louis, Katherine's Engaged Scholarship Award, Hayley's birthday party, our trip to Southern France, Sicily, and Barcelona, and the arrival of our new puppy, Sadie.

View more truk blog entries





Introducing Sadie (the Lady)
2014-06-26 07:09:07

image



image



image



We got a new family member yesterday: Sadie (the Lady)! She is a beautiful 7-week old border collie.

View more truk blog entries





Beach to Bullring in Barcelona
2014-06-10 09:27:00

image



Today, we started out at the Picasso Museum. In some ways, this has been a very Picasso-infused trip. From the 1890s as a boy through his death in 1973, he did his best work along the stretch of sea we've been experiencing the past few weeks. The Picasso Museum in Antibes explored his later work, especially pottery, while the Barcelona museum focuses on his earliest drawings, sketches, and paintings. Both are excellent, and I'm still not the biggest Picasso fan, I can definitely understand the genius behind some of the best art in the 20th Century.

Next, we headed down to the beach. I'm always going to be in favor of a beach that you can visit via subway. The Barcelona beach is exceptionally nice, a great place to stroll and take in the sea air.

The Barcelona Metro is the perfect teleportation machine to get around this diverse and busy city. Next up, we visited an old bullring that the Barcelonenos modified to be a surreal 5-story mall.

What a fun and unexpected city!

image



image

View more truk blog entries





Gothic Barcelona
2014-06-09 08:57:00

image



We started our first day in Barcelona at the must-see site for most of the 17 million visitors that visit the city every year, and as all of the other museums are closed on Monday, about half of that number descended on the Gaudi-designed masterpiece Sagrada Familia with us.

The church really is amazing and worth the admission price and crowds, but I'm not sure how "sacred" the place can feel with the crush of people all around. It has the feeling of a circus combined with an international photoshoot. I can't imagine what it would be like in the real high-season, July and August.

We also explored Barcelona's old gothic city, the marina, and Parc de Montjuic, with some tasty tapas on way back to the apartment. A fun day...

image



image



image



image



image



image

View more truk blog entries





Barcelona!
2014-06-08 05:05:00

image



Katherine and I just made it into the apartment in Barcelona, and this is the view from our window. This is a famous market but is under renovation this year.

Many thanks to Laura, Antonio, Guisey, Toy, Chicho, and Antonella for hosting us and putting up with all of our questions (in English, no less!) over the past week! It was sad to leave Sicily, but we know we will return someday.

View more truk blog entries





Island Hopping
2014-06-06 05:26:00

image

Katherine, Laura, Antonio, Chicho, Toy, and I rented a boat today for a look around some of the other Aolian Islands. We visited Lipari and Volcano, as well as stay off of the coast of Salina, where Chicho and Toy caught our dinner. What a wonderful day!

image



image

View more truk blog entries





Panarea
2014-06-05 05:32:00

image



For a change of pace, Katherine, Antonio, and I took the ferry over to Panarea, a nearby island to Salina. We walked around, got some sun, ate a granita, hung out a a beach, and viewed an archeological site. Panarea is a beautiful place, much smaller than Salina.

image



image



image

View more truk blog entries





The Rambling Continues:
Read All truk Blog Entries Here

truk and kath and friends

What is truk.com? It covers the travels and thoughts of truk, this guy that lives in Midtown Memphis. More questions? Contact me...

2014 - Memphis
January One Second Everyday Video from August to December 2013
March New Orleans
April January to March 2014 Daily Video | Mississippi River Ride with Antonio
May Southern France, Sicily, Barcelona - Carcassone | Castles of Southwestern France | Hill Towns of Provence | Avignon | Riams | Grand Canyon du Verdon | Les Baux | Arles | Cassis | Pont du Gard | Nimes | Aix-en-Provence | Cotes du Rhone | Roussillon | Rural Provence | Apt and L'Isle-Sur-la-Sorgue | Monaco | Nice
June Antibes | French Riviera | Salina | Panarea | Island Hopping | Gothic Barcelona | Beach to Bullring in Barcelona | Welcome, Sadie the Border Collie!

2013 - Memphis
January Tahoe with Aaron, Jeff, Larry, and Scott
May Memorial Day at the Parkers' Lakehouse
October 7th Annual Walk Across Memphis
November Farewell, Jake

2012 - Memphis
March Goodbye, Mazzy
May Pennington Family Visit
July Sicily - Mount Etna Up | Mount Etna Down | Cefalu | Tindari | Palermo | Northwest Sicily | Erice | Southwest Sicily | Enna | Villa Romana del Casale | Caltagirone to Scicli | Siracusa | Salina
August Median Sunflowers
September More Scenes of Sicily
December 6th Annual Walk Across Memphis

2011 - Memphis
April San Diego and Tijuana
May Memphis Flood, Rising, Rising
June Double Negative, Nevada Land Art
August Ireland - Dingle Peninsula | Dunbeg Fort | Kilmalkedar Church | Giants Causeway | Dunluce Castle | Derry | Dublin Museum | Trinity College

2010 - Still in Memphis
January
San Francisco Bay and Wineries
April Douglas Family Visit | End of the Road
July - West and Central US Badlands | Black Hills, SD | East Glacier National Park (1, 2) | Polebridge, MT | Northwest Glacier | Missoula Anniversary | Yellowstone (1, 2, 3, 4) | Grand Tetons | Spiral Jetty | Park City | Vegas | Grand Canyon | Hoover Dam
September Labor Day Motorcycle Ride
October 4th Annual Memphis Walk
November Circle Arkansas Ride

2009 - Still in Memphis
January
Sydney and New South Wales (pics)
May Douglas Visit
July - Italy (Amalfi Coast with text | Capri | Herculaneum | Southern Tuscany with text | Florence with text | Northern Tuscany | Rome | Vatican)
August Otha Turner Family Goat BBQ Picnic
October Ozark Trail Ride day 1 | day 2 | day 3
November Dave Gets a KLR 650 | Hot Springs Honeymoon | Rhodes Cajunfest 2009
December Memphis Walk North(east) | Savannah Off-Road Motorcycling | San Francisco with Jeff, Katy, Larry, and Michelle

2008 - Still in Memphis
April
Visiting Jeff and Katy
May - Spain and Morocco with Dave and Christina (Segovia | Casa | Central, Northern Spain | Picos de Europa | Bilbao and San Sebastian | La Roija Region | Madrid | Marrakech | Fez | Southern Spain)
October Memphis Walk South
December Sydney (pics)

2007 - Memphis
February
Snowboarding in Tahoe
May Memorial Day at the Parker's Lakehouse
June Larry and Michelle's Wedding on Shelter Island, NY
July Hawaii (Oahu | Big Island (plus kayaking at Captain Cook) | Kauai text 1 and text 2 | Hawaii (Mostly Budget) Accommodation Review)

2006 - Memphis
January
Las Vegas
April Jacob Born
July Douglas Visit in Rhode Island
September Iowa Visit
October Urban Wolf River Trip
November and December Costa Rica (Volcan Arenal | Monteverde | Montezuma | Uvita | Porto Jimenez | Corcovado | Aerial Photos)

2005 - Memphis
January
New Years at Dave and Christina's
May Kath's PhD Graduation from Duke University
June Kath's PhD Graduation Party at the Shack-Up Inn

December New Zealand and Australia (Tahiti and Wellington | Marlborough and Kaikoura | Abel Tasman and the West | Manapouri and Milford Sound | The Catlins | Dunedin and Christchurch | Sydney)

2004 - Memphis
January Jeni's Baby Shower | Birth of Emmett
February Snowboarding in Park City
May Memorial Day at the Parker's Lakehouse
July Yucatan Movie [QT 7] (Isla Mujeres, Tulum, Chichen Itza, Valladolid, Merida, Celestun) | Paris
October Visiting the Douglas's in Boston

2003 - Memphis
January New Years

February Aaron's Snowboarding Bachelor Party

June Jaymie's Bachelor Party

October Dave & Christina Wedding Shower

September Dave's Bachelor Party

October Dave & Christina's Wedding

truk and kath in australia

2002 - Australia - Memphis
January Tasmania - Port Arthur [1st] | Tasmania - East Coast [2nd]
April Queensland [1st to 15th]
June Sydney Opera House [11th] | Broken Hill [15th] | South Australia [20th] | Western Australia [26th]
July Darwin, Litchfield, Katherine Gorge [16th
] | Arnham Land [19th] | Kakadu National Park
September Prague Wedding Reception, People, More People, Buildings | Croatia Mainland, Vis
November Kath's Surprise Birthday Party
| Thanksgiving
December Christmas

2001 - Australia
September Peyton's Baptism [2nd] | Apartment [9th] | Cliffwalk [18th] | Footy Final [23rd]
October Koori Knockout [1st] | Vigoro [20th] | North Sydney Beaches [30th]
November Sports Awards [9th]
December Tasmania - Cataract Gorge [30th] | Tasmania - Cradle Mountain [30th] | Tasmania - West and South [31st]

1999 - Southern Africa
Cape Town | Garden Route | Jo'berg | Victoria Falls | Stellenbosch

1998 - Peru
Lima | Huancayo | Ayacucho | Cuzco | Machu Picchu/Puno | Lake Titicaca | La Paz | Arequipa | Colca Canyon | Nazca/Pisco/Lima

1997 - Prague, Greece, Turkey
Prague | Cesky Krumlov | Romania | Alexandropolis | Samothraki | Ruins | Prague

1996 - Australia
Sydney

1995 - Asia, Silk Route
Turkmenistan | Kyrgyzstan | Kashi/Urumqi | Iron Rooster | Beijing | Great Wall | Hong Kong | Philippines

1994-1995 - Prague

Other Trips - Greers Ferry Lake (2001)






Arkansas Expats
A collection of friends


Peebo.net
Jeff Parker's blog


Issafly
Brad Sim's blog


truk.com
truk and kath's website


The Tackle Box by Matt


The Adventures of Pete Big Bear
Follow the adventures of Pete Big Bear while he roams around the US


A Foolish Consistency
CinemaApprentice's blog; great advice on movies and books


From a Place Near a River
Lady Di is online and loving it


Life Beyond Expection
Kath's CF blog


Jethro Williams Travel Blog
A man from England, working his way around the world


Dave and Christina


Travelogues

please: contact truk

©1996-2014 • All Rights Reserved | dedicated to Sara Low