Upgrading to ESXi 5.5 Update Manager vs ISO

Technology

I recently upgraded a cluster of ESXi 5.1 hosts to 5.5 and learned a lesson about using a NIC that requires a third party driver.  All of my hosts are using Qlogic 10Gb CNA cards which requires a driver to be loaded (a custom vib).  Unfortunately if you try to upgrade to 5.5 using Update manager it disables the third party drivers before actually doing the update.  Since Update manager does the update over the network interface as soon as that driver is disabled the host loses contact with vCenter and Update manager and the upgrade hangs.  Luckily nothing has actually been done yet and the host can simply be rebooted and it will come back up as though nothing has been changed.  The solution is to always do the upgrade using the ISO (either as a physical DVD or USB flash drive or network mounted via a management card like an iLO on HP servers) so that there is no networking required during the upgrade.  Once the upgrade is completed I was able to install the newer driver for ESXi 5.5 and a reboot later and my host was back in business.

So the lesson learned is be very careful if using third party drivers not part of the base ESXi image.  It always seems to be safer to interact with the host directly rather than using vCenter for upgrades (never had an issue yet with patches though) if third party drivers are involved at all.

The next time I have to upgrade I think I may try to create a custom ISO with my third party drivers already integrated just to avoid the extra steps.  If I do I’ll try to post about the process and how it went.

No Comments

Way behind

Uncategorized

It’s been a while since I’ve added an entry on here and I’ll be adding a bunch soon.  I’ve got a bunch of half written posts and some new recipes I just have to get the pictures the way I want.  I’m still here and more should be coming soon!

No Comments

Monster Cookies

Uncategorized

Just made our 2012 Christmas batch of Monster Cookies and thought I’d share.

Ingredients:

  • 12 Eggs (yes a full dozen)
  • 2 lbs Brown Sugar
  • 4 c White Sugar
  • 1 Tbsp Vanilla
  • 1 Tbsp White Karo Syrup
  • 8 tsp Baking Soda
  • 1 lb Butter Softened (yes a pound)
  • 3 lb Peanut Butter
  • 18 c Oats (Not quick oats)
  • 5 bags various candies (I used M&Ms, Chocolate Chips, Peanut Butter Chips, Butterscotch Chips & Heath bits this time)
Tools:
  • If you have a mixer with detachable beaters I strongly recommend using one of them in a power drill to do the actual mixing as you can burn out a standard electric hand mixer with this recipe.

Recipe:

  • Beat the dozen eggs
  • Add in the Brown and white sugars and beat until well mixed
  • Add in the Vanilla, Karo Syrup and Baking Soda and mix in
  • At this point start using the power drill if you can
  • Add in the Butter and beat until fully mixed
  • Add in the Peanut Butter and beat until fully mixed
  • At this point you will need to use either a strong wooden spoon or your hands
  • Add in the Oats and mix thoroughly
  • Add in the candies and mix to get an even distribution throughout the batter
  • Preheat the oven to 350F
  • Spoon the batter onto an ungreased cookie sheet (you can make these a variety of sizes to try out different sizes, I used a Pampered Chef medium scoop with a  squeeze handle for ease and I wanted them to be a consistent size)
  • Bake for 10-15 minutes (varies by the pan type used and the size of the cookie so keep an eye on them until you figure out the time you need)

This recipe will make anywhere from 12 to 25+ dozen cookies depending on how large you make them.  Using the scoop I ended up with around 27 dozen cookies this time.

No Comments

Gingered Carrots with Mahi Mahi & Tat Soi Salad with Lemon-Ginger Vinaigrette

Recipes

We needed a recipe to try something new with Mahi Mahi we had in the freezer and wanted to try to use at least a few things from our recent CSA box as well.  We had fresh ginger in our box (first year of ginger for our CSA!) and some beautiful tat soi heads and these two recipes sounded like they’d go well together.    The Mahi Mahi recipe is compliments of everydaypaleo.com and the tat soi recipe is from our CSA newsletter (Andrea Yoder).  My wife did the cooking tonight and both dishes turned out excellent and the flavor combination was perfect!

Tat Soi Salad with Lemon-Ginger Vinaigrette (By: Andrea Yoder)

Ingredients:

  • 1 head of tat soi, greens & stems in bite sized pieces, washed and dried
  • 1 medium carrot, finely shredded
  • Lemon zest from 2 lemons
  • 2 Tbsp lemon juice
  • 1 Tbsp honey
  • 1 Tbsp fresh ginger, minced or pressed with a garlic press
  • 4 Tbsp olive oil
  • Pinch of salt
  • Freshly ground black pepper
  • 1/4 cpu almonds, chopped & toasted

Recipe:
Prepare the vinaigrette by combining the lemon zest, lemon juice, honey and ginger in a bowl.  Slowly drizzle in the oil and stir to combine.  Best if made 15-20 minutes before server to allow flavors to mix.

Prepare the salad just before serving by placing the tat soi and shredded carrots in a bowl and mix.  Top with pinch of salt and freshly ground black pepper.  Add the vinaigrette and toss to combine.  Don’t use too much vinaigrette, just enough to give a light coating.  Top with toasted almonds.

Gingered Carrots with Mahi Mahi (double this recipe if feeding more than 2 or 3)

Ingredients:

  • 4 carrots, sliced
  • 3 tablespoons grass fed butter or coconut oil
  • 5 green onions, diced
  • ½ teaspoon fresh grated ginger
  • 1 teaspoon crushed garlic
  • 1 pound of mahi mahi pieces or other white fish of your choice
  • 1 tablespoon fresh lemon juice
  • ½ cup chicken broth
  • 1 tablespoon coconut aminos or 1/2 teaspoon of wheat free tamari
  • Fresh ground black pepper and sea salt to taste

Recipe:

  • Saute the carrots in 2 tablespoons of the butter or coconut oil over medium heat for 5-7 minutes or until the carrots start to brown.
  • While the carrots are cooking, mix the chicken broth, coconut aminos or tamari and black pepper together in a bowl and set aside.
  • Add the green onions to the carrots and cook for another minute.
  • Add the lemon juice, ginger, and garlic to the pan and sauté just until the veggies are coated with the ginger and garlic.
  • Sprinkle with a little sea salt, stir again, and remove the carrot mixture from the pan and set aside.
  • Add the remaining tablespoon of butter or coconut oil to the same pan and melt over medium heat.
  • Add the fish pieces and cook for 1 minute on each side, making sure that your pan is nice and hot so that the fish sear and turn golden brown on each side.
  • Now, pour the chicken broth mixture that you made earlier over the fish, cover and cook for another 1 – 2 minutes or until the fish is tender and flakes apart easily.  Do not overcook!!
  • Serve the fish over the gingered carrots

 

 

No Comments

Microsoft Cluster Fun

Uncategorized

I had an interesting experience recovering a single node windows 2008 R2 cluster running multiple MSSQL 2008 instances.  We suffered a power failure that caused the server to reboot and after coming up the cluster service would crash at start.

Initially the only thing to go on was a single entry in the System Event Log for Event ID 1573:

Node ‘Servername’ failed to form a cluster.  This was because the witness was not accessible.  Please ensure that the witness resource is online.

I checked on the quorum disk and it’s there and marked as reserved as expected.  Head scratching commenced for a bit.  Tried a reboot just to make sure and had the same issue.  Tried to manually start the service and had the same issue.  Did some googling on the error and chased down a few items that ended up not being anything.

Tried to start the service with the fixquorum flag with no result.  Also tried to use the resetquorumlog with no result.

I discovered the cluster log command to generate a text file log of the cluster service which is when I finally started to make some progress:

Open a command prompt and run cluster log /g

This will output a file Cluster.log in C:\Windows\Cluster\Reports

On initial review of the log I found:

00000990.00000cf8::2012/11/14-13:18:25.643 ERR   mscs::QuorumAgent::FormLeaderWorker::operator (): ERROR_FILE_NOT_FOUND(2)’ because of ‘OpenSubKey failed.’

Which told me there was something wrong in the registry hive for the cluster.  The hive for the cluster is located in C:\Windows\Cluster and is a file called CLUSDB.  This file is automatically expanded and loaded under HKLM in the registry when the cluster service starts.  It was during this process that the server was crashing out so something was corrupted or wrong in the file.

My first attempt at a fix was to recover the CLUSDB file from a midnight snapshot taken about 3 hours prior to the power issue that caused the reboot.  Unfortunately this did not solve the problem which made me realize that something had changed or corrupted the file prior to the reboot and it just didn’t show itself until the reboot.  I went back to the Cluster.log file to see if I could find any more information.  I was regenerating the cluster.log file (cluster log /g) after each attempt to start the service to see if anything was changing and I notice something common with each startup:

000014b0.00000cf8::2012/11/14-13:25:39.708 DBG   [RCM] Resource ‘SQL Server (INSTANCENAME)’ is hosted in a separate monitor.
000014b0.00000cf8::2012/11/14-13:25:39.708 DBG   [RCM] rcm::RcmAgent::Unload()
000014b0.00000cf8::2012/11/14-13:25:39.708 INFO  Shutdown lock acquired, proceeding with shutdown

On each startup it would fail after the same INSTANCENAME and start to shutdown the service but I knew there should have been more Resources listed which meant the problem may be with the resource right after the last INSTANCENAME noted in the log.

With the cluster service stopped (so it wouldn’t try to restart and the hive wouldn’t be loaded) I launched regedit.  I navigated to HKLM and did a File->Load Hive and selected the CLUSDB file in C:\Windows\Cluster and gave it the name “Cluster” when prompted.  I then expanded the new cluster folder and then the resource folder and started to go through the list.  I quickly realized the order of resources in the folder matched how they were being noted in the Cluster.log file.  The resource that was next after the INSTANCENAME that was last noted in the Cluster.log was the Available Storage resource.  In looking at the keys for that resource I realized it had other resource ID’s listed in the “contains” key which should be storage resources that were in the Available Storage group except I knew that there shouldn’t be any.  I made note of the two resource ID’s in the contains key and went through the rest of the resources to make sure they didn’t actually exist and they didn’t.  I then went back to the contains key for the Available Storage resource and edited it and removed the two entries.  I then highlighted the Cluster folder under HKLM and unloaded the hive File->Unload Hive and then closed out regedit.  I started up the cluster service manually and this time everything started up correctly.

So what happened?

Roughly 2 weeks prior to this outage an Instance had been removed from the cluster.  It had 4 storage devices associated with it which were initially moved to the available storage group after being removed from the instance group and then were deleted as disks from the cluster.  Apparently this process (done via the failover cluster gui) failed to fully remove 2 of the 4 objects from the registry correctly.  I’ve found a few other people suggesting to always use the command line cluster program to remove resources to be extra safe which I plan to do from now on.  The problem did not show up until the next time the cluster service restarted.

1 Comment

Cinnamon Toasted Oven Roasted Whole Chicken

Recipes

I found this recipe on www.paleo-project.com and the link on that site is now dead unfortunately but that is where the credit goes.  I made this for the first time tonight and it turned out awesome.  The chicken was extremely moist and very flavorful.  I made just a few changes which really just came down to increasing the amount of spices in the rub as it felt a little light in the original recipe.

Ingredients:

  • 1 Whole Chicken (around 4 lbs)
  • 2 tbsp honey
  • 2 tsp salt
  • 1 tsp ground nutmeg
  • 1 tsp ground cloves
  • 1 tsp allspice
  • 1 tsp ground cinnamon
  • 5 garlic cloves (crushed)
  • 1 tsp whole cloves
  • 1 tsp coconut or olive oil

Recipe:

  • Mix together the salt, nutmeg, ground cloves, allspice and cinnamon in a small bowl
  • Cover a baking pan with aluminum foil (I used a cookie sheet with a lip high enough to hold the juices)
  • Grease the spot the chicken will rest with the oil
  • Preheat oven to 500 F
  • Rinse chicken in cool water and pat dry (don’t forget to remove the extra bits often in a plastic bag inside the cavity!)
  • Put the crushed garlic and whole cloves
  • Spread the honey all over the outside of the chicken
  • shake or rub the spice mixture all over the outside of the chicken
  • Space the chicken on the pan breast side up
  • Cook for 15 minutes at 500 F
  • Lower the temperature to 450 F and cook for another 15 minutes
  • Lower the temperature to 425 F, baste the chicken with liquid from the pan and cook for another 30 minutes or until the chicken is around 180 F in the breast
  • Remove the chicken from the oven and allow to rest for 20 minutes before serving

 

No Comments

Jewel Studded Salmon with Cilantro Cream Cheese

Recipes

While I haven’t made this in a little while I wanted this to be the first recipe I shared simply because of the picture.  The recipe itself I found originally on www.grouprecipes.com and my picture is on that page as well which I posted the first time I made it.  This is not a Paleo (how I’m eating now) recipe (cream cheese) but would probably qualify as Primal (how I will be eating in a few more weeks).  I’ll post more on the subject of Paleo and Primal eating soon but here is the recipe:

Ingredients

  • 2 large salmon filets with skin removed
  • 2 cloves garlic
  • 4 tbsp softened cream cheese
  • 4 stacks of fresh cilantro chopped
  • 1 tsp kosher salt

Instructions

  • Peal and mince the garlic
  • Mix the garlic, cilantro and cream cheese together in a bowl
  • Butterfly the salmon filets (if the filets are not thick enough to butterfly easily you can also just cut a line down the middle of the filet not quite all the way through and then fold the filet in half to be just like butterflied)
  • Spread the cream cheese mixture inside of each filet and fold closed (cream cheese should be on the inside)
  • Place on a hot oiled grill and cook for about 6 minutes on each side
  • Sprinkle with kosher salt on both sides while grilling

Results

No Comments

When power saving is not your friend

Technology

I’ve been investigating a performance problem in a VM on one of our ESXi 5 clusters that led to an interesting discovery about power savings settings on the ESXi host.  Basically under certain scenarios (and perhaps specific CPUs) they physical CPUs will be down clocked even though a VM is trying to use 100% of its CPU.

The physical host servers are HP DL385 G7 with 2 AMD Opteron 6174 12 core processors @ 2.2GHz and 128 GB of RAM.  They boot from an integrated SD Flash card and all other storage is provided by our Compellent SAN.

In the bios there are 3 key settings under the Power Management Options:

HP Power Profile – This defaults to “Balanced Power and Performance” but I’ve changed it to “Maximum Performance”

HP Power Regulator – This defaults to “HP Dynamic Power Savings Mode” but changes automatically to “HP Static High Performance Mode” after changing the power profile setting

Advanced Power Management -> Minimum Processor Idle Power State – This defaults to “No C-states” and that is what we want it set to

The VM I’m testing with has 4 vCPU and 8GB RAM assigned to it.  This VM is the host for a Lotus Domino server with some custom applications.  When the application is used it can cause the CPU to go to 100% utilization within the VM.

From testing the same processes over and over we observed that each process would take 50-150% longer to run with the bios set to Balanced vs having it set to Max.

What I believe is happening is that while the VM is running at 100% cpu it only using 4 of the 12 cores of a single physical socket (and 4 of 24 total in the host) and the other VMs on this host are all light CPU load so the physical host perceives itself to be lightly loaded and so is down clocking the CPU.  So our VM running at 100% CPU is not getting 2.2GHz of clock speed but some lesser amount depending on how much down clocking the host has done.  Since that down clocking is dynamic that would also account for the performance variance we are seeing.

In googling around I’ve found other people using the AMD Opteron 61xx series processors with VMWare having a similar issue.  It’s possible this is just an issue with that line as I don’t believe a CPU should slow the clock speed dynamically if a single core is being used completely (rather than relying on an average load accross all cores to determine if it should save power by down clocking).

We have another cluster that uses AMD Opteron 6282 SE processors I plan to do some additional testing on to see if the problem exists there as well.  I’ll update this post once I’ve had a chance to do that.

For now all of our hosts using the 6174 processors have been set to force max performance (more power and heat unfortunately).

No Comments

Relaunch

Personal

I’ve finally taken the time to rebuild and relaunch my website.  I’m going for more of a mixture this time and hopefully this version will have some staying power.  In addition to technology and other geek related posts I plan to bring in another aspect I enjoy which is cooking through recipes, pictures and other food commentary.

No Comments