Monthly Archives: May 2008

ESX Storage Journey Takes II – iSCSI Storage Limitation

30th May 2008

Preparations for file server migration
Deadline is near to virtualize one of our File Print+ Symantec AV server.  I’ll need to virtualize and hook up a LUN from a Dell MD3000i via iSCSI which is connected to ESX and map it as a raw partition to the file server.

This is one hell of a work considering that we’ll need to p2v the server running on Win2k and upgrade it to Win2k3 at the same time ensuring clients will be able to access to the services after upgrading. But what the heck, as far as the management is concerned, we are the ones who makes all things possible. 

I’ll need to test out the environment for this new virtualized system before we put it to production.  I would also need to test out if the 2TB storage limit is in effect if we mount it as a raw partition on guest via the host iSCSI initiator.

Equipment
The MD3000i unit is still pending arrival so I’ll be running some test on our existing Clarion 3-20, flexing out its iSCSI capabilities which we’ve never even thought of using.  Test is conducted on a Dell 2900 with 16Gb of ram attached with 2 FC HBA and 6 NICs running ESXi.

Setup
I started off by firing off the Navisphere console and setup a 2.4TB LUN to test out the 2TB barrier of ESXi.  On the ESXi, I’ve created a new VMkernel port and have assigned to it a dedicated NIC.

With CHAP disabled, the LUN, 2.4TB appears on the VI client after a click on the rescan; I was trilled.  That is just part one, we’ll need to test it on the guest to be sure. 

I mounted the raw partition as "virtual compatibility mode" into the guest and proceed to power up the Win2k3 Ent R2.  Only 367.28GB was detected out of the 2.4TB.  Thinking it might be due to "virtual compatibilty mode" I proceed to delete the drive from the guest and created a new raw partition as "Physical mode".  This time its worse; it states there unknown under storage device.  I’ve gotten the latest vmware-tools installed but issue still persist.  (*refer Note)

After several more attempts, I proceed to remove the virtual hard drive from the guest and did a iSCSI rescan from ESXi and poof goes the drive; it went missing.  Could it be EMC?  I don’t really trust that unit; its like a total black box with a few click-me buttons.

Note: It was the next day after all these test has been done that I’ve realized that I’ve got the FC HBAs connected to the SAN as well.  Navisphere clearly shows the two connections as separate entities during testing but yet in this case, the LUN was connected via FC instead. 

I’ve tried unplugging the HBA  and found that LUN size higher then 2TB is not detectable via ESXi’s iSCSI initiator – below.  Further investigation shows only FC is able to detect LUNs higher then 2TB but the 367Gb persist in Win2k3 Ent.  Could that be some kind of limitation on the iSCSI initiator?

 

iSCSI Storage limitation
It took me a lot more tries to remove and adding in the partition from Navisphere, making sure that I’ve added in the correct LUN into the appropriate Storage Group. No good, the LUN is not showing up anymore in ESXi.  And finally it came up right after I recreated and attached it to a 2TB LUN. (I’ve unplugged both the HBAs at this point)

Now, I’ll need to mount it in guest to make sure it works.   With the LUN mounted as a raw partition in the guest, Win2k3 Ent R2, disk manager shows a solid 2TB drive ready to be used.

It was an awfully long, tedious and frustrating day spent testing on the solution but least I now know that its going to work.  Will be spending more time on VMware’s Community forum on this issue while preparing for the next phase of our file server migration.

till then…..


ESX Storage Journey Takes I

Working with iSCSI on ESXi

Awhile ago I was working on ESX Network Performance tuning and at the same time I was also looking at possibilities of improving iSCSI transfer rates using the same way.

Sadly on my test lab over here, we have a Clarion 3-20 box which does neither have support for iSCSI port binding nor any features that helps in loadbalancing.  I’m not so sure about other SAN boxes out there but I really hope the Dell MD3000i, which is due to arrive to our lab 2 weeks from now, has this feature available.

Not giving up, I proceed to explorer for other available options and bingo!  There is a new feature in ESX3.5, iSCSI Round Robin path loadbalancing, which is labeled (Experimental) below.  Smart of them to label Experimental; that also means non-production environment only, use at own risk. 

All glory to google, I found someone who has tested this feature and has posted in VMware community. http://communities.vmware.com/thread/131295

iSCSI CHAP Authentication in ESXi & 3.5
The setup for the system mentioned in the communities page above, posted by Damin mentioned that he is using a ISCSI box from WASABI Systems.  I wondered if he has encountered any CHAP authentication issues while connecting to ESX?

I’ve spent endless hours trying to get ESX to connect to a CHAP authentication enabled iSCSI target in the Clarion box only to find that it won’t work 😦  The target will disappear from ESX the moment authentication is enabled.

Sometime ago I’ve setup a linux based iSCSI box for ESX running on iET, iSCSI Enterprise Target with similar issues.  This has lead me to believe there is something really wrong somewhere in ESX’s iSCSI authentication.

It is worth to note that there are no issues while connecting from Microsoft initiators in both cases.  I’ll be doing more research on the issue and will update.


ESX Tips – ESXi Console II

Accessing the ESXi console
Thanks to http://www.run-virtual.com/?p=223
updated 04 Sept 08

ESXi has a busybox console build into it for troubleshooting purpose.  To enable it, press Alt-F11(for update 2 press ALT-F1) at the welcome screen and type "unsupported" without the "".

Congratulations, you now have access to ESXi console.  But we’ll need to enable SSH for accessing the console remotely.  Here we’ll need to edit inetd.conf and removed the # infront and restart inetd.

—————————————————————————————————-
Tech Support Mode successfully accessed.
The time and date of this access have been sent to the system logs.

WARNING – Tech Support Mode is not supported unless used in
consultation with VMware Tech Support.

~ #vi /etc/inetd.conf
—————————————————————————————————-

Find this line below and remove the # in front.  Then save exit Vim editor.
#ssh     stream  tcp     nowait  root    /sbin/dropbearmulti     dropbear ++min=0,swap,group=shell -i <– This line remove the leading #

Find and kill inetd service and restart it.
~ # ps |grep inetd
1705 1705 busybox              inetd
#kill -9 1705
#inetd

ESXi server is now ready to accept remote connections.  Do bear in mind that there is no firewall settings in the VI console as such, if security is a concern to you, you’ll need to disable ssh after use via the physical console.


ESX Tips – ESXi Console I

Lesson of the Day 29th May 08

Lost access to your ESX Server?
Never place more then one vmkernel within the same subnet.  It will cause you to lose connection to your remote console at just one wrong click at the gateway setting. (applies to all ESX builds)

If that happens, restarting the server wouldn’t help and updating IP from Physical ESXi SP1 console would not bring it back either.

Solution
You’ll need to remember the IP address of the new vmkernel network which you’ve just created as VMware console has now binded to that new IP.  Try to connect to the console using that new IP; make sure you have access to that subnet its running on.


ESX Network Performance Journey Takes I

20th May 2008, Tues 10am

"Despite the VMware ESX features page claim that NIC teaming provides load balancing, basic NIC teaming only provides outbound load balancing. To get inbound load balancing with NIC teaming, however, you must go the extra step and configure VLAN trunking and the port channel on the Ethernet switch to which these VMware ESX Server physical adapters are connected"
David Davis
http://searchvmware.techtarget.com/tip/0,289483,sid179_gci1311518,00.html

Another one of the sales/marketing crap we as consumer face everyday.  They’ll have thousand and one reasons to convince you that their product is the best of breed just to get money out of your pocket.

What the heck since we’re already on the boat, lets start sailing.  Slowe had made a good start in this exploration with his hands on experience at his fantastic site below.
http://blog.scottlowe.org/2006/12/04/esx-server-nic-teaming-and-vlan-trunking/

I’m dealing with nearly the same situation over here but with ESX connected to two Cisco 4506 which are supposed to failover to each other.  With NO experience with load-balancing, I have totally no clue what I’m going to face during this recklessly planned mission to test it on production environment.

But performance is our mission, heros will have to take some risks to accomplish the task; without the lady in red.

Lets stir some water

#show etherchannel load-balance
EtherChannel Load-Balancing Operational State (src-dst-ip):
Non-IP: Source XOR Destination MAC address
  IPv4: Source XOR Destination IP address
  IPv6: Source XOR Destination IP address

Crap, don’t have a clue what that means.  Best guess is that there might be a chance ESX NIC load-balancing will work without adjusting load-balance method in the production Cisco switch.  That, we wouldn’t know until we’ve setup the ESX for testing.

Back to drawing board.  Me need to read up more before i proceed from here.   Till next time ….


How secure is your network I

I was just browsing through some security sites and accidentally ran into some hacking sites instead. Sometimes ignorance is just not bliss.

http://www.airscanner.com – has some very interesting articles on network vulnerabilities all the way from ARP poisoning to WIFI hacking.  I’m sure you wouldn’t want to have a free port open in your cisco switch after this.