ESX CLI Reference Guide

Show contents of file:
less vm1.vmx

Show contents of directory including file sizes
less -ltr 

Show disk usage of san disks
vdf -h

Show local disk usage
df -h

list virtual switches
esxcfg-vswitch -l

List the service consoles
esxcfg-vswif -l


Restarting ESX Host services:

Management agent on the host:  
   
service mgmt-vmware restart

Restart Network:                          
   
service network restart

Web Access
   
service vmware-webAccess restart

VMkernel authorization daemon
   
/etc/init.d/vmware-vmkauthd stop
/etc/init.d/vmware-vmkauthd start

VMware late init tools
   
service vmware-late restart

Virtualcenter agent
   
service vmware-vpxa restart

Setup NTP using CLI
If you have ever had issues setting time on an esx host through the gui or you've noticed that the GUI time, date and timezone is different to what the actual system time is set to (as seen in the CLI), and its affecting your vmware HA functionality and configuration you need to setup your time sync using the cli. I've also noticed that a virtualized Windows NT 4.0 vm appears to give preference to the hardware time as opposed to the domain time.


1) Login via the console as root or use putty with your user id then type su - and press enter, then enter the root password.

2) Type the below. This will backup the file then open it in the vi editor to be modified

cp /etc/ntp.conf /etc/ntp.conf.bk
vi /etc/ntp.conf

3) Once in the vi editor, press i to edit or insert a line. Use the cursor keys to go to the location of the line you want to add.

add the following lines so it looks like this:

restrict 127.0.0.1
restrict default kod nomodify notrap
server 0.tbc-dns1.thebearscore.ca
server 1.lincolnnavigator.thebearscore.ca
driftfile /var/lib/ntp/drift

4) Continue scrolling down through the file and rem out all other lines using a # except for the line with the description mentioning that it is a dummy driver in case the system is unable to reach an ntp server.

5) To save and exit the vi editor, press ctrl c to exit edit mode. Then type :wq to save and exit. To exit only press ctrl c then type :q and press enter.

6) Type cp /etc/ntp/step-tickers /etc/ntp/step-tickers.bk

7) Type vi /etc/ntp/step-tickers

9) Type vi /etc/hosts and press enter

10) Add the lines EXACTLY AS BELOW and lined up with other entries in the file. Use the same capitalization as in the other files.

192.168.5.225            0.tbs-dns1.thebeerscore.ca
192.168.15.33            1.navigatorlincoln.thebeerscore.ca

Save and exit using ctrl c, :wq and press enter

10a) IN ESX 3.0.x and 3.5 onlY: type esxcfg-firewall --enableService ntpClient and press enter ******NOTE THE CAPITAL S in service and C in client. IT IS CASE SENSITIVE

11) Type service ntpd restart and press enter. Type it again if the service fails to stop, sync, and start.

12) Type chkconfig --level 345 ntpd on

13) Only after you've had a succesful stop, sync, and start of the ntpd. Run this command:

hwclock --systohc

14) Type clock and hwclock and verify that the time is correct


Stop and Start VMs from the command line when it does not work from the GUI
To list VMs:  vmware-cmd -l

To Start stop or restart:

vmware-cmd /vmfs/volumes/48a33628-e00c99f2-a997-00188b4d7043/fix/fix.vmx start

For the rare occasion where you may need to allow all incoming and outgoing traffic through the ESX host firewall perhaps say for troubleshooting purposes:

esxcfg-firewall --allowIncoming

esxcfg-firewall --allowOutgoing


Change ESX IP config from console
To change the management/service console IP address of the ESX 3.x host, you need to change the configuration of the vswif. By default this is vswif0 and this is assumed in this document. Login to the service console with root permissions, either by using root or doing a su - to get the permissions.

Once in the service console run the command "esxcfg-vswif -d vswif0". This command deletes the existing vswif0. Don't worry if you get a message about nothing to flush. Then you need to run the command to change the ip address, subnet mask and broadcast address. They are also specified in that order when the command is given. An example command is below.

"esxcfg-vswif -a vswif0 -p Service\ Console -i 10.1.1.1 -n 255.255.255.0 -b 10.1.1.255"

In this command the -a switch is to add a vswif, the \ in the Service\ Console is deliberate, the -i is the ip address, the -n is the netmask and the -b is the broadcast address.

You now need to change your default gateway, you can do this by editing the network file located at /etc/sysconfig/network. To do this at the command prompt, follow the steps below.

"cd /etc/sysconfig"

"vi network"

While in vi, go to the location of the default gateway using the arrow keys.

Hit "i" which will perform an insert and change the default gateway to your liking.

Hit the escape key twice to exit insert mode.

type ":wq!" to write (i.e save) and quit.

Then you have to restart the vmware management services.

Service network restart
Service mgmt-vmware restart

Open a remote console to your virtualcenter server, do a ping to make sure the ESX host is pingable after the IP change. Make sure you are seeing the new IP address and it is assumed you have already changed that in DNS. If you are seeing the host correctly, open virtualcenter and disconnect then reconnect the host.

Once the host is connected in virtualcenter we need to change a few bits of configuration information, namely the vmkernel ip address, subnet and gateway. This is so we can vmotion correctly. Click on your host and bring up the configuration tab. Select networking and then properties on the vitual switch.

Select your vmkernel and hit the edit button. Change your ip address here for vmotion and subnet mask. You will not be able to change the default gateway until you hit ok and go back in. Once you have selected ok, then hit edit again on the vmkernel. Select the edit button on the default gateway and change the default gateway on the menu that appears. Select ok, ok again and then close.

Please note all the commands in this document should be used without the "".

Edit the /etc/hosts file to change the names and IPs if you are not using DNS


Adjusting ESX Server Time Zone   
KB Article    1436

Note: It is not necessary to reboot the ESX Server host after following this procedure.

   1. Log into the ESX Server service console as root.

   3. Type vi /etc/sysconfig/clock

Type i to edit
Change zone to look like the below:

      ZONE=America/Toronto
      UTC=false
      ARC=false

   4. Copy the desired time zone file to /etc/localtime . Continuing the example using US/Pacific :

      cp /usr/share/zoneinfo/America/Toronto /etc/localtime

      Answer y to replace the file representing the old time zone.

   5. Confirm that /etc/localtime has been updated with the correct zoneinfo data using the following steps:

   6.
                 Reference the zoneinfo file used in step 2 and compare it to / etc/localtime using the diff command.

            [root@esxhost]# diff /etc/localtime /usr/share/zoneinfo/US/Pacific

            If the files are identical, your prompt will return without any output.

         2.
            If /etc/localtime is not the same as the /usr/share/zoneinfo/ , the output from the diff command will be something like the following:

            Binary files /etc/localtime and /usr/share/zoneinfo/US/Pacific differ

            If the files are not the same, repeat step 4.

 

After updating /etc/localtime with the correct zoneinfo data, confirm the system and hardware clocks are correct. Use the Linux date command to check and set the correct time if necessary. Set the hardware clock to match the correct system time.

   1. Set the system clock to the local date and time:

      date MMDDhhmmYYYY
   2. Update the hardware clock with current time of the system clock:

      /sbin/hwclock --systohc

Note: Upon booting, ESX Server runs /etc/rc.d/init.d , which reads /etc/sysconfig/clock to set the system clock based on the hardware clock's current time and the configured time zone information. To synchronize ESX Server to an external time reference, see http://kb.vmware.com/kb/1339 .


   2. Find the desired time zone under the directory /usr/share/zoneinfo .



To determine if VMs are powered on or off from console when the GUI isnt showing what appears to be accurate:
To list VMs:  vmware-cmd -l
Then to see if a particular vm is on or off:
vmware-cmd /vmfs/volumes/48a33628-e00c99f2-a997-00188b4d7043/fix/fix.vmx getstate

HA error troubleshooting tips:

    * Check to make sure DNS is configured properly
    * Check to see if you can resolve DNS
    * Check DNS records
    * Make sure you are using FQDN’s
    * Check your /etc/hosts file
    * Make sure your using lower case
    * Check Service Consoles have the same names and networks
    * Disable and Re-enable HA
    * Select Reconfigure for HA on the ESX host

Out of these troubleshooting tips the most common problem with HA is a DNS issue, so it is best to start troubleshooting DNS first.

ESX is apparently case sensitive on hostnames.  So, my server labled in the hosts file as CORPESX01.DOM.LOC needed to be labeled as corpesx01.DOM.LOC in order to resolve properly.  A simple nslookup from my Windows machine of one of the IP addresses confirmed this capitalization scheme.  I went through each host file making them look as follows:

10.10.10.1        corpesx01.DOM.LOC        corpesx01
10.10.10.2        corpesx02.DOM.LOC        corpesx02
10.10.10.3        corpesx03.DOM.LOC        corpesx03

Re-enable HA on cluster

Flush DNS client cache on ESX host
/etc/init.d/nscd restart


Installing Dell Server Open Manage (server administrator) on ESX

1) Download from:

http://support.dell.com/support/downloads/download.aspx?c=us&l=en&s=gen&releaseid=R183700&SystemID=pwe_r900&servicetag=410JLF1&os=LIN4&osl=en&deviceid=2331&devlib=0&typecnt=0&vercnt=2&catid=-1&impid=-1&formatcnt=2&libi"

2) Copy to /user/home directory (you may need to make your account an admin through the vi client for the host

3.Browse to the location where you downloaded the file and unzip the
package using the following command: gunzip xxxx.tar.gz
4.Untar the package using the following command: tar –xvf xxxx.tar
5.Change current directory to the location where the files were
extracted.
6.Run the following command to perform an express install:
sh linux/supportscripts/srvadmin-install.sh --express
7.Start the applicable services by running the following command:
sh linux/supportscripts/srvadmin-services.sh start
8. Type the following command to open TCP port 1311 inbound to allow access from your browser:
esxcfg-firewall --openPort 1311,tcp,in,dellov


Stop and Start VMs from console
To list VMs:  vmware-cmd -l

To Start stop or restart:

vmware-cmd /vmfs/volumes/48a33628-e00c99f2-a997-00188b4d7043/fix/fix.vmx start

Syncing time on esx using hardware clock - not NTP
To see the current ESX time:

# date

Thu May 1 14:57:55 BST 2008

To manually adjust the time:


date -s “MM/DD/YYYY HH:SS”

or

date -s  MMDDhhmmYY


for example to set the date and time to May 02, 2008 an 6:23 pm enter the following:

#date -s 0502182308


To check the current time for the hardware clock type:
#hwclock
Sync the hardware clock with the Service Console:
#hwclock –systohc

Vmware Vsphere 4.1   Training Notes


Fault tolerance - an extension of HA with no downtime. Two identical in-sync machines on different hosts

Data recovery -

vShield zones
VMsafe

ESX5 will be 5i only - no more full blown esx with the service console

ESXi Console configuration settings - DCUI  (direct console user interface)
Lockdown mode - restrict remote access using root

-2 Gateways: vmkernel and service console

To perform mgmt tasks from a remote command prompt:
vCLI - vmware vsphere command line interface. Set of commands run from a remote machine. Packaged as an application.vMA - vmware vsphere management assistant. A platform for running a variety of toolkits: vCLI, vsphere SDK for Perl, vsphere API.
Packaged as a linux based virtual applianceVsphere PowerCLI - automation tool for admin of a vsphere environment. Distributed as a snap-in to windows powershell.

Component and Plugins
Update manager component on VC server needs update manager client on client

Linked Mode install of Virtual Centre Server

Virtual Centre Datacenter - meant to be a logical division of resources by geographic or resource location
Shared storage, networks, hosts, VMs per datacentre
You cant vmotion between datacentres

Virtual Centre Folders
Organize objects based on similarity e.g. Intel CPU hosts in one folder, AMD in another
DB VMs in one folder, File server in another

vpxa - agent running on esx host that communicates with vc server (for instructions). Communicates using vpxuser account on esx host.
hostd - does the work requested via vpxa

Vmkernel port - provides network access to vmkernel and between vmkernels- such as when accessing an iscsi network, vm request to accessNFS share, or for vmotion where memory is transferred from one esx kernel to another
Port Group -  vlans within a virtual switch

Licensing
Vcentre licensing  - ?,?, foundation, standard
Vsphere licensing - ?,?,?, enterprise plus

Network configuration for physical switches to support trunking and nic teaming: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004048

Switch Port Group Properties
Promiscuous Mode - sniffing vm traffic for troubleshooting
Mac address changes - use with MS NLB to allow accepting of packet with same mac address as in case with NLB
Forged transmits
You can expand a lun by increasing its size using an extent - combining two luns using vmware


Features of Vmware ToolsDriver: vmxnet, balloon driver for memory management, sync driver for quiescing IOVirtual machine heartbeat  - monitoringTime syncAbility shutdown machine

Use diskpart instead of EXTpart from Dell to expand partitions


To Determine disk performance problems, monitor two disk latency data counters
Kernel command latency: the avg time spent in the vmkernel per scsi command.
Physical device command latency


VDR - Vmware data recovery - Backups for small/medium sized environments (less than 1000 VM)
- 10 VDR appliances per virtual centre
- 100 VMs per appliance

Does de-duplication. Backs up to separate datastore.

-Limit of two deduplicated stores per appliance
-Per store: up to 1TB for vmdk or RDM.

Provide storage space equal to the amt of used disk space on all VMs.

Ensure that VMs backed up by VDR are appropriately licensed.

1) Install the client plug in
2) Install the backup appliance
3) Add a hard disk to the backup appliance

VCentre Linked Mode

Manage multiple VCentre servers from one console - up to 10 VCentre servers

dVS

Think of it as a template setup in virtual center that creates standard switches on esx hosts.

Create a dVS, Create a port group, assign an uplink NIC dvupling port to the dVS

Create another dVS and vmkernel port group and assign another dvuplink NIC for vmkernel

To setup vMotion - create another dVS switch, create a port group called vMotion, go to host, add a new virtual adapter - select option to use for Vmotion

Host Profiles

Create a reference host, export profiles, import to to other new hosts - consistent config

Select Cluster, click on Profile Compliance Tab, click check compliance now. See hosts that are not compliant



DRS

Affinity - set a rule that keeps two machines on the same host

Anti-affinity - keep machines on different esx hosts


HA

The HA agent monitors the hearbeats between the primary and secondary hosts to detect the host failure.

A heartbeat is sent every second (by default) over the heartbeat network.

On esxi hosts the mgmt network is used.
On esx hosts, the service console network is used.


Before changing the network config on the esx hosts, deselect Enable Host Monitoring in HA settings to prevent false host isolation actions

To view heartbeat logs and see if heartbeats are being sent, log in to 1 of first 5 esx hosts (they are dedicated as primary for heartbeats), ssh to esx host, type without quotes “tail -f /var/log/vmware/aam/vmware_esxservername.log



Fault Tolerance

Provides a higher level of uptime than HA. Zero downtime and data loss.

HA must be enabled.

A primary Vm is selected. Changes in memory are written to secondary VM. vLockstep technology is used to do this.

Setup another vmkernel port and switch to use with this as its extremely bandwidth intensive


Update Manager

1) Create baseline of patches to apply
2) Scan esx host of vm for compliance against baseline
3) Attach baseline to host(s)
4) Remediate means to apply the patches

Stage means to download the patches to the esx host or vm, but not apply them yet

You can patch a template with update manager - it will convert to vm, apply patches, then convert back to template for you


Management Network redundancy

Two options -

1) On vmkernel switch and port group, select option to enable management traffic

2) Team 2 physical NICs to existing vswitch that has management network


To view full esx version:   cat /proc/vmware/version |grep vmkernel


Powering off a frozen or unresponsive vm (virtual machine) on an esx host unresponsive virtual machine vm.pdf

Troubleshooting an invalid VM error and SCSI LUN locking

Vmware port usage

ESX3.x-VC2.x-ServiceConsole-Guide.pdf