Issue with Nested ESXi and Multiple VMkernel Ports

While working with Nested ESXi in my lab, I had an issue where I could communicate with the IP address on vmk0, but after adding multiple additional VMkernel Ports could not communicate with any of the additional IP addresses. It’s a simple network for testing, everything on the same subnet and no VLANs involved.

I hadn’t done too much reading on the subject before, other than knowing I needed to implement Promiscuous Mode for the Port Group on the physical ESXi servers. It seemed strange that I could communicate with one of the addresses, but not the rest. I tracked down the following posts, but both suggested that only Promiscuous Mode need be enabled.

http://blog.paulregan.co.uk/2013/04/nested-esxi-install-second-vmkernel.html

http://vinf.net/2009/06/12/vsphere-esxi-as-a-vm-vmkernel-traffic-not-working/

I was running a Distributed Switch on the physical ESXi servers, so I tested moving one of the VMkernel ports to a Standard Switch with Promiscuous Mode enabled on the Port Group. It worked fine there, so was naturally curious why.

This communitites posting showed that Forged Transmits also needed to be enabled. The difference between the Standard and Distributed switches is that Forged Transmits is Accepted by default on a Standard switch

and Rejected by default on a Distributed switch

hence my experience above.

For more information check out these two posts from William Lam and Chris Wahl who are about two years ahead of me on this ;-)

http://www.virtuallyghetto.com/2013/11/why-is-promiscuous-mode-forged.html

http://wahlnetwork.com/2013/04/29/how-the-vmware-forged-transmits-security-policy-works/