Category Archives: Uncategorized

ESXi on ARM: Deploying my smallest home lab server after my largest server deployment

Over the summer I deployed a large enterprise SuperMicro Server with a half terabyte of RAM and 36 cores provided by 2 Intel Xeon E5-2683 v4’s. I deployed a nested VMware Cloud Foundation 4 with Tanzu Kubernetes Grid on this system and I’m still learning. My last blog post links to a YouTube presentation on my experience

The massive nested VCF server in my garage

I learned through Twitter yesterday morning that VMware released an ESXi on Arm Fling free technology preview. I ordered a new Raspberry Pi 4B with 8GB of RAM from Amazon in the morning and had ESXi live on the system by the end of the day. The Raspberry Pi is close to the size of one of the Intel Xeon processors in the SuperMicro VMware Cloud Foundation Server I deployed over the summer. The electrical power requirements for the Raspberry Pi is insignificant compared to the SuperMicro enterprise server running VMware Cloud Foundation.

Raspberry Pi 4B 8GB RAM circled in red inside my half rack

Kit Colbert published a blog last week describing use cases for this game changing technology. In addition, he presented on it in the “The Datacenter of the Future [HCP3004]” session last week during VMworld. VMworld session recordings are available through the site and registration is free. I was excited to gain hands on experience.

WOW – this technology is amazing. After deploying ESXi to a USB memory stick, I connected this host to my vCenter server. Next I created and connected an NFS datastore from my QNAP NAS to the host.

ESXi on ARM alive in my vCenter

I pulled out my iPad mini and saw the new host in the vSphere Client fling.

rpi1-esxi is the Raspberry PI on the vSphere Client Fling for iPad

I downloaded and deployed the ARM version of Ubuntu 20.04 and RHEL 8.2 as VM’s. I compiled VMware Tools on the Ubuntu VM and installed the GUI ( to both Linux VM’s. I still have a little memory to spare on the Raspberry Pi for another VM. Both of the VM’s were responsive even with the GUI. It is hard to tell that ESXi and Linux is running on ARM since the Operating Systems are unchanged. The largest obstacle is the ARM requirement for software. I now understand through this experience why Apple is rumored to release a MacBook with an ARM processor.

Ubuntu VM on left, Red Hat VM on right

Over the summer I learned about the reimagined VMware Cloud Foundation from the top down with Tanzu Kubernetes Grid, NSX-T, vSAN, and SDDC Manager. Now with ESXi on ARM I am learning about the next chapter of VMware Cloud Foundation. If you always wanted a VMware vSphere home lab this is the most inexpensive path to get started.

Don’t let the Bear Hibernate

Black bear increasing energy stores in the authors backyard for hibernation last winter

I’ve continued contributing computing resources non-stop to science researchers since my March post. A byproduct is learning how my home lab operates at full throttle and the energy implications. My last blog discussed some of my original sustainability learnings.

Impact to host @ 95% throttle

VMware vRealize Operations Manager providing additional compute capacity

I drove CPU usage to approximately 95% when I started to donate all of my excess compute capacity. Shortly after operating at full throttle, an alert popped up in VMware vRealize Operations Manager 8.0 console. This alert provided proactive performance improvement recommendation – and an idea for this blog post.

VMware vRealize Operations Manager 8.0 alert. VMware knowledge base article explaining how to fix the issue shown above.

I learned that the most energy efficient setting for my home lab servers was to turn off all of the processor energy savings features. This lesson was counter intuitive. Once my home lab was operating at full utilization the servers wasted processing power and energy by attempting to turn on power saving features. The default server configuration assumed that the current task was a momentary spike in demand. Once the sprint was over, the processor would start shutting down excess capacity. Due to the high utilization, another spike in demand quickly arrived and the processor would switch to maximum capacity. Now the processor would need to ramp up. This incorrect assumption led to a reduction in processing capacity and slowed the scientific research workload. The energy consumption didn’t decrease but the amount of work completed was reduced.

Who Should Sleep?

Sleep states and hibernation for bears and computers are necessary to save energy stores when nothing is happening. Both species go through a “waking-up” state which takes time and energy. Our Pacific Northwest bears benefit from powering off unnecessary functions in the winter but a server processor at full capacity does not. This only slows down the workload while wasting energy which isn’t a sustainable solution.

Turning Off Power Saving Features

The 3 SuperMicro SuperServer E300-8D’s in my home lab have rudimentary power management features. The p-state and c-state features allow processors to shutdown excess capacity. This feature is similar to an energy efficient pickup truck engine which turns off pistons that aren’t needed at highway cruising speed. Following are the default AMI BIOS p-state and c-state settings for these servers. I have disabled both settings that are highlighted.

Sustainable Configuration

The alerts stopped once I configured the servers for compute intensive workloads running non-stop. Enterprise servers are complex and default settings reduce the time and understanding required to stand-up infrastructure. VMware vRealize Operations Manager highlighted this mis-configuration which I wouldn’t have found otherwise. This is one example of many where this tool has pointed out hidden problems and taught me something new. I never expected that turning off all power management features is the most sustainable option.