Source: https://vmscrub.com/installing-macos-12-monterey-on-vmware-esxi-7-update-3/

So it’s been a little over a year since the last macOS on ESXi install guide I’ve written for Big Sur and VMware ESXi 7 Update 1. Subsequently, Apple has released macOS 12 Monterey and VMware has updated ESXi to 7 Update 3.

Although mostly unchanged from the previous guide, I thought I’d write an updated and more simplistic guide to running macOS on vSphere 7.

Below you’ll find the step-by-step instructions on how I got this installed in my home lab.

DISCLAIMER

This should only be run in a lab environment and not in a production environment. Don’t be that person!

– Please note: This guide is written for an Intel compute based machine running a Type 1 bare metal hypervisor and not a Type 2 like VMware Workstation or VMware Fusion. –

Prerequisites

First we’ll need the above 3 items in order to complete this whole process. Relax! I’ll hold your hand through all the steps in obtaining these prerequisites.

VMware ESXi 7 Update 3

1. Download VMware ESXi 7 Update 3

So first things first! We gotta grab the latest ESXi version and install it prior to installing macOS 12. Go to the downloads page here and select the version of ESXi you have and want to install/update.

2. Install/Upgrade VMware ESXi 7 Update 3

Once you’ve selected the correct install for your licensed version of vSphere, you’ll simply need to install it. I’ve written a previous guide on how to Upgrade from an offline bundle here. This is the path I went with in order to update the current ESXi install in my lab.

macOS 12 Monterey ISO

Next we’ll need to obtain the macOS 12 Monterey ISO. I’m sure it’ll be easy to search the net and find the ISO, but ensure you get it from a valid trusted source. Otherwise make sure you validate the SHA-1, CRC and MD5 values of the image to make sure it hasn’t been tampered with before installing it.

I’ll take the safer route and create it myself using a macOS native machine. If you have access to a MacBook, iMac or Mac mini already on Monterey, you should be able to create an ISO from any one of those machines.

Can’t you hook a brotha up and let me direct download the ISO from your site?

1. Download macOS 12 Monterey Installer

Once you have a macOS native machine running Monterey in front of you, navigate to the App Store here to download the obtain the Monterey installer. Once you click the link, you should see a browser pop-up asking you to open App Store.app.

From there, the App Store app should open up to the Monterey page. click on the Get button to download Monterey.

Once you click on Get you’ll be greeted with a download prompt. Click on Download then wait for the download to complete. Download times will vary based on your network speed.

When the download is finished, you’ll be prompted with the install window to Continue. However…

DO NOT CLICK CONTINUE!

Instead, simply press Command + Q to at this install window. Once you do, simply click on Quit at the prompt to close the window.

If you did everything correctly, you should now have the Install macOS Monterey.app listed in your Application Folder. This app is what we’ll use to create our ISO in the next step.

2. Mount Installer then Convert to ISO

Now that we have the Monterey App, we’ll run a few commands to first mount the installer then convert it to an ISO with the purpose of mounting it to a VM in ESXi.

Launch the terminal and run the following command.

sudo -i

Running this command will prompt for the administrator password. Once you type it in, you’ll now be logged in as the root user which will be the account we’ll use to execute the rest of the commands.

From the root# prompt, enter in the following command:

hdiutil create -o /tmp/monterey -size 13800.1m -volname monterey -layout SPUD -fs HFS+J

This will create the directory /tmp/monterey.dmg

** This 13800.1m size for this guide was based on the version 12.2 of macOS and may be different for any updates/patches on future 12.x versions. **

After this we’ll need to mount the dmg using the following command:

hdiutil attach /tmp/monterey.dmg -noverify -mountpoint /Volumes/monterey

You should see the following output below:

You should now also be able to see the mounted disk under Disk Utility.

Next we’ll create the install media from our new mount with this command:

sudo /Applications/Install\ macOS\ Monterey.app/Contents/Resources/createinstallmedia –volume /Volumes/monterey –nointeraction

You should see the following result below while making the disk bootable.

Afterwards, you can eject the attached install disk using the following command:

hdiutil eject -force /volumes/Install\ macOS\ Monterey

You should see that the install disk is ejected now.

Next we’ll need to convert the image to a .cdr with the following command:

hdiutil convert /tmp/monterey.dmg -format UDTO -o /Users/erickdimalanta/Desktop/monterey.cdr

Don’t forget to change the path /Users/erickdimalanta/Desktop/monterey.cdr to your own specific path you are working out of. Once you run the command, you should now see a monterey.cdr file in your working directory that is about 14.47 GB.

Now that we have our media, we simply need to rename the file extension to a .iso. You can do so with the command below by running an in-place mv(move) command.

mv /Users/erickdimalanta/Desktop/monterey.cdr /Users/erickdimalanta/Desktop/monterey.iso

Again, make sure you set “YOUR” correct path if you are copying and pasting the command above.

3. Cleaning Up

Finally let’s clean up after ourselves since leaving remnants of leftover data is just icky!

We are going to clean up the initial directory where we created the monterey.dmg in the following path /tmp/monterey.dmg. Use the below command to cleanup.

rm -rf /tmp/mont*

This above command will make sure to force delete everything in the specified directory without prompting for confirmation. So basically we are removing everything in the /tmp folder that starts with “mont.” Make sure nothing else starts with “mont” otherwise you’ll remove that also.

The below screenshot shows how to simply validate and double check afterwards.

4. Upload Monterey ISO to Datastore

For the last step in this section, we’ll need to transfer the newly created Monterey.iso to our VMware environment. You can upload it to a datastore or Content Library if you are also running a vCenter. To keep it simple, I’ll showcase uploading it to a datastore.

Login to your ESXi host
Navigate to Storage and select the Datastore you want to place the .iso in.
Click on Datastore Browser
Select the Datastore
Select the folder you’ll place the .iso in. I’ll use the ISO folder.
Then click on Upload

Navigate to the directory where you created the monterey.iso. Select the monterey.iso and click on Open.

Review the Recent Tasks pane, do a run around the block, then monitor to completion.

macOS Unlocker Installation

Once the .iso is finally uploaded to the datastore, it’s now time to prep our ESXi host with the ability to run macOS in a VM. The unlocker is written python that modifies the vmware-vmx file to allow macOS to boot. Without this unlocker, the machine simply doesn’t work and is suck in a sad infinite boot loop.

1. Download the Unlocker

You’ll first need to download the unlocker zip file here.

The above link should be a direct download to the unlocker from github. The file name is esxi-unlocker-master.zip.

2. Upload to Datastore

Once you’ve downloaded the esxi-unlocker-master.zip, you’ll need to upload this to the same location we uploaded the monterey.iso in a previous step above.

3. SSH into ESXi host to Install the Unlocker

Bring up your terminal if you’ve previously closed it and SSH into your host.

– Make sure you enable SSH on your host if disabled. –

From there, login to your ESXi host then navigate to your unlocker location as shown by the screenshot below.

Once in the parent location of the unlocker, we’ll need to unzip the esxi-unlocker-master.zip file. You can do so with the command below:

unzip esxi-unlocker-master.zip

The below image should be your resulted output.

4. Set permissions on the extracted folder

Before we navigate to the newly extracted folder and install the patch. We’ll need to set the permissions on the folder and files within. Not doing so will result in a “Permission denied” error as shown below.

So since I’m here to save you the headache so that this install guide works the first time.

Enter the following command below:

chmod 775 -R esxi-unlocker-301/

This command will grant permissions to the folder and all the files recursively within.

5. Install Unlocker Patch

Once you’ve set the permissions on the folder and files above, we can now install the patch. First navigate to the extracted folder that contains all the files necessary for the patch.

Enter in the following command:

cd esxi-unlocker-301/

Once there, we can actually use a patch validation checker to make sure the patch is installed or not. I know we haven’t installed the patch yet, but let’s just run that first to see the resulted output.

Run the following command to validate:

./esxi-smctest.sh

After running the command, it will obviously show false since we’ve not yet installed the patch. Well let’s do it now!

Run the following command to install the patch:

./esxi-install.sh

Once you’ve run the install ./esxi-install.sh, we’ll need to now restart the server for the configuration changes to take effect.

**Make sure you have no running VMs before typing reboot. **

In the same terminal window, you can simply type reboot then press Enter.

reboot

6. Validate Successful Patch Install

When the ESXi host is back up from it’s reboot, let’s SSH into the host again and navigate to the same location where we ran the patch installer.

Once there run the validation command again to now see a different result.

./esxi-smctest.sh

This should now display smcPresent = true which means we are now ready to install macOS without running into any boot issues.

macOS 12 Monterey VM Install

Whew! It’s about time amirite?

If you’ve followed the above steps correctly, you should now be able to continue on and install Monterey!

1. Create a New Virtual Machine

Go back to your ESXi host and navigate to the Virtual Machine menu item on the left. Then click on Create / Register VM.

Another screen will pop up. From there select Create a new virtual machine and click on Next.

Next, type out a name for your new VM. Select the following options from the drop down menu for the rest of the categories.

Name: ANY NAME SHORTER THAN 80 CHARACTERS
Compatibility: ESXi 7.0 U2 virtual machine (U3 wasn’t available despite being on U3)
Guest OS Family: Mac OS
Guest OS version: Apple macOS 12 (64-bit)

Finally, click Next when finished.

From here, select which datastore you’ll want to place your machine in.

The next option will be to configure the hardware specs of the VM.

You are going to want to set your CPU, Memory, Storage size, type of provisioning, Network, and most importantly configure your CD/DVD Drive 1 to boot from the monterey.iso we’ve created. This is located in the ISO folder on the datastore we uploaded it to in an earlier step.

Click on Next once done.

Finally, confirm all your settings are what you want, then click on Finish when complete.

2. Power On VM & Format VM Storage

Sweet! Now that you are done creating your VM, let’s power this bad boy on to see if it bypasses the boot loop error since we applied the Unlocker patch.

Boom! Success! Boot loop is no more!

Anyway, continuing on with the install. Select your language of choice and click on the bottom right arrow. Next we’ll need to format the newly provisioned disk so that macOS can recognize the storage to install itself onto.

Select Disk Utility and click on Continue.

Next, highlight the VMware Virtual SATA Hard Drive Media and click on the Erase option at the top.

Enter in a Name to label your disk and keep the defaults for Format – APFS and Scheme – GUID Partition Map.

Click the Erase button when ready

The formatting process will proceed. Once complete, click Done.

We can now quit Disk Utility by navigating to the top menu and clicking on Disk Utility > Quit Disk Utility.

3. Install macOS 12 Monterey

Now we can install the OS. From here select Install macOS Monterey then click on Continue.

On the next screen click Continue

Click Agree twice for the Software License Agreement

Click on the newly formatted HDD then click on Continue.

macOS Monterey will start to install now…

Don’t believe the time remaining is even close to accurate. I would go bake a frozen pizza and watch a movie or something, then come back waaaayyyyy later.

When the never ending string of installation reboots are completed, the install will be complete! Again, I won’t go through this self-explanatory setup wizard.

Install VMware Tools (Optional but necessary)

Finally, the last thing to do after you’ve gone through the initial OS setup wizard, is to install VMware tools. You can also unmount the monterey.iso from the VM at this point since it’s no longer needed.

Wut the heck are VMware tools?

1. VMware Tools Error 21001

You’ll likely receive the below error when attempting to mount the VMware Tools from the host.

Failed – The required VMware Tools ISO image does not exist or is inaccessible.

This occurs because the darwin.iso doesn’t exist in the directory “/usr/lib/vmware/isoimages.“ Which is fine as these are the iso images that shipped with this version of ESXi that I installed. According to the KB 2129825 we have a few options in resolving this. I’m going to simply go with the option to download the latest VMware tools for this specific OS.

2. Download VMware Tools

Make sure you are logged into your Customer Connect account. Then click Here to Download the latest VMware Tools. I selected the .zip option.

Next, navigate to your download location and extract the .zip. On macOS you can simply double-click the file to extract.

Sweet! Now you should have the darwin.iso image we need to install VMware Tools onto our macOS 12 Virtual Machine.

3. Mount VMware Tools ISO to VM

Now that we have the VMware Tools image for macOS, we’ll need to upload the .iso to a datastore the same way we Uploaded the monterey.iso in an earlier step.

Once the iso is uploaded there, Edit settings on the VM and select the darwin.iso to mount just like we did when we were creating the VM. When done, click Save.

So…who in the heck is Darwin?

4. Install VMware Tools in Monterey

Next we’ll hop back over to our VM and we can now see that VMware Tools is mounted and ready for install.

Double click on the Install VMware Tools icon. From there the Tools installer will launch. Click on Continue to proceed.

Click Install on the next screen

You’ll then be prompted to enter in the password you set when you went through the initial OS setup wizard. Once done, click on Install Software. Then click OK on the administer prompt.

You’ll then likely receive a System Extension Blocked prompt during this process. Click on Open Security Preferences. At this point the Security & Privacy window will open up. Once opened, click on the lock in the bottom left and enter in your OS password, then click Unlock.

Once done and the lock icon is unlocked, click on the Allow button to allow VMware Tools to continue loading.

You should now receive a prompt to Restart before using the system extensions. But before doing so, click on the open Installer in the dock below.

The VMware Tools window should showcase that the installation was successful. Go back to the Security & Privacy window in the background and click on Restart from there.

Once the machine has rebooted, you should now see that VMware Tools are installed for this VM. From there, you can either eject the mounted VMware Tools image from the OS or disconnect it from the CD/DVD Drive 1 option in the VM under Edit settings.

Schweet! All done! You should now have a macOS 12 environment that you can test and break all you want! Have fun!