let's call them the Media VM and Gaming VM,
Now that is a cool dad 
The hardware setup is one computer with one monitor, keyboard and mouse. I guess possible configurations could include:
- the hypervisor somehow allows to switch display between Media VM or Gaming VM
- or, the Media VM is always displayed on the monitor, and inside this a console or remote desktop tool can be used to show the Gaming VM
- or, two graphics cards are used so each VM can own (by passthrough) its own GPU and the monitor gets connected with one cable to each graphics card
You will not be able to achieve this by installing ESXi. However, you can get this resolved by installing VMware Workstation in the server PC connected to your TV.
On installing the Workstation, you can create multiple VMs and since the display is already routed to the TV, you can switch between the VMs as on need.
AFAIK, workstation does not provide you a passthrough, but it uses the underlying hardware itself. If yuou have a GPU, the same will be used by the VMs.
Added to this, if you have a separate laptop or PC, you can have your work VM running in the Workstation and the console can be redirected to your laptop/PC via LAN while the mediaVM or the GamingVM still runs in the TV screen. check out WSX