I recommend using these steps:
1. Upgrade vSphere first
Please note: you can not upgrade directly from 3.5 to 5.1. You need to update to ESX 4.1 first (see http://kb.vmware.com/kb/1022140) and then to ESXi 5.1 (see http://kb.vmware.com/kb/2032757)
2. Connect the new storage box. Create new datastores and format them as VMFS5.
3. Use Storage VMotion to migrate xour VMs from the old storage box (VMFS3 datastores) to the new VMFS5 datastores.
- Andreas