1.Make&Model of ESX/ESXI ?
ESXi 5.1.0 799733
2.I mean spikes of 2000-3000ms every few minutes during busy periods .Are we seing this in Storage?
Under the Storage Adapter and Storage Path graphs.
3.You can follow http://kb.vmware.com/kb/1008205 and do post the esxtop ( DAVG-Device latency - Average amount of time, in milliseconds, to complete a SCSI command from the physical device-If DAVG value is going above 10 on a constant then we will have performance issues)
Jumps all over the place, 3, 6, around 10 and saw it spike to 37.44
4.Please post the kernel logs
ESX 3.5 and 4.x – /var/log/vmkernel
ESXi 3.5 and 4.x – /var/log/messages
ESXi 5.x - /var/log/vmkernel.log
2013-03-08T02:49:39.779Z cpu12:52738)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x4124428246c0, 5253) to dev "naa.600508b1001cd10a455c9eee86ca4c83" on path "vmhba1:C0:T0:L3" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-03-08T02:49:39.779Z cpu12:52738)ScsiDeviceIO: 2316: Cmd(0x4124428246c0) 0x85, CmdSN 0x46bcb from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T02:49:39.791Z cpu12:52738)ScsiDeviceIO: 2316: Cmd(0x4124428246c0) 0x4d, CmdSN 0x46bcc from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T02:49:39.793Z cpu12:52738)ScsiDeviceIO: 2316: Cmd(0x4124428246c0) 0x1a, CmdSN 0x46bcd from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2013-03-08T02:49:39.832Z cpu7:4577)<4>hpsa 0000:04:00.0: Device:C6:B0:T0:L1 Command:0x85 CC:05/20/00 Illegal Request.
2013-03-08T02:49:39.832Z cpu15:4111)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x4124428246c0, 5253) to dev "naa.600508b1001c032196ef616590f37bca" on path "vmhba1:C0:T0:L1" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-03-08T02:49:39.832Z cpu15:4111)ScsiDeviceIO: 2316: Cmd(0x4124428246c0) 0x85, CmdSN 0x46bce from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T02:49:39.843Z cpu0:4112)ScsiDeviceIO: 2316: Cmd(0x412402d50380) 0x4d, CmdSN 0x46bcf from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T02:49:39.843Z cpu0:4506)ScsiDeviceIO: 2316: Cmd(0x412402d50380) 0x1a, CmdSN 0x46bd0 from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2013-03-08T02:52:37.584Z cpu7:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T02:52:37.584Z cpu7:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T02:52:37.584Z cpu7:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T02:57:37.583Z cpu1:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T02:57:37.583Z cpu1:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T02:57:37.583Z cpu1:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T03:02:37.585Z cpu2:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T03:02:37.585Z cpu2:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T03:02:37.585Z cpu2:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T03:07:37.585Z cpu4:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T03:07:37.585Z cpu4:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T03:07:37.585Z cpu4:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T03:12:37.585Z cpu0:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T03:12:37.585Z cpu0:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T03:12:37.585Z cpu0:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T03:17:37.583Z cpu3:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T03:17:37.583Z cpu3:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T03:17:37.583Z cpu3:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T03:19:39.865Z cpu8:4104)<4>hpsa 0000:04:00.0: Device:C6:B0:T0:L3 Command:0x85 CC:05/20/00 Illegal Request.
2013-03-08T03:19:39.865Z cpu4:4100)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x412402df3680, 5253) to dev "naa.600508b1001cd10a455c9eee86ca4c83" on path "vmhba1:C0:T0:L3" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-03-08T03:19:39.865Z cpu4:4100)ScsiDeviceIO: 2316: Cmd(0x412402df3680) 0x85, CmdSN 0x47a1a from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T03:19:39.876Z cpu4:4100)ScsiDeviceIO: 2316: Cmd(0x412402df3680) 0x4d, CmdSN 0x47a1b from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T03:19:39.876Z cpu4:4100)ScsiDeviceIO: 2316: Cmd(0x412402df3680) 0x1a, CmdSN 0x47a1c from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2013-03-08T03:19:39.910Z cpu8:4104)<4>hpsa 0000:04:00.0: Device:C6:B0:T0:L1 Command:0x85 CC:05/20/00 Illegal Request.
2013-03-08T03:19:39.910Z cpu4:4100)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x412402df3680, 5253) to dev "naa.600508b1001c032196ef616590f37bca" on path "vmhba1:C0:T0:L1" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-03-08T03:19:39.910Z cpu4:4100)ScsiDeviceIO: 2316: Cmd(0x412402df3680) 0x85, CmdSN 0x47a1d from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T03:19:39.921Z cpu4:4100)ScsiDeviceIO: 2316: Cmd(0x412402df3680) 0x4d, CmdSN 0x47a1e from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T03:19:39.921Z cpu4:4100)ScsiDeviceIO: 2316: Cmd(0x412402df3680) 0x1a, CmdSN 0x47a1f from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2013-03-08T03:22:37.584Z cpu3:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T03:22:37.584Z cpu3:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T03:22:37.584Z cpu3:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T03:27:37.584Z cpu3:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T03:27:37.584Z cpu3:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T03:27:37.584Z cpu3:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T03:32:37.583Z cpu3:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T03:32:37.583Z cpu3:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T03:32:37.583Z cpu3:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T03:33:56.686Z cpu2:5879)<3>hpilo 0000:01:04.2: Closing, but controller still active
2013-03-08T03:37:37.585Z cpu3:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T03:37:37.585Z cpu3:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T03:37:37.585Z cpu3:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T03:42:37.584Z cpu3:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T03:42:37.584Z cpu3:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T03:42:37.584Z cpu3:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T03:45:39.756Z cpu8:5878)<3>hpilo 0000:01:04.2: Open could not dequeue a packet
2013-03-08T03:47:37.585Z cpu2:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T03:47:37.585Z cpu2:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T03:47:37.585Z cpu2:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T03:49:39.943Z cpu7:4103)<4>hpsa 0000:04:00.0: Device:C6:B0:T0:L3 Command:0x85 CC:05/20/00 Illegal Request.
2013-03-08T03:49:39.943Z cpu7:4103)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x412401c84ec0, 5253) to dev "naa.600508b1001cd10a455c9eee86ca4c83" on path "vmhba1:C0:T0:L3" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-03-08T03:49:39.943Z cpu7:4103)ScsiDeviceIO: 2316: Cmd(0x412401c84ec0) 0x85, CmdSN 0x48925 from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T03:49:39.953Z cpu7:4103)ScsiDeviceIO: 2316: Cmd(0x412401c84ec0) 0x4d, CmdSN 0x48926 from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T03:49:39.953Z cpu7:4103)ScsiDeviceIO: 2316: Cmd(0x412401c84ec0) 0x1a, CmdSN 0x48927 from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2013-03-08T03:49:39.976Z cpu7:4103)<4>hpsa 0000:04:00.0: Device:C6:B0:T0:L1 Command:0x85 CC:05/20/00 Illegal Request.
2013-03-08T03:49:39.976Z cpu7:4103)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x412401c84ec0, 5253) to dev "naa.600508b1001c032196ef616590f37bca" on path "vmhba1:C0:T0:L1" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-03-08T03:49:39.976Z cpu7:4103)ScsiDeviceIO: 2316: Cmd(0x412401c84ec0) 0x85, CmdSN 0x48928 from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T03:49:39.986Z cpu7:4103)ScsiDeviceIO: 2316: Cmd(0x412401c84ec0) 0x4d, CmdSN 0x48929 from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T03:49:39.986Z cpu7:4103)ScsiDeviceIO: 2316: Cmd(0x412401c84ec0) 0x1a, CmdSN 0x4892a from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2013-03-08T03:52:37.585Z cpu2:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T03:52:37.585Z cpu2:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T03:52:37.585Z cpu2:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T03:57:37.585Z cpu0:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T03:57:37.585Z cpu0:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T03:57:37.585Z cpu0:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T04:02:37.583Z cpu0:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T04:02:37.584Z cpu0:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T04:02:37.584Z cpu0:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T04:07:37.585Z cpu0:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T04:07:37.585Z cpu0:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T04:07:37.585Z cpu0:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T04:12:37.585Z cpu6:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T04:12:37.585Z cpu6:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T04:12:37.585Z cpu6:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T04:17:37.585Z cpu6:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T04:17:37.585Z cpu6:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T04:17:37.585Z cpu6:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T04:19:40.011Z cpu6:4102)<4>hpsa 0000:04:00.0: Device:C6:B0:T0:L3 Command:0x85 CC:05/20/00 Illegal Request.
2013-03-08T04:19:40.011Z cpu6:4102)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x412403964a40, 5253) to dev "naa.600508b1001cd10a455c9eee86ca4c83" on path "vmhba1:C0:T0:L3" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-03-08T04:19:40.011Z cpu6:4102)ScsiDeviceIO: 2316: Cmd(0x412403964a40) 0x85, CmdSN 0x49777 from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T04:19:40.021Z cpu6:4102)ScsiDeviceIO: 2316: Cmd(0x412403964a40) 0x4d, CmdSN 0x49778 from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T04:19:40.022Z cpu6:4506)ScsiDeviceIO: 2316: Cmd(0x412403964a40) 0x1a, CmdSN 0x49779 from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2013-03-08T04:19:40.053Z cpu6:4506)<4>hpsa 0000:04:00.0: Device:C6:B0:T0:L1 Command:0x85 CC:05/20/00 Illegal Request.
2013-03-08T04:19:40.053Z cpu6:4506)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x412403964a40, 5253) to dev "naa.600508b1001c032196ef616590f37bca" on path "vmhba1:C0:T0:L1" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-03-08T04:19:40.053Z cpu6:4506)ScsiDeviceIO: 2316: Cmd(0x412403964a40) 0x85, CmdSN 0x4977a from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T04:19:40.064Z cpu6:4506)ScsiDeviceIO: 2316: Cmd(0x412403964a40) 0x4d, CmdSN 0x4977b from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T04:19:40.064Z cpu6:4102)ScsiDeviceIO: 2316: Cmd(0x412403964a40) 0x1a, CmdSN 0x4977c from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2013-03-08T04:22:37.583Z cpu2:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T04:22:37.583Z cpu2:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T04:22:37.583Z cpu2:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T04:27:37.584Z cpu2:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T04:27:37.584Z cpu2:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T04:27:37.584Z cpu2:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T04:32:37.583Z cpu2:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T04:32:37.583Z cpu2:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T04:32:37.583Z cpu2:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T04:37:37.583Z cpu3:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T04:37:37.583Z cpu3:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T04:37:37.583Z cpu3:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T04:42:37.584Z cpu0:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T04:42:37.585Z cpu0:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T04:42:37.585Z cpu0:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T04:47:37.584Z cpu0:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T04:47:37.584Z cpu0:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T04:47:37.584Z cpu0:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T04:49:40.087Z cpu4:4100)<4>hpsa 0000:04:00.0: Device:C6:B0:T0:L3 Command:0x85 CC:05/20/00 Illegal Request.
2013-03-08T04:49:40.087Z cpu14:52738)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x412441cf0480, 5253) to dev "naa.600508b1001cd10a455c9eee86ca4c83" on path "vmhba1:C0:T0:L3" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-03-08T04:49:40.087Z cpu14:52738)ScsiDeviceIO: 2316: Cmd(0x412441cf0480) 0x85, CmdSN 0x4a59b from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T04:49:40.098Z cpu9:4105)ScsiDeviceIO: 2316: Cmd(0x412441cf0480) 0x4d, CmdSN 0x4a59c from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T04:49:40.098Z cpu9:4105)ScsiDeviceIO: 2316: Cmd(0x412441cf0480) 0x1a, CmdSN 0x4a59d from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2013-03-08T04:49:40.131Z cpu9:4105)<4>hpsa 0000:04:00.0: Device:C6:B0:T0:L1 Command:0x85 CC:05/20/00 Illegal Request.
2013-03-08T04:49:40.131Z cpu9:4105)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x412441cf0480, 5253) to dev "naa.600508b1001c032196ef616590f37bca" on path "vmhba1:C0:T0:L1" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-03-08T04:49:40.131Z cpu9:4105)ScsiDeviceIO: 2316: Cmd(0x412441cf0480) 0x85, CmdSN 0x4a59e from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T04:49:40.142Z cpu9:4105)ScsiDeviceIO: 2316: Cmd(0x412441cf0480) 0x4d, CmdSN 0x4a59f from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T04:49:40.142Z cpu9:4105)ScsiDeviceIO: 2316: Cmd(0x412441cf0480) 0x1a, CmdSN 0x4a5a0 from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2013-03-08T04:52:37.583Z cpu0:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T04:52:37.583Z cpu0:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T04:52:37.583Z cpu0:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T04:57:37.582Z cpu0:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T04:57:37.583Z cpu0:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T04:57:37.583Z cpu0:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T05:02:37.584Z cpu2:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T05:02:37.584Z cpu2:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T05:02:37.584Z cpu2:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T05:07:37.584Z cpu2:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T05:07:37.585Z cpu2:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T05:07:37.585Z cpu2:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T05:12:37.583Z cpu6:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T05:12:37.583Z cpu6:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T05:12:37.583Z cpu6:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T05:17:37.584Z cpu6:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T05:17:37.585Z cpu6:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T05:17:37.585Z cpu6:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T05:19:40.168Z cpu4:5884)<4>hpsa 0000:04:00.0: Device:C6:B0:T0:L3 Command:0x85 CC:05/20/00 Illegal Request.
2013-03-08T05:19:40.168Z cpu15:52738)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x4124403e36c0, 5253) to dev "naa.600508b1001cd10a455c9eee86ca4c83" on path "vmhba1:C0:T0:L3" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-03-08T05:19:40.168Z cpu15:52738)ScsiDeviceIO: 2316: Cmd(0x4124403e36c0) 0x85, CmdSN 0x4b3b5 from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T05:19:40.179Z cpu15:52738)ScsiDeviceIO: 2316: Cmd(0x4124403e36c0) 0x4d, CmdSN 0x4b3b6 from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T05:19:40.179Z cpu15:52738)ScsiDeviceIO: 2316: Cmd(0x4124403e36c0) 0x1a, CmdSN 0x4b3b7 from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2013-03-08T05:19:40.202Z cpu4:5884)<4>hpsa 0000:04:00.0: Device:C6:B0:T0:L1 Command:0x85 CC:05/20/00 Illegal Request.
2013-03-08T05:19:40.203Z cpu12:52740)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x4124403e36c0, 5253) to dev "naa.600508b1001c032196ef616590f37bca" on path "vmhba1:C0:T0:L1" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-03-08T05:19:40.203Z cpu12:52740)ScsiDeviceIO: 2316: Cmd(0x4124403e36c0) 0x85, CmdSN 0x4b3b8 from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T05:19:40.213Z cpu12:4108)ScsiDeviceIO: 2316: Cmd(0x4124403e36c0) 0x4d, CmdSN 0x4b3b9 from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T05:19:40.213Z cpu12:4108)ScsiDeviceIO: 2316: Cmd(0x4124403e36c0) 0x1a, CmdSN 0x4b3ba from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2013-03-08T05:22:37.584Z cpu0:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T05:22:37.585Z cpu0:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T05:22:37.585Z cpu0:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T05:27:37.583Z cpu1:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T05:27:37.583Z cpu1:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T05:27:37.583Z cpu1:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T05:32:37.583Z cpu3:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T05:32:37.583Z cpu3:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T05:32:37.583Z cpu3:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T05:37:37.584Z cpu8:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T05:37:37.585Z cpu8:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T05:37:37.585Z cpu8:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T05:42:37.583Z cpu12:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T05:42:37.583Z cpu10:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T05:42:37.583Z cpu10:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T05:47:37.582Z cpu9:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T05:47:37.584Z cpu9:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T05:47:37.584Z cpu9:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T05:49:40.235Z cpu4:5884)<4>hpsa 0000:04:00.0: Device:C6:B0:T0:L3 Command:0x85 CC:05/20/00 Illegal Request.
2013-03-08T05:49:40.235Z cpu14:4110)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x4124403dba00, 5253) to dev "naa.600508b1001cd10a455c9eee86ca4c83" on path "vmhba1:C0:T0:L3" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-03-08T05:49:40.235Z cpu14:4110)ScsiDeviceIO: 2316: Cmd(0x4124403dba00) 0x85, CmdSN 0x4c201 from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T05:49:40.246Z cpu14:4110)ScsiDeviceIO: 2316: Cmd(0x4124403dba00) 0x4d, CmdSN 0x4c202 from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T05:49:40.246Z cpu14:4110)ScsiDeviceIO: 2316: Cmd(0x4124403dba00) 0x1a, CmdSN 0x4c203 from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2013-03-08T05:49:40.268Z cpu4:5884)<4>hpsa 0000:04:00.0: Device:C6:B0:T0:L1 Command:0x85 CC:05/20/00 Illegal Request.
2013-03-08T05:49:40.268Z cpu14:4110)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x4124403dba00, 5253) to dev "naa.600508b1001c032196ef616590f37bca" on path "vmhba1:C0:T0:L1" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-03-08T05:49:40.268Z cpu14:4110)ScsiDeviceIO: 2316: Cmd(0x4124403dba00) 0x85, CmdSN 0x4c204 from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T05:49:40.279Z cpu14:4110)ScsiDeviceIO: 2316: Cmd(0x4124403dba00) 0x4d, CmdSN 0x4c205 from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T05:49:40.279Z cpu14:4110)ScsiDeviceIO: 2316: Cmd(0x4124403dba00) 0x1a, CmdSN 0x4c206 from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2013-03-08T05:52:37.584Z cpu9:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T05:52:37.585Z cpu10:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T05:52:37.585Z cpu10:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T05:57:37.584Z cpu10:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T05:57:37.584Z cpu10:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T05:57:37.584Z cpu10:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T06:02:37.583Z cpu10:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T06:02:37.583Z cpu10:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T06:02:37.583Z cpu10:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T06:07:37.584Z cpu10:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T06:07:37.584Z cpu10:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T06:07:37.584Z cpu10:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T06:12:37.583Z cpu10:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T06:12:37.584Z cpu10:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T06:12:37.584Z cpu10:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T06:15:30.697Z cpu14:5877)<3>hpilo 0000:01:04.2: Closing, but controller still active
2013-03-08T06:17:37.583Z cpu10:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T06:17:37.583Z cpu14:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T06:17:37.583Z cpu14:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T06:19:40.304Z cpu4:4100)<4>hpsa 0000:04:00.0: Device:C6:B0:T0:L3 Command:0x85 CC:05/20/00 Illegal Request.
2013-03-08T06:19:40.304Z cpu12:52738)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x412442771e00, 5253) to dev "naa.600508b1001cd10a455c9eee86ca4c83" on path "vmhba1:C0:T0:L3" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-03-08T06:19:40.304Z cpu12:52738)ScsiDeviceIO: 2316: Cmd(0x412442771e00) 0x85, CmdSN 0x4d0b0 from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T06:19:40.315Z cpu12:52738)ScsiDeviceIO: 2316: Cmd(0x412442771e00) 0x4d, CmdSN 0x4d0b1 from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T06:19:40.315Z cpu12:52738)ScsiDeviceIO: 2316: Cmd(0x412442771e00) 0x1a, CmdSN 0x4d0b2 from world 5253 to dev "naa.600508b1001cd10a455c9eee86ca4c83" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2013-03-08T06:19:40.340Z cpu4:4506)<4>hpsa 0000:04:00.0: Device:C6:B0:T0:L1 Command:0x85 CC:05/20/00 Illegal Request.
2013-03-08T06:19:40.340Z cpu9:4105)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x412442771e00, 5253) to dev "naa.600508b1001c032196ef616590f37bca" on path "vmhba1:C0:T0:L1" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-03-08T06:19:40.340Z cpu9:4105)ScsiDeviceIO: 2316: Cmd(0x412442771e00) 0x85, CmdSN 0x4d0b3 from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T06:19:40.351Z cpu9:4105)ScsiDeviceIO: 2316: Cmd(0x412442771e00) 0x4d, CmdSN 0x4d0b4 from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-03-08T06:19:40.351Z cpu9:4105)ScsiDeviceIO: 2316: Cmd(0x412442771e00) 0x1a, CmdSN 0x4d0b5 from world 5253 to dev "naa.600508b1001c032196ef616590f37bca" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2013-03-08T06:22:37.584Z cpu10:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T06:22:37.584Z cpu12:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T06:22:37.584Z cpu12:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T06:26:19.904Z cpu4:4100)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0xad (0x4124003a2d80, 5881) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x30 0x2. Act:NONE
2013-03-08T06:26:19.904Z cpu4:4100)ScsiDeviceIO: 2316: Cmd(0x4124003a2d80) 0xad, CmdSN 0x4d439 from world 5881 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x30 0x2.
2013-03-08T06:26:19.959Z cpu4:4100)ScsiDeviceIO: 2316: Cmd(0x4124003a2d80) 0xad, CmdSN 0x4d446 from world 5881 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x30 0x2.
2013-03-08T06:26:20.011Z cpu4:4100)ScsiDeviceIO: 2316: Cmd(0x4124003a2d80) 0xad, CmdSN 0x4d454 from world 5881 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x30 0x2.
2013-03-08T06:26:20.025Z cpu4:4100)ScsiDeviceIO: 2316: Cmd(0x4124003a2d80) 0xad, CmdSN 0x4d457 from world 5881 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x30 0x2.
2013-03-08T06:26:20.070Z cpu4:4100)ScsiDeviceIO: 2316: Cmd(0x4124003a2d80) 0xad, CmdSN 0x4d464 from world 5881 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x30 0x2.
2013-03-08T06:26:20.123Z cpu4:4100)ScsiDeviceIO: 2316: Cmd(0x4124003a2d80) 0xad, CmdSN 0x4d471 from world 5881 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x30 0x2.
2013-03-08T06:26:20.177Z cpu4:4100)ScsiDeviceIO: 2316: Cmd(0x4124003a2d80) 0xad, CmdSN 0x4d47f from world 5881 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x30 0x2.
2013-03-08T06:26:20.187Z cpu4:4100)ScsiDeviceIO: 2316: Cmd(0x4124003a2d80) 0xad, CmdSN 0x4d482 from world 5881 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x30 0x2.
2013-03-08T06:26:20.249Z cpu4:4100)ScsiDeviceIO: 2316: Cmd(0x4124003a2d80) 0xad, CmdSN 0x4d495 from world 5881 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x30 0x2.
2013-03-08T06:26:20.311Z cpu4:4577)ScsiDeviceIO: 2316: Cmd(0x4124003a2d80) 0xad, CmdSN 0x4d4a2 from world 5881 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x30 0x2.
2013-03-08T06:26:20.370Z cpu7:4103)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0xad (0x412402d55f80, 5881) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x30 0x2. Act:NONE
2013-03-08T06:26:20.370Z cpu7:4103)ScsiDeviceIO: 2316: Cmd(0x412402d55f80) 0xad, CmdSN 0x4d4b2 from world 5881 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x30 0x2.
2013-03-08T06:26:20.380Z cpu7:4103)ScsiDeviceIO: 2316: Cmd(0x412402d55f80) 0xad, CmdSN 0x4d4b5 from world 5881 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x30 0x2.
2013-03-08T06:26:20.443Z cpu7:4103)ScsiDeviceIO: 2316: Cmd(0x412402d55f80) 0xad, CmdSN 0x4d4c8 from world 5881 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x30 0x2.
2013-03-08T06:26:20.503Z cpu7:4103)ScsiDeviceIO: 2316: Cmd(0x412402d55f80) 0xad, CmdSN 0x4d4d6 from world 5881 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x30 0x2.
2013-03-08T06:26:20.566Z cpu7:4103)ScsiDeviceIO: 2316: Cmd(0x412402d55f80) 0xad, CmdSN 0x4d4e4 from world 5881 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x30 0x2.
2013-03-08T06:27:37.582Z cpu9:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T06:27:37.583Z cpu9:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T06:27:37.583Z cpu9:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.
2013-03-08T06:32:37.584Z cpu9:4684)<6>hpsa 0000:04:00.0: Direct-Access device c6b0t0l2 added.
2013-03-08T06:32:37.584Z cpu9:4684)WARNING: LinScsiLLD: scsi_add_device:1527:Failed to add device on channel=0, id=0, lun=2
2013-03-08T06:32:37.584Z cpu9:4684)<4>hpsa 0000:04:00.0: scsi_add_device c6b0t0l2 failed, device not added.