Hey guys,
With a fresh VDP 6.1.3 install on top of vSphere 6.5, most backup jobs hang at 92%. This is not the issue documented in the release notes, and the backups almost never complete. I DID NOT have this issue with VDP 6.1 under vSphere 6.0.
I have tried backing up photon OS machines, centos 6.5 machines, centos 6.6+ machines, and centos 7 machines. So far the only ones to successfully complete are some of those old centos 6.5 ones. I'm not sure if the O/S version is a red herring here, or if it's related to some other pattern I just can't identify at the time.
I have tried backing up images with the quiesce option both on and off, and it doesn't seem to make a difference in the result. The backups hang regardless and are unsuccessful.
I should mention the storage was created along with the appliance and is on VSAN, if that makes any difference, but - this absolutely all worked with VDP 6.1 and vSphere 6.0.
I looked in a failed job's vmimage log in /usr/local/avamarclient/var/ and noticed this at the end:
2017-01-17T04:02:15.693+04:00 avvcbimage Info <9679>: Virtual disk connection to '[vsanDatastore-Neo-Cloud] 50c54958-6741-9944-301e-001ec93c076e/photonos-2cpu-4gb-is-vcpsc1.vmdk' has been closed
2017-01-17T04:02:15.693+04:00 avvcbimage Info <19718>: Found Disk from Device Key: '2001'.
2017-01-17T04:02:15.693+04:00 avvcbimage Info <9674>: virtual disk [vsanDatastore-Neo-Cloud] 50c54958-6741-9944-301e-001ec93c076e/photonos-2cpu-4gb-is-vcpsc1_1.vmdk will be connected readonly
2017-01-17T04:02:15.693+04:00 avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLib_OpenEx: Open a disk.2017-01-17T04:02:15.693+04:00 avvcbimage Info <16041>: VDDK:VixDiskLibVim: VixDiskLibVim_GetNfcTicket: Get NFC ticket for [vsanDatastore-Neo-Cloud] 50c54958-6741-9944-301e-001ec93c076e/photonos-2cpu-4gb-is-vcpsc1_1.vmdk.
2017-01-17T04:02:16.008+04:00 avvcbimage Info <16041>: VDDK:VixDiskLibVim: Request RandomAccessRO diskKey = 2001, readOnly = 1, openSnapshot = 0.
2017-01-17T04:02:16.094+04:00 avvcbimage Info <16041>: VDDK:VixDiskLibVim: VixDiskLibVim_FreeNfcTicket: Free NFC ticket.
2017-01-17T04:02:16.101+04:00 avvcbimage FATAL <5889>: Fatal signal 11 in pid 15065
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] FATAL ERROR: <0001> uapp::handlefatal: Fatal signal 11
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 00000000006b93b1
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 00000000006ba127
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 00000000006ba44b
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 00000000006ba53e
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 0000000000628810
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 00007f2e56c90850
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 00007f2e5050193f
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 00007f2e5051c277
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 00007f2e5051c75e
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 00007f2e5052b142
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 00007f2e5052b678
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 00007f2e56ecab73
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 00007f2e56ecb123
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 00000000004284bc
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 00000000004291f9
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 00000000004993c4
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 000000000070ac4c
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 00007f2e56c88806
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] | 00007f2e558cd9bd
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread] ERROR: <0001> uapp::handlefatal: aborting program pid=15065, sig=11
2017-01-17T04:02:16.103+04:00 avvcbimage FATAL <5890>: handlefatal: Aborting program with code 176, pid=15065, sig=11
2017-01-17T04:02:16.103+04:00 avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLib_Exit: Unmatched Init calls so far: 1.2017-01-17T12:00:07.999+04:00 avvcbimage Info <40660>: vcbimage_progress::terminate
the log ends there.
This looks serious to me, but it's hard to figure out where to go from here.
Does anyone have any advice as to how to proceed?
Thanks for any assistan