We recently changed domain administrator password and stop the domain administrator permission to the vCentre. Looks like now the VDP machine stop working.
The symptoms are: cannot access VDP via web, only ssh.
I looked trough the vdr-server.log and there are a lot of these errors:
2015-04-15 14:30:05,470 INFO [com.emc.vdr.server.VDRServer$1]-server.ConnectionService: Trying to establish connection with vCenter.
2015-04-15 14:30:05,471 INFO [com.emc.vdr.server.VDRServer$1]-server.VDRExternalServerAccessConfig: loading default values forcefully: true
2015-04-15 14:30:05,471 INFO [com.emc.vdr.server.VDRServer$1]-server.VDRExternalServerAccessConfig: Get mc access...
2015-04-15 14:30:05,480 INFO [com.emc.vdr.server.VDRServer$1]-service.AdapterUtils: MCS Web Services URL: https://hydra.ecom90.co.uk:9443/services/mcService MCUserId="MCUser" MCUserPswd="*****************************"
2015-04-15 14:30:05,492 INFO [com.emc.vdr.server.VDRServer$1]-httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
2015-04-15 14:30:05,492 INFO [com.emc.vdr.server.VDRServer$1]-httpclient.HttpMethodDirector: Retrying request
2015-04-15 14:30:05,513 INFO [com.emc.vdr.server.VDRServer$1]-httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
2015-04-15 14:30:05,513 INFO [com.emc.vdr.server.VDRServer$1]-httpclient.HttpMethodDirector: Retrying request
2015-04-15 14:30:05,514 INFO [com.emc.vdr.server.VDRServer$1]-httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
2015-04-15 14:30:05,514 INFO [com.emc.vdr.server.VDRServer$1]-httpclient.HttpMethodDirector: Retrying request
2015-04-15 14:30:05,514 INFO [com.emc.vdr.server.VDRServer$1]-http.HTTPSender: Unable to sendViaPost to url[https://hydra.ecom90.co.uk:9443/services/mcService?wsdl]
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(Unknown Source)
at com.emc.avamar.mcsdk.adapter.service.TrustAllSSLProtocolSocketFactory.createSocket(TrustAllSSLProtocolSocketFactory.java:128)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:542)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:199)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:400)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:225)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:435)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:402)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
at com.avamar.mc.sdk.client.McServiceStub.login(McServiceStub.java:5084)
at com.avamar.mc.sdk.client.McServiceClient.login(McServiceClient.java:119)
at com.avamar.mc.sdk.client.McServiceClient.connect(McServiceClient.java:105)
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.connect(ServiceInstance.java:313)
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.<init>(ServiceInstance.java:225)
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.<init>(ServiceInstance.java:187)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromAdapterDefault(VDRExternalServerAccessConfig.java:67)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromPropsDefault(VDRExternalServerAccessConfig.java:126)
at com.emc.vdr.server.ConnectionService.run(ConnectionService.java:40)
at java.lang.Thread.run(Unknown Source)
2015-04-15 14:30:05,514 ERROR [com.emc.vdr.server.VDRServer$1]-service.AdapterUtils: ServiceInstance failed to establish connection
org.apache.axis2.AxisFault: Connection refused
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:203)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:400)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:225)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:435)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:402)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
at com.avamar.mc.sdk.client.McServiceStub.login(McServiceStub.java:5084)
at com.avamar.mc.sdk.client.McServiceClient.login(McServiceClient.java:119)
at com.avamar.mc.sdk.client.McServiceClient.connect(McServiceClient.java:105)
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.connect(ServiceInstance.java:313)
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.<init>(ServiceInstance.java:225)
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.<init>(ServiceInstance.java:187)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromAdapterDefault(VDRExternalServerAccessConfig.java:67)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromPropsDefault(VDRExternalServerAccessConfig.java:126)
at com.emc.vdr.server.ConnectionService.run(ConnectionService.java:40)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(Unknown Source)
at com.emc.avamar.mcsdk.adapter.service.TrustAllSSLProtocolSocketFactory.createSocket(TrustAllSSLProtocolSocketFactory.java:128)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:542)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:199)
... 17 more
2015-04-15 14:30:05,515 ERROR [com.emc.vdr.server.VDRServer$1]-service.AdapterUtils: ServiceInstance failed to obtain domain "/"
java.lang.NullPointerException
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.getDomain(ServiceInstance.java:496)
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.getRootDomain(ServiceInstance.java:462)
at com.emc.vdr.server.VDRServer.getVCenterClient(VDRServer.java:402)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromAdapterDefault(VDRExternalServerAccessConfig.java:86)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromPropsDefault(VDRExternalServerAccessConfig.java:126)
at com.emc.vdr.server.ConnectionService.run(ConnectionService.java:40)
at java.lang.Thread.run(Unknown Source)
2015-04-15 14:30:05,515 ERROR [com.emc.vdr.server.VDRServer$1]-server.VDRServer: VDRServer.getVCenterClient cannot obtain VCenterClient
java.lang.NullPointerException
at com.emc.vdr.server.VDRServer.getVCenterClient(VDRServer.java:403)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromAdapterDefault(VDRExternalServerAccessConfig.java:86)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromPropsDefault(VDRExternalServerAccessConfig.java:126)
at com.emc.vdr.server.ConnectionService.run(ConnectionService.java:40)
at java.lang.Thread.run(Unknown Source)
2015-04-15 14:30:05,515 ERROR [com.emc.vdr.server.VDRServer$1]-server.VDRExternalServerAccessConfig: Unable to fetch the vCenter credential from web service: null
2015-04-15 14:30:05,515 ERROR [com.emc.vdr.server.VDRServer$1]-server.ConnectionService: Unable to get the vi access: no protocol:
2015-04-15 14:30:05,515 INFO [com.emc.vdr.server.VDRServer$1]-server.ConnectionService: Retry after 5 seconds ...
In a desperate attempt I created a new VDP machine and got the same symptoms
I tried the dpnctl start all but it errors (see attached file).
Those machine worked until few days ago and now they gone!
Does anyone know how to resolve these issue?
Thanks in advance.
Rob
P.S.
I am using vCenter 5.1 VDP 5.1