Troubleshooting SDWAN Bootstrap


Connectivity Issues Between CPE and Controller/Versa Director during Staging

Follow these steps to troubleshoot connectivity issues between Customer Premise Equipments (CPE) and Controller/Versa Director nodes:
  1. Verifying IPsec local and remote profile matches.
    • Run these CLI commands on CPE where staging is failed.
      show configuration org org-services <org> ipsec
show orgs org-services <org> ipsec vpn-profile <vpn-profile> ike history
    • Run this CLI command on Staging Controller. 
      show configuration org org-services <org> ipsec vpn-profile <staging profile>
  2. Verifying Internet Key Exchange (IKE) packets are reaching Staging Controller or not. 
    • Run this CLI command on CPE where staging is failed. 
      tcpdump <WAN interface> filter host <staging controller IP>
    • Run this CLI command on Staging Controller. 
      tcpdump <WAN interface> filter host <staging CPE public IP>

  3. When IKE and Encapsulating Security Payload (ESP) packets sent by the CPE or the Controller is not seen each other, please look into underlay network (FW) and allow these ports.

    • 500 and 4500/UDP-IKE/IPsec

    • IP Protocol 50-ESP

  4. When the above steps fails to provide the desired output, please capture the screenshots of both CPE and staging controller along with tcpdump and share it with Versa Support team.


Pushing Post Staging configuration into CPE

Follow these steps to troubleshoot post staging configuration issues:
  1. Run this CLI command on Versa Director. 
    /opt/versa/vnms/scripts/netconf-check.sh <TVI IP address of staging CPE>
  2. When the above step fails, please verify IPsec connectivity between controller and CPE refer Connectivity Issues Between CPE and Controller/Versa Director during Staging mentioned in this article.
  3. When IPsec is stable, ping CPE TVI IP address on Versa Director, if it is pinging run this CLI command on the Versa Director.
  4. When the above command fails 
    • Run this CLI command on Versa Director. 
      $ sudo tcpdump -ni <South bound interface through which CPE is connected> host <TVI IP of CPE> and port 2022
    • Run this CLI command on CPE. 
      $ sudo tcpdump -ni any host <Director south bound interface IP> and port 2022

  5. When the packets exchanged between Versa Director and CPE is not seen each other, underlay would be dropping the packets. Please allow the below mentioned ports on FW and verify the asymmetrical routing is not causing packet drops on underlay FW.

    Protocol/Port
    		Purpose
    UDP/4790
    		VXLAN communication between Versa FlexVNF Hub, Branch and Controller.
    UDP/4500 and 500
    		IKE/IPsec Protocol.
    2022    
    		netconf/SSH from Versa Director to Versa FlexVNF.
    4000
    		Service to allow Versa Director to execute network utilities like ping, traceroute etc.
    TCP/3000, 3001 and 3002
    		High Availability (HA) between FlexVNF instance, if HA nodes exist and are configured.
    IP/50
    		Encapsulating Security Payload (ESP)


  6. Run this CLI command on CPE if the TCP packets sent by Versa Director is seen by CPE but not responding.

  7. When the above steps fails to provide the desired output, please capture screenshots of CPE, Controller and Versa Director along with tcpdump and share it with Versa Support team.


CPE is Disconnected After Pushing Post Staging Configuration into CPE

Follow these steps to troubleshoot CPE issues:
  1. Refer Pushing Post Staging Configuration into CPE mentioned in this article. 
  2. When the above step did not resolve the issue please verify whether the template pushed is configured correct or not.
  3. Ping the controller IP from WAN routing instance. When pinging fails, please verify the interface and the IP address associated with it is properly configured or not.
  4. When the controller IP is pinging successfully 
    • Run this CLI command on CPE to verify VXLAN packets sent by CPE is reaching the controller and vice versa. 
      tcpdump <WAN interface> filter host <Post staging controller IP> and port 4790
    • Run this CLI command on post staging controller to verify VXLAN packets sent by CPE is reaching the controller and vice versa. 
      tcpdump <WAN interface> filter host <CPE public IP> and port 4790
  5. When the VXLAN packets exchanged between Versa Director and CPE are not seen each other, underlay would be dropping the packets. Refer the table (mentioned in this article) and allow the ports on FW and verify the asymmetrical routing is not causing packet drops on underlay FW.
  6. When all the above mentioned steps works fine, verify the connection status between Versa Director and CPE. Ping CPE TVI IP address from Versa Director. If the pinging action is successful, run these CLI commands from Versa Director SSH. 
    ssh <CPE TVI IP address> -l admin -p 2022
/opt/versa/vnms/scripts/netconf-check.sh <TVI/Overlay IP address of CPE>
  7. When the above CLI command fails, run these CLI commands on Versa Director and CPE. 
    • Run this CLI command on Versa Director. 
      $ sudo tcpdump -ni <South bound interface through which CPE is connected> host <TVI IP of CPE> and port 2022
    • Run this CLI command on CPE. 
      $ sudo tcpdump -ni any host <Director south bound interface IP> and port 2022
  8. When the packets sent by Versa Director is reaching CPE but not responding, verify netconf port is enabled and check the status of all the running services using vsh status CLI command.
  9. When the above steps fails to provide the desired output, please capture screenshots of CPE, Controller and Versa Director along with tcpdump and share it with Versa Support team.