VPS IP Routing Conflict Case Study | How Database Mart Fixed Asymmetric Routing in 6 Hours

Incident Timeline

03/29

IP Request

📩 Privacy Concern

03/31 · 13:56

Critical Outage

🚨 SSH / Panel Down

03/31 · 19:24

Root Cause Found

🔍 Asymmetric Routing

03/31 · 19:55

Service Restored

✅ Policy Routing Fixed

* From total outage to full restoration in under 6 hours — without any data loss or server reset.

The Request

Security-Driven IP Addition Request

On March 29, 2026, a customer submitted a request to add an additional dedicated IPv4 address to her VPS. The reason: her current IP had been exposed publicly and was under constant attack — unauthorized SSH attempts were consuming bandwidth and spiking CPU usage.

Her explicit condition: no server reset or reinstall — she had critical data inside the VPS that could not be lost.

"I just need my IP to be changed so that my new IP will be protected by the attackers who are trying to get into it. I request you to please add an additional IP address to my VPS."

— Customer, March 30, 2026

The Outage

Total SSH & Management Panel Lockout

On March 31 at 13:56, the customer reported a critical outage:

Unable to log in via PuTTY (SSH)
Unable to connect via WinSCP (SFTP)
Server Management portal stuck on a continuous loading circle

Earlier that day (16:08), the system had logged a failed IP addition attempt with a "VPS connection check timeout" error. The new IP addition process had broken the server's network configuration.

"I am currently facing critical access issues with my VPS… if the new IP addition is making this problem appear then please cancel the IP address, I just want my server to be working perfectly. Also please do not reset the server or VPS at all as there is data inside."

— Customer, March 31, 2026

Precision Diagnosis

Uncovering the Asymmetric Routing Issue

Engineer Queta took over the investigation and quickly identified the root cause using a simple but powerful diagnostic technique:

"I pinged xxx.xxx.xxx.xxx (primary), but the reply came back from xxx.xxx.xxx.xxx (secondary)."

— Queta, Support Department, March 31, 19:24

This was a classic asymmetric routing failure. The VPS had multiple IP addresses and network interfaces. After the additional IP was configured, the kernel selected the wrong source address for outgoing packets. As a result, packets sent to the original IP were returned using the secondary IP, causing asymmetric routing and breaking stateful connections such as SSH.

The VitalPBX firewall running on the server had created routing rules that conflicted with the newly added second network interface, exacerbating the issue.

Most providers would have recommended a full OS reload. Instead, Queta implemented a surgical fix using policy-based routing.

The Solution

Surgical Policy-Based Routing Fix

Queta applied the following routing rules to ensure each IP used the correct return path:

# Add routing tables
echo "100 eth0table" >> /etc/iproute2/rt_tables
echo "200 eth1table" >> /etc/iproute2/rt_tables

# Route for original IP (eth0)
ip route add xxx.xxx.xxx.0/25 dev eth0 src xxx.xxx.xxx.xxx table eth0table
ip route add default via xxx.xxx.xxx.1 dev eth0 table eth0table

# Route for new IP (eth1)
ip route add xxx.xxx.xxx.0/25 dev eth1 src xxx.xxx.xxx.xxx table eth1table
ip route add default via xxx.xxx.xxx.1 dev eth1 table eth1table

# Policy rules to bind source IP to correct table
ip rule add from xxx.xxx.xxx.xxx table eth0table
ip rule add from xxx.xxx.xxx.xxx table eth1table

This configuration ensured that traffic originating from each IP would use the correct routing table, solving the asymmetric routing problem at the kernel level — no reboot, no data loss, no server reset.

Resolution

Service Fully Restored — All Data Intact

By 19:55 on March 31, just 31 minutes after the root cause was identified, Queta confirmed the fix:

"After applying the routing rules and fixing the return path, the server is now responding correctly and SSH access is functioning as expected."

— Queta, Support Department, March 31, 19:55

The customer was able to reconnect via PuTTY and WinSCP immediately. The Server Management portal also refreshed once the network state stabilized.

Key outcomes:

SSH and WinSCP access fully restored
Server Management portal loading normally
All data preserved — zero data loss
No server reset or reinstall required
Additional IP successfully configured and functional

"All problems and issues are still there nothing is changed or fixed in my vps"

— Customer, April 1, 04:05

The customer later reported that the issue appeared unresolved at 04:05 on April 1. After reviewing the timeline, we confirmed that the test was performed before the final routing and SSH configuration adjustments were completed. After the routing correction, everything worked perfectly.

Final Follow-up

Jonathan's Confirmation & Cleanup

On April 1 at 04:37, engineer Jonathan provided a final summary, confirming that the SSH configuration had been adjusted to allow password authentication and that the handshake protocols were now cleared.

"We have accessed your server via the back-end console and temporarily updated the SSH configuration to allow Password Authentication… Please rest assured that no data was reset or modified during this process; we only adjusted the login permission settings."

— Jonathan, Support Department, April 1, 2026

RESOLVED NO DATA LOSS NO REBOOT

Geek Tips — Asymmetric Routing

How to spot an asymmetric routing issue in 2 minutes:

Run ping -c 3 YOUR_SERVER_IP from your local machine.
Run tcpdump -i any icmp on the server while pinging.
If the reply comes from a different IP address than the one you pinged — that's asymmetric routing.
Fix it with policy-based routing as shown above, or temporarily disable the secondary interface to restore connectivity.

💡 This is exactly how Queta diagnosed the issue in 30 minutes — a simple ping + tcpdump comparison.

VitalPBX + Additional IPs = Routing Headaches?

VitalPBX includes a firewall that automatically creates routing rules. When you add a second IP, the firewall may override the default routing behavior. Quick checks:

# Check current routing rules
ip rule show
ip route show table all

# Check VitalPBX firewall rules
/usr/sbin/iptables -L -n -v | grep -i vital

🔍 If you see rules that bind traffic to a specific source IP, you'll need to add policy-based routing as shown in this case study.

SSH "Connection Refused" after IP change?

Check if your VPS provider has strict SSH key authentication enabled. In this case, Jonathan had to temporarily allow password authentication via the back-end console:

# Edit SSH config
vim /etc/ssh/sshd_config
# Set PasswordAuthentication yes
# Restart SSH service
systemctl restart sshd

🔐 Remember to revert to key-only authentication after troubleshooting for security.

"We don't just add IPs.
We make sure the packets know exactly where to go —
so your data stays safe, and your server stays online."