coffee
Well-Known Member
- Reaction score
- 1,832
- Location
- United States
Theres nothing more frustrating then installing a new server and have everything working only to accidentally hard boot the server and find that there is no more internet connection. You probably checked and bounced your network card (s) several times and still no internet access. So, You go in and disable the firewall. Still no hope. Thinking thru what could have gone wrong the thought of just reinstalling starts to creep into the back burner of your mind. Everything seems to check out as you go thru a mental list of possibilities:
1. Network card initialized - check
2. Network card getting dhcp lease assignment - check
3. Firewall disabled - check.
4. Recheck port forwarding - check.
5. Internal network up and running? - check.
Everything seems to be working fine. Still you have no internet connection. Actually the problem is not that complicated to figure out. I will take a minute to share what to look for and how to get the problem corrected. Lets introduce our test subject:
Your basic server running Ubuntu Server. Consisting of two network cards and providing NAT and port forwarding from other client computers on the network.
Our network interfaces:
enp3s0 = Connected to our ISP modem
enp4s5 = The internal network
enp3s0 Link encap:Ethernet HWaddr f4:6d:04:cd:1b:8f
inet addr:50.148.94.226 Bcast:255.255.255.255 Mask:255.255.248.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3841957 errors:0 dropped:0 overruns:0 frame:0
TX packets:2151817 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4798301226 (4.7 GB) TX bytes:414771059 (414.7 MB)
enp4s5 Link encap:Ethernet HWaddr 00:e0:4c:1f:95:1c
inet addr:10.0.1.1 Bcast:10.0.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3223094 errors:0 dropped:0 overruns:0 frame:0
TX packets:4460507 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1601626081 (1.6 GB) TX bytes:5319549104 (5.3 GB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:38719 errors:0 dropped:0 overruns:0 frame:0
TX packets:38719 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:3307574 (3.3 MB) TX bytes:3307574 (3.3 MB)
So, To review really quickly: Everything seems ok. Network cards are up and running. Our enp3s0 card is getting an address from the ISPs dhcp server. NAT is running and the firewall is off. Pinging any internal ip on the network is fine. Nothing from the outside however. We cannot even ping the broadcast address from the outside.
Here is what you do:
The first thing to check is the routing tables.
coffee@dino2:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 50.148.88.1 0.0.0.0 UG 0 0 0 enp4s5
10.0.1.0 * 255.255.255.0 U 0 0 0 enp4s5
50.148.88.0 * 255.255.248.0 U 0 0 0 enp3s0
coffee@dino2:~$
You will notice that there is something strange with the routing table. The default gateway is most likely the problem as it wants to route packets to the internal network card (enp4s5). Therefore nothing is going to get out of the internal network. So, We need to fix this and here is how.
First thing first, Be sure you have local access to your server. We will be taking down both network cards and then flush the routing tables.
Lets take down the network cards right now:
sudo ifdown enp4s5
sudo ifdown enp3s0
Now we will totally flush out the routing tables as they appear to be corrupted.
ip route flush all
Now when we check our routing tables again we should see:
coffee@dino2:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
Now we are back to square one with the routing tables. Next bring up your external network card and make sure it gets an ip address from the ISPs dhcp server:
ifup enp3s0
After the dhcp finishes negotiating check your routing table and you should see something similar to:
coffee@dino2:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 50.148.88.1 0.0.0.0 UG 0 0 0 enp3s0
We now have the correct gateway and network card. At this point we can now bring up our internal network card and display the routing table again.
ifup enp4s5
coffee@dino2:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 50.148.88.1 0.0.0.0 UG 0 0 0 enp3s0
10.0.1.0 * 255.255.255.0 U 0 0 0 enp4s5
50.148.88.0 * 255.255.248.0 U 0 0 0 enp3s0
Remember to restart your firewall (which will also start NAT) and you should be good to go.
Using ufw?
sudo ufw restart
Using shorewall?
sudo shorewall restart
Corrupted routing tables can drive you crazy if you do not know what you are looking for and not that familiar with linux as a server. However, If you took a look at the logs you could guess the situation as there were probably a slew of packets being denied on the internal network card.
Instead of going thru the logs to find your problem you can also do it in real time. If your not local to the server you can ssh into the box and watch the error log in question in real time with this command:
tail -f /var/log/kern.log
(cntr - x to quit)
You would readily see the packets destined for the internet being rerouted to the internal network
1. Network card initialized - check
2. Network card getting dhcp lease assignment - check
3. Firewall disabled - check.
4. Recheck port forwarding - check.
5. Internal network up and running? - check.
Everything seems to be working fine. Still you have no internet connection. Actually the problem is not that complicated to figure out. I will take a minute to share what to look for and how to get the problem corrected. Lets introduce our test subject:
Your basic server running Ubuntu Server. Consisting of two network cards and providing NAT and port forwarding from other client computers on the network.
Our network interfaces:
enp3s0 = Connected to our ISP modem
enp4s5 = The internal network
enp3s0 Link encap:Ethernet HWaddr f4:6d:04:cd:1b:8f
inet addr:50.148.94.226 Bcast:255.255.255.255 Mask:255.255.248.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3841957 errors:0 dropped:0 overruns:0 frame:0
TX packets:2151817 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4798301226 (4.7 GB) TX bytes:414771059 (414.7 MB)
enp4s5 Link encap:Ethernet HWaddr 00:e0:4c:1f:95:1c
inet addr:10.0.1.1 Bcast:10.0.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3223094 errors:0 dropped:0 overruns:0 frame:0
TX packets:4460507 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1601626081 (1.6 GB) TX bytes:5319549104 (5.3 GB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:38719 errors:0 dropped:0 overruns:0 frame:0
TX packets:38719 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:3307574 (3.3 MB) TX bytes:3307574 (3.3 MB)
So, To review really quickly: Everything seems ok. Network cards are up and running. Our enp3s0 card is getting an address from the ISPs dhcp server. NAT is running and the firewall is off. Pinging any internal ip on the network is fine. Nothing from the outside however. We cannot even ping the broadcast address from the outside.
Here is what you do:
The first thing to check is the routing tables.
coffee@dino2:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 50.148.88.1 0.0.0.0 UG 0 0 0 enp4s5
10.0.1.0 * 255.255.255.0 U 0 0 0 enp4s5
50.148.88.0 * 255.255.248.0 U 0 0 0 enp3s0
coffee@dino2:~$
You will notice that there is something strange with the routing table. The default gateway is most likely the problem as it wants to route packets to the internal network card (enp4s5). Therefore nothing is going to get out of the internal network. So, We need to fix this and here is how.
First thing first, Be sure you have local access to your server. We will be taking down both network cards and then flush the routing tables.
Lets take down the network cards right now:
sudo ifdown enp4s5
sudo ifdown enp3s0
Now we will totally flush out the routing tables as they appear to be corrupted.
ip route flush all
Now when we check our routing tables again we should see:
coffee@dino2:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
Now we are back to square one with the routing tables. Next bring up your external network card and make sure it gets an ip address from the ISPs dhcp server:
ifup enp3s0
After the dhcp finishes negotiating check your routing table and you should see something similar to:
coffee@dino2:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 50.148.88.1 0.0.0.0 UG 0 0 0 enp3s0
We now have the correct gateway and network card. At this point we can now bring up our internal network card and display the routing table again.
ifup enp4s5
coffee@dino2:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 50.148.88.1 0.0.0.0 UG 0 0 0 enp3s0
10.0.1.0 * 255.255.255.0 U 0 0 0 enp4s5
50.148.88.0 * 255.255.248.0 U 0 0 0 enp3s0
Remember to restart your firewall (which will also start NAT) and you should be good to go.
Using ufw?
sudo ufw restart
Using shorewall?
sudo shorewall restart
Corrupted routing tables can drive you crazy if you do not know what you are looking for and not that familiar with linux as a server. However, If you took a look at the logs you could guess the situation as there were probably a slew of packets being denied on the internal network card.
Instead of going thru the logs to find your problem you can also do it in real time. If your not local to the server you can ssh into the box and watch the error log in question in real time with this command:
tail -f /var/log/kern.log
(cntr - x to quit)
You would readily see the packets destined for the internet being rerouted to the internal network
