Local IP address no longer works???

[Markverhyden]

Have you engaged Fortinet support on this? How are you trying to access the target machine? Have your defaulted the Fortigate and flashed the firmware? To be honest I would probably just retire that fixed IP if possible.

I'm not a big fan of Fortinet. 3-4 years ago I had a site which needed a point to point VPN for data and VoIP. While data worked fine VoIP was a major problem in terms of QoS. Since it was a new install I worked with support and nothing solved the problem. Ended up dumping the Fortinet devices and went with pair of Cisco RV's.

This is a situation where you could spend many hours and if you cannot default the Fortigate for practical reasons it's a treadmill. That's why I suggested retiring the IP.

 

[BrandonTech]

So, this is not the first machine to have this issue? What do the machines have in common? That may provide a clue. I still think something is tripping the firewall. Did the previous machine have the same function or share any functions? Same users (even intermittently)? Same network branch?

This is the second machine to my knowledge. The first one occurred several months back. The glaring commonality would be that they both had static IP's. Other than that, I cannot think of anything at this time, but the symptoms were the exact same. (No internet access unless configured with DHCP or a different static IP.)

The previous machine was on a totally different branch, i.e. different subnet, different gateway, etc. Both do share the same firewall (so add that to things they have in common). Everything gets routed back to our mainframe through the site gateways and then out through the firewall. The machines do not serve the same functions. The original machine that had the issue has just been left to DHCP. Not sure how they are monitoring their HVAC at this point. :/

I agree with you that it must be something in the firewall in relation to the IP address, but I haven't seen anything that it could be yet. Also, keep in mind that the issue follows the faulty IP address. In other words, you can give a totally different unrelated machine the same IP address and it causes that PC to lose internet connection as well.

One thing I may try once I return, other than capturing packets, is to put the currently affected machine into a firewall policy with nothing turned on that could block it. All app control, filtering, traffic shaping, etc. will be off just to see if that makes a difference.

 

[FremontPC]

This is the second machine to my knowledge. The first one occurred several months back....

Cool. Keep on with the updates when you get a chance. Much appreciated.

 

[BrandonTech]

Have you engaged Fortinet support on this? How are you trying to access the target machine? Have your defaulted the Fortigate and flashed the firmware? To be honest I would probably just retire that fixed IP if possible.

I'm not a big fan of Fortinet. 3-4 years ago I had a site which needed a point to point VPN for data and VoIP. While data worked fine VoIP was a major problem in terms of QoS. Since it was a new install I worked with support and nothing solved the problem. Ended up dumping the Fortinet devices and went with pair of Cisco RV's.

This is a situation where you could spend many hours and if you cannot default the Fortigate for practical reasons it's a treadmill. That's why I suggested retiring the IP.

I feel we have had nothing but trouble since installing the Fortigate. I do not recommend them at all. I have been accessing the target machine both onsite and remotely. The last time I checked, firmware was updated. I will not be able to default the firewall, too much to reconfigure for one crazy IP issue.

On the first machine that had the issue, we did end up discontinuing use of the IP address. It just irks the crap out of me that the IP could just "die".

 

[Mike McCall]

This is the second machine to my knowledge. The first one occurred several months back. The glaring commonality would be that they both had static IP's. Other than that, I cannot think of anything at this time, but the symptoms were the exact same. (No internet access unless configured with DHCP or a different static IP.)

The previous machine was on a totally different branch, i.e. different subnet, different gateway, etc. Both do share the same firewall (so add that to things they have in common). Everything gets routed back to our mainframe through the site gateways and then out through the firewall. The machines do not serve the same functions. The original machine that had the issue has just been left to DHCP. Not sure how they are monitoring their HVAC at this point. :/

I agree with you that it must be something in the firewall in relation to the IP address, but I haven't seen anything that it could be yet. Also, keep in mind that the issue follows the faulty IP address. In other words, you can give a totally different unrelated machine the same IP address and it causes that PC to lose internet connection as well.

One thing I may try once I return, other than capturing packets, is to put the currently affected machine into a firewall policy with nothing turned on that could block it. All app control, filtering, traffic shaping, etc. will be off just to see if that makes a difference.

Hmm...are the affected machines the only ones with static ip's, or are there more on the network?

 

[BrandonTech]

Hmm...are the affected machines the only ones with static ip's, or are there more on the network?

There are more though not many if you don't count our servers. There would be a similar setup at each of our 4 schools' lunchrooms.

 

[Mike McCall]

No go on pinging the IP address. Firewall isn't showing anything blocked, but it doesn't show anything received either (all sent) when pinging and tracert.

What's outside of the firewall?

 

[BrandonTech]

What's outside of the firewall?

I believe the next stop is our service provider's switch. Since we are a city school system we have a direct uplink to them as they are the city utilities board. I am not aware of any other devices that we own on the WAN side of the firewall thought there could be some type of edge device. I will double check this tomorrow.

 

[Mike McCall]

I believe the next stop is our service provider's switch. Since we are a city school system we have a direct uplink to them as they are the city utilities board. I am not aware of any other devices that we own on the WAN side of the firewall thought there could be some type of edge device. I will double check this tomorrow.

The reason I ask is because of your comment about not receiving any packets back from your ping/tracert. Of course, the firewall could simply neglect to pass anything on and not bother telling you.

 

[Barry Mossip]

Have you tried to clear the arp on the ws?

 

[NETWizz]

Have you verified you have the correct default gateway and the IP and subnet match the local network only the IP itself is unique?

Can you ping outside by IP address to the Internet without DNS? i.e. ping 8.8.8.8 ??

I do not see that a firewall would pick on a particular computer unless you specifically put in a policy or rule to do this. I would still look to the local config on the computer.

I doubt the switch would cause problems unless someone put an access control list on it that references the host, so this is not making sense.

I suspect your IP addressing is the culprit. Cares to post the details here? It is probably a 10.x.x.x behind a NAT anyway.

 

[BrandonTech]

Have you verified you have the correct default gateway and the IP and subnet match the local network only the IP itself is unique?

Can you ping outside by IP address to the Internet without DNS? i.e. ping 8.8.8.8 ??

I do not see that a firewall would pick on a particular computer unless you specifically put in a policy or rule to do this. I would still look to the local config on the computer.

I doubt the switch would cause problems unless someone put an access control list on it that references the host, so this is not making sense.

I suspect your IP addressing is the culprit. Cares to post the details here? It is probably a 10.x.x.x behind a NAT anyway.

Yes to verifying, No to pinging outside of network when on "bad" IP.

IP issues image shows the settings with the bad IP. In order to make the machine work, I can change one number. Change the last octet of the IP address to any other available IP address on the subnet and it works. (or at least the ones I have tried)

 

[BrandonTech]

Another strange thing, I cannot log into the Firewall management GUI from the workstation no matter what. I can log into the Analyzer component, but it runs on a server and not the firewall.

I can ping the IP address of the Firewall, but I cannot log in via a browser. I have never noticed this elsewhere.

I am attaching to this post what I was able to pull up from the analyzer. Notice the ping and dns traffic is all "sent" when using the "bad" IP and not received. Unexplainable for me, is the small bit of traffic it says is received via HTTP, unless it is cached from our servers. Nothing is happening at the workstation that proves it is connected to anything other than the LAN.




Notice the destinations here?? You can see my ping to Google public DNS didn't receive anything back.

For comparison here is what traffic looks like with a good IP. Much more received.


I still intend to get back in the firewall and see about excluding the local bad address (10.30.40.19).

Sorry for the long post.

 

[NETWizz]

Can you show me an IP address of a working machine?

OR an IP address provided by DHCP? Please?

Just looking at that, I can tell you that the network is 10.30.40.0/22 and has a usable range of 10.30.40.1 through 10.30.44.254/22

10.30.44.255 is reserved for the broadcast. 10.30.40.1 is, indeed, in that subnet, which means it can be the router.


All that said, you will NOT be able to reach those DNS servers 10.30.50.25 and 10.3.50.3 if they are on the same Local-Area-Network (i.e. Same Layer-2)... Of course, if the computers can get to the subnet your DNS servers are on via the router (or Layer-3 switch) then there is no reason this setup would not work... provided 10.30.40.19 is truly unique.

If you unplug this computer and ping 10.30.40.19 do you get a response? WILL this exact IP work on another, different computer?

Provided you get no response above... From another computer on the same Layer-2 Network (i.e. the LAN), can you ping to this computer 10.30.40.19 when it is connected and the Interface is up/up?

If yes, please verify the ARP table on the local computer initiating the ping. To do this launch an elevated command prompt and flush the ARP tables like this
c:\users\someone>arp -d *
c:\users\someone>ping 10.30.40.19
[output redacted]
c:\users\someone>arp -a
c:\users\someone>ping 10.30.40.19
Interface: 10.30.40.123 --- 0xe
Internet Address Physical Address Type
10.30.40.1 74-8e-f8-91-dd-aa dynamic

You will see something like the above if it is across the Router's Gateway (or Switch's gateway)...

Otherwise you will see the Physical Address of the computer...

OR you will see the Physical Address of a Switch Interface connecting to another Switch...

Either way, figure out what the Physical Address is and check it matches the actual computer.

If it does NOT match, the next step is to logon to the switch the computer initating the ping is logged onto and check it.

switch# show mac (possibly show mac-address table)

Regardless your switch should know which interface it is on or which interface to switch the traffic too... If it goes to another switch you can generally do a show cdp neighbors, show lldp neighbors, show fdp neghbors, etc... depending on the type of switches...

Go the the next switch... lather.. rinse... repeat.

Verify it works on the Local Network... if not figure out if there are some stickey mac address stables on the switch or something...

Regardless once it is working on the local network, it is next important to verify it to the router, which is directly connected to the network... it should work... Now, do an EXTENDED ping... this is where you do a ping specifying the router's source IP or Interface. (Choose the source as the WAN interface or WAN IP). Hence, internally the router will do a layer-3 ping actually routing the ICMP between its directly-connected networks...

Then try a remote router... then a remote switch...then a remote computer... etc.

Figure out where it breaks down.

 

[YeOldeStonecat]

I still intend to get back in the firewall and see about excluding the local bad address (10.30.40.19).
.

I still have my money on this....the Firewall has some rule about that IP address (and probably a few neighboring ones too).

So if what I'm reading to far is correct....you could go to ANY computer on your network and give it that .19 address and they will stop getting to/from the internet.

And you could go to THIS computer with the .19, and change it's IP to a .1xx or something else up in the handout range, and it WILL get to/from the internet just fine.

Gotta be a firewall ACL that covers .19 with something special.

 

[Mike McCall]

I still have my money on this....the Firewall has some rule about that IP address (and probably a few neighboring ones too).

So if what I'm reading to far is correct....you could go to ANY computer on your network and give it that .19 address and they will stop getting to/from the internet.

And you could go to THIS computer with the .19, and change it's IP to a .1xx or something else up in the handout range, and it WILL get to/from the internet just fine.

Gotta be a firewall ACL that covers .19 with something special.

^What he said!

 

[BrandonTech]

Well. Found it.

Sifting through the firewall settings I came across some options for Virtual IPs. Fortigate uses these to map public IP addresses directly to a private IP address. Guess who I found in there. Both culprit "bad" IP addresses. Now, mind you, I'm not sure why this particular computer has a public IP address mapped to it, but I can find that out later. Usually when I run up on things like this, it was from an old configuration that is no longer in use and was never changed.

Port mapping was off, and I feel that may have something to do with why it wasn't sending any traffic across the Virtual IP. In other words, I believe the firewall didn't know what to do with it the packets when they came back since it could't figure out what port to use when sending across the Virtual IP mapping.

If the public IP address is necessary, I will determine if I can set specific ports or a range.

Thank you all very, very much for the help. Without everyone's contribution, I do not believe I would have found the cause of the issue. It is nice being able to bounce this stuff off of you guys. I am fairly new in the field (in regards to networks of this size), and no one made me feel like an idiot. I appreciate it.

 

[Mike McCall]

Well. Found it.

Sifting through the firewall settings I came across some options for Virtual IPs. Fortigate uses these to map public IP addresses directly to a private IP address. Guess who I found in there. Both culprit "bad" IP addresses. Now, mind you, I'm not sure why this particular computer has a public IP address mapped to it, but I can find that out later. Usually when I run up on things like this, it was from an old configuration that is no longer in use and was never changed.

Port mapping was off, and I feel that may have something to do with why it wasn't sending any traffic across the Virtual IP. In other words, I believe the firewall didn't know what to do with it the packets when they came back since it could't figure out what port to use when sending across the Virtual IP mapping.

If the public IP address is necessary, I will determine if I can set specific ports or a range.

Thank you all very, very much for the help. Without everyone's contribution, I do not believe I would have found the cause of the issue. It is nice being able to bounce this stuff off of you guys. I am fairly new in the field (in regards to networks of this size), and no one made me feel like an idiot. I appreciate it.

Yay!

If it makes you feel any better, today I was playing around with Security Policy on my server. Every server should have one, right? So, I go chugging through the wizard merrily accepting the defaults. Once I applied the resulting policy I found I was completely locked-out of the server. Fun times! I did manage to get back in (mostly by accident), and rolled-back the offending policy. I got me some reading to do.

No, sir. You are NOT an idiot!

 

[YeOldeStonecat]

.

Thank you all very, very much for the help. Without everyone's contribution, I do not believe I would have found the cause of the issue. It is nice being able to bounce this stuff off of you guys. I am fairly new in the field (in regards to networks of this size), and no one made me feel like an idiot. I appreciate it.

Hey, good "mysteries" like this...we call get to learn, especially when masters of IP such as NETWizz drop in and dazzle us.
Some of us grew up counting sheep to fall asleep. NETWizz was blurting out subnetting when he was an infant!

 

[NETWizz]

I am just glad you found it... I was beating my head on this one...