weird wifi issue - slow card payments

[Big Jim]

Business is a pub, card payments are taken via wireless terminals (the bases can be hard wired)

Internet is provided by starlink.
getting over 100mbps wired and a good 70/80 wireless

when I arrived the starlink router was in an upstairs room, with wires trailing across the floor to the middle of another room to a spot that gave them wifi signal in the main part of the bar, but poor coverage elsewhere.

on Friday/Saturday I have installed 2 Unifi UAP-AC-Pro in the bar/dining room downstairs, and and AP-AC-M outside.

about 10 minutes after we switched the card payment terminal over to the new SSID I created they were getting random long processing times on the machine, around 30s to a minute after inputting the amount, then the same again once the customer had paid.
I tried switching over to 4G and to wired, and also back to the original starlink SSID, and they all seemed to resolve the issue for 10/15 minutes then it came back again. however 4G signal is poor in the area

Whilst I was onsite I monitored this for around 30 - 45minutes and it seemed completely random, sometimes payment would be pretty much instantaneous, othertimes having the problem above.
but able to browse the internet/ youtube just fine on my phone.

I thought perhaps it was a coincidental issue with the payment processor that would resolve itself, so I just called the customer back and they said they have spoken to dojo who have checked the payment terminals over and they are up to date and all fine, but the payments are still slow.
payment terminal link below, it is running on android.

Card machines that take fast reliable payments | Dojo

See our full range of Wi-Fi and 3G/4G enabled card machines with sleek 5-inch touchscreens and Apple and Google Pay as standard.

dojo.tech

my next plan of attack is to turn off the unifi APs completely and move the starlink router back to where it was and see if the issue resolves itself.

 

[backwoodsman]

My first thought is a channel conflict with a nearby wifi.

 

[Big Jim]

My first thought is a channel conflict with a nearby wifi.

oh yes I thought about this as well, I ran channel optimisation in the unifi controller and it changed nothing (everything set to channel 1), so I manually changed all the AP channels

the AP layout is in a triangle, so I set the 2 "lower" points of the triangle to channel 8 and the middle point to 13, with the starlink router being on channel 1 (all from memory so might be slightly different channels)

 

[YeOldeStonecat]

So if I'm reading your original post correctly..the issue happens with and without the Unifi APs...and it happens even with the CC terms are using ethernet (not wifi)...to connect?

I wonder if changing MTU on the gateway would help, I've ready that Starlink can be a little odd with it, almost PPPoE like...try 1492 and 1472 for giggles. I've see some CC terminals just be very..."odd"..with certain non-standard internet connections.

 

[Computer Bloke]

everything set to channel 1

Ah, well, there's your problem.

I've seen more problems with marginal 2.4GHz WiFi devices on channel 1 and 13 than you could shake a stick at. (And using channel 8 is just plain antisocial!)

As a quick test, and to simplify your diagnostic process, please try setting everything to channel 6 with 20MHz bandwidth. It's not fast or pretty but it'll stack the deck firmly in your favour when it comes to predictability and (hopefully) reliability, and at least you can stop worrying about things channel-hopping when your back's turned.

Edited to add: Are the EFTPOS terminals using the same WiFi network as other devices (streaming video, perhaps) or the pub's customers (aaarrgh)? If so, you might want to consider setting up a dedicated network for EFTPOS only - taking money in a pub is important.

 

[britechguy]

20MHz bandwidth

Which I have found, on more than one occasion, is critically important (channel aside, and I'm not trying to say your other suggestions should not be taken).

 

[YeOldeStonecat]

While I agree that channel staggering is important, the OP stated he put the CC swipes back on the original single AP SSID, as well as wired. And when I see those 2x things...esp the "wired"...

 

[Big Jim]

I setup the unifi SSIDs and tested that they were working correctly with my laptop.
2 SSIDS 1 for customer's devices (this includes the payment terminals)
and a seperate one for the customers (captive guest)

once I was happy they were ok, we connected one of the payment terminals to the new SSID, then went upstairs for 10 minutes to tidy up some cabling and move the starlink router into another room, when I came back down the bar manager complained about the terminals being slow.


prior to unifi installation, everything was connecting directly to the single starlink SSID (including customers), which is/was on channel 1


in terms of devices on the network, they have several IP cameras, a single POS, 2 card terminals, then a handful of staff devices.
I did limit the customer SSID to 10Mbps, so it shouldn't have been caused by a rogue customer hogging bandwidth.


In terms of the issue being present prior, they are telling me it wasn't. Obviously I have to assume they are telling the truth.
Whilst I was there the issue appeared random though, the terminal would be fine for 10 minutes then suddenly go slow, and we tried using the old network and wiring one of them in and they would both still randomly go slow, although I don't know if the wired terminal used the wire when it was taken out of the base, the ops guy seemed to think it used bluetooth to connect to the base when it was wired, but i'm not entirely sure.

I just can't see how this could have been caused by adding APs to an existing system and changing nothing else, but how do I explain this to a customer ?
In my mind this must be due to the nature of satellite based internet, but how do I explain that when they are insinuating there were no issues prior to unifi being installed.

I am (as a test) going to switch off all the unifi equipment and put the starlink router back where it was to see if that resolves the issue.


I don't know if this picture helps (taken from google maps)
red line is the location of the bar
blue dots are the Unifi APs (the lower one being the outdoor one), these are all downstairs
green dot, old location of starlink router (upstairs)
yellow dot new location of starlink router (upstairs)

 

[YeOldeStonecat]

Is there a Unifi switch in the mix? These days I always want to have Unifi APs going to a Unifi switch. I know years ago they'd generally work "fine" being independent...plugging into LAN ports of a router or some other brand switch. But these days, there's just so many different types of traffic, and countless tweaking/performance benefits when they plug into a Unifi switch, and of course the ease of management, switch port profiles, all that good stuff. And...insights...lots of good insights and troubleshooting.

I'm still not looking away from MTU yet, it appears to work fine when on the Starlink router and its wireless, however...those are baked into one unit. Generally you usually manage MTU from the router itself, however, I've seen threads in the UI forums where some people had quirky on certain types of WAN connections and had to tweak the MTU on the APs directly (mss clamping in the UI world).

 

[Big Jim]

no not a unifi switch, I don't see the benefit in small setups like this, although I do have a couple of spare flex mini switches I could try , however they obviously don't provide POE
the switch is an unmanaged netgear (as are all the switches they have actually)

topography currently
Starlink router > netgear 8 port POE switch>netgear 8 port non POE switch>netgear POE switch (installed by CCTV people),
WAPs connected to the first netgear switch,
2nd switch has a single PC connected to it and shortly will have 3 x printers for their food ordering system
possibly has the DVR/NVR connected as well (haven't traced that wire yet as it wasn't in scope of the job)


prior to any changes everything was wireless direct to the starlink router, the CCTV system wasn't connected to it.



I'm not 100% sure how you access the starlink router, I believe it has to be via a mobile app, I did put the address into a browser and was just greated with a landing page

 

[YeOldeStonecat]

Yeah Starlink gateways....you get into the details with a phone app. The landing page for the webui...quite limited. I always bridge them (there's a different phrase they use)...but so it passes the...WAN IP to the WAN port of your own router. I didn't say "public IP" because..the WAN IP you get from regular starllnk accounts is not a properly routable IP that you can get to from another public IP. So you can't "port forward" and get to stuff from WAN to LAN. You need a business account from starlink to do that.


Benefit of the whole Unifi stack...or at least only Unifi switches and APs....well, here's a scenario now where you can benefit from the deep insights you'd get into troubleshooting, traffic, and management. And the CCTV people daisy chaining 3x switches in a row...makes me start to twitch and go into convulsions. But the info, and stats...you can get from rolling up your sleeve and digging into the Unifi controller...so much useful info. And tweaking things. Important things like "port isolation" on the switch ports that face each AP. Important to have. Can't do that with other switches. Managing mDNS traffic, blocking WLAN to LAN broadcast traffic, etc.

 

[britechguy]

Yeah Starlink gateways....you get into the details with a phone app.

Mini-rant to follow, and not limited to Starlink: What %&#*(@-ing genius decided that it is appropriate to have direct access to wireless access point full controls ONLY through smartphone app?!! And, even worse, that the number of details tends to be greatly decreased over what was available in a conventional webpage interface.

I happen to have one of these wireless access points here at home and my jaw still drops that the only way I can get to any of its "real" controls is via a smartphone app.

 

[Computer Bloke]

Starlink router > netgear 8 port POE switch>netgear 8 port non POE switch>netgear POE switch (installed by CCTV people),

Please forgive me if I'm stating the obvious, but with that number of unmanaged switches installed by different people there's a chance that one of them is looped back into itself. You can sometimes get away with that for a surprisingly long time, but when it bites it bites hard.

 

[Big Jim]

Yeah Starlink gateways....you get into the details with a phone app. The landing page for the webui...quite limited. I always bridge them (there's a different phrase they use)...but so it passes the...WAN IP to the WAN port of your own router. I didn't say "public IP" because..the WAN IP you get from regular starllnk accounts is not a properly routable IP that you can get to from another public IP. So you can't "port forward" and get to stuff from WAN to LAN. You need a business account from starlink to do that.


Benefit of the whole Unifi stack...or at least only Unifi switches and APs....well, here's a scenario now where you can benefit from the deep insights you'd get into troubleshooting, traffic, and management. And the CCTV people daisy chaining 3x switches in a row...makes me start to twitch and go into convulsions. But the info, and stats...you can get from rolling up your sleeve and digging into the Unifi controller...so much useful info. And tweaking things. Important things like "port isolation" on the switch ports that face each AP. Important to have. Can't do that with other switches. Managing mDNS traffic, blocking WLAN to LAN broadcast traffic, etc.

when I say CCTV people, the new owner believes that a lot of this wiring was done by the old owner themselves.
But essentially what they did was add a netgear POE capable switch underneath the bar and just connected it to the existing switch under there.
They could have in theory just replaced the switch under the bar as the 16 port switch has enough free ports.

the switch layout as it is, is neccessary because, the starlink router is upstairs, the 1st switch is also up there with direct cat5 runs to each AP.
there is a single cat5 that has been run under carpets etc from upstairs to underneath the bar. Then a bunch of IP cameras have been cabled to underneath the bar, rather than direct to the DVR. I could remove the "middle" switch, but it is brand new and the 16 port "CCTV switch" is quite old looking and very dirty and very noisy, so I didn't trust it to be 100% reliable.

 

[Big Jim]

Please forgive me if I'm stating the obvious, but with that number of unmanaged switches installed by different people there's a chance that one of them is looped back into itself. You can sometimes get away with that for a surprisingly long time, but when it bites it bites hard.

This is a good point.
I 100% know what is in switch 1 as I installed it, uplink to router, cat 5 to each AP, CAT5 to downstairs under the bar

switch 2, I will double check but fairly sure we only have
payment terminal
PC
DVR
CAT5 run I put into the kitchen
CAT5 run I put into dining room
link to switch 3
link to switch1

Switch 3 not 100% sure, from memory had around 5/6 cables into the POE capable ports, not in scope, I will disconnect it whilst testing as I believe it is purely CCTV cameras
I have temporarily connected an AP to it though and that appears to work fine and is recognised in the unifi controller.



I am sure when I initially visited, the guy I was talking to said the original 1-3MB broadband was still connected and working, but they weren't paying for it and the starlink was ordered because of the slow speed.
If it is that could be really useful. I disconnected the router myself so I know its still on site.

 

[YeOldeStonecat]

Mini-rant to follow, and not limited to Starlink: What %&#*(@-ing genius decided that it is appropriate to have direct access to wireless access point full controls ONLY through smartphone app?!! And, even worse, that the number of details tends to be greatly decreased over what was available in a conventional webpage interface.

Agreed again!

 

[John Kennedy]

Agree with YeOldeStonecat on multiple fronts. Considering the problem happens whether you use wifi or not to connect the terminals, the wifi itself isn't the problem. The switches or internet connection seem much more likely culprits. The wifi is only the problem if the devices work wired and not with wifi. If they don't work wired, then the problem has to be with the equipment in that chain.

Satellite internet by its very nature is great at download and bad at upload. Being able to browse the internet or watch video isn't a good indicator of the ability to use a CC terminal. Test the ping times to the CC terminal server (wherever they connect) over time with something on the wired network. Does it fluctuate? By how much? It seems like the terminals are either dropping packets or suffering from really high latency. That could be due to switches, starlink, cabling, MTU, etc but it won't be the wifi and you'll be able to prove it to the customer.

 

[HCHTech]

If you introduce a new access point, and you unadvisedly assigned it a static IP that conflicts with an existing device, then that's going to affect your wired performance. I, ahem, might have run into this once. The IP was vacant when I took measurements, I swear!

 

[John Kennedy]

If you introduce a new access point, and you unadvisedly assigned it a static IP that conflicts with an existing device, then that's going to affect your wired performance. I, ahem, might have run into this once. The IP was vacant when I took measurements, I swear!

Yes, that would cause things to stop working entirely in some way. It would not cause slowness as the OP indicated.

 

[Big Jim]

Agree with YeOldeStonecat on multiple fronts. Considering the problem happens whether you use wifi or not to connect the terminals, the wifi itself isn't the problem. The switches or internet connection seem much more likely culprits. The wifi is only the problem if the devices work wired and not with wifi. If they don't work wired, then the problem has to be with the equipment in that chain.

Satellite internet by its very nature is great at download and bad at upload. Being able to browse the internet or watch video isn't a good indicator of the ability to use a CC terminal. Test the ping times to the CC terminal server (wherever they connect) over time with something on the wired network. Does it fluctuate? By how much? It seems like the terminals are either dropping packets or suffering from really high latency. That could be due to switches, starlink, cabling, MTU, etc but it won't be the wifi and you'll be able to prove it to the customer.

Any suggestions on a good tool to do this ?
Something that will output in a nice format that I can show to the customer would be excellent