Recently I was working on my lab network, and I have an 1831 access point and a 3702 AP. My comments are specific to Mobility Express, which to be fair is just a regular WLC, running on an access point, with all AP’s in FlexConnect mode only. The AP’s are responsible for all packet processing, NBAR/AVC, anything you are doing goes on in the AP’s.
Naturally, I wanted to get the most out of my network, but I ran into a few challenges, and I will document them here….
First, I am far from a CLI expert on the WLC stuff, I have spent most of my life running WLCs with the GUI – but the Mobility Express series GUI is very simple. I got much better at the CLI during this.
The latest GUI on 8.5 has an “expert” mode now that lets you play with some of the RF settings, the 8.3 version is pretty simplistic. So I popped in the 8.5.103 version, and was liking the new GUI. Everything seemed like it was working…. I applaud Cisco for improving the Mobility Express GUI – it was more simple than some home Linksys offerings in the beginning, this is a step in the right direction.
Let me outline my environment….
- I live in a rural area – there is ZERO wireless noise here, and I control the spectrum pretty well. I don’t deploy stuff without considering the impact.
- I have about a dozen client devices
- For all my testing – I kicked everyone off 5ghz, and ran on just a single AP. Nothing else was in the air – I confirmed this using a spectrum analyzer.
Running the latest bit me
Until I had a problem with my Macbook Air (Early 2014 Model). If you go and look, many people complain about Apple Macbook Air’s and wireless issues – so many different opinions, some blame Apple, some say replace your “router” or access point but I couldn’t find any kind of real problem.
Not a surprise. I ran 8.5.103 – and I was having weird problems. All of my clients were fine except my Macbook Air – as long as it was on 2.4ghz, it was fine – but bump it up to 5ghz, and as soon as traffic started flowing – the AP would simply start ignoring the client. Client thought it was associated, AP saw it as associated — but no traffic moved. It would sometimes come back, sometimes not, if I bounced client adapter – it would come right back. 2.4 was solid.
Doing what I always tell my clients – run the “Gold Star” release in this case 8.3.122 – So I put that version in, and let the APs upgrade. Everything seemed better now – connectivity was solid. After my findings below, I went back to test 8.5.103 again…
AVC Hurting Performance
So being that it was “working” I switched to performance testing. I run a iPERF3 server on my QNAP here at home – confirming performance I was getting 995mb/sec from my wired desktop to the NAS… Ok we are good to test.
My Macbook Air was connected with the following…
Performance Signal Strength: -53 dBm
Signal Quality: 43 dB
Connection Speed: 867 Mbps
Channel Width: 80 MHz
Capabilities 802.11ac (5GHz) Spatial Stream: 2
Time for a test…
Connecting to host QNAP, port 5201
[ 4] local port 56551 connected to QNAP port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 15.5 MBytes 130 Mbits/sec
[ 4] 1.00-2.00 sec 16.1 MBytes 135 Mbits/sec
[ 4] 2.00-3.00 sec 16.0 MBytes 134 Mbits/sec
[ 4] 3.00-4.00 sec 15.9 MBytes 133 Mbits/sec
[ 4] 4.00-5.00 sec 16.1 MBytes 135 Mbits/sec
[ 4] 5.00-6.00 sec 15.8 MBytes 133 Mbits/sec
[ 4] 6.00-7.00 sec 15.8 MBytes 132 Mbits/sec
Ok this isn’t right… Something isn’t working…. So I contacted my good friend @wifijanitor
– Steve to bounce some ideas off him. We quickly got to “It’s all configured correctly”
So I started disabling this and that 802.11(insert feature here) and everything one by one. Problem remained.
Finally, I disabled AVC – Application Visibility and Control…
[ 4] 71.00-72.00 sec 46.4 MBytes 390 Mbits/sec
[ 4] 72.00-73.00 sec 46.3 MBytes 389 Mbits/sec
Well look at this… The only thing we could figure out is that the AP must be getting hammered by the AVC… So, I investigated that….
AP CPU with AVC Enabled
Whoa, that is 100%… This is with my iPerf, i’m getting 140-150 mbit. Ok, let’s try with it disabled.
AP CPU with AVC Disabled – Heavy Load with iPerf
Whoa… That’s not a good thing… That means even the performance i’m getting now is probably being hampered by the CPU on board… Close to 400mbit throughput, and the CPU is high. According to the system it is nothing but packet process. There has to be a choke point… I wonder what would happen if I had more CPU – i’m not able to clear up any more CPU, everything (I think) is disabled.
AP CPU with AVC Disabled – 100 Mbit Stream
Ok so i’m trying to prove my theory… This is AVC Disabled, 100MB Stream using iPerf. About 30% CPU utilization…
AP CPU with AVC Enabled – 100 Mbit Stream
Now I re-enable AVC and run the exact same 100mbit stream. wow ok we are looking at 75%-ish cpu. Clearly AVC is causing a CPU bump – that has to be my problem at higher speeds.
Conclusions and Recommendations
– With AVC running in FlexConnect mode, the AP is responsible for the nBAR engine, which is limited compared to what you get in a real WLC. If you need/want AVC – plan on installing a full WLC, between the limited AVC capability (well document) in FlexConnect mode, and the un forseen performance issues I have seen (not well documented) It shouldn’t be used in Mobility Express or FlexConnect installs.
– Running the latest code can bite you (I knew this!)
– Always validate your installations, not just for connectivity, but for performance
– If you are using Mobility Express – Learn the CLi, because there are just some things you cannot do in the GUI.
– I did go back to the latest 8.5 release to see if AVC was the cause of my 5GHZ issues in 8.5.103 – but it was not.