Tips for Cisco Wireless Performance on Mobility Express

Recently I was working on my lab network, and I have an 1831 access point and a 3702 AP.   My comments are specific to Mobility Express, which to be fair is just a regular WLC, running on an access point,  with all AP’s in FlexConnect mode only.   The AP’s are responsible for all packet processing, NBAR/AVC, anything you are doing goes on in the AP’s.

Naturally, I wanted to get the most out of my network, but I ran into a few challenges, and I will document them here….

First, I am far from a CLI expert on the WLC stuff,  I have spent most of my life running WLCs with the GUI – but the Mobility Express series GUI is very simple.     I got much better at the CLI during this.

The latest GUI on 8.5 has an “expert” mode now that lets you play with some of the RF settings, the 8.3 version is pretty simplistic.   So I popped in the 8.5.103 version, and was liking the new GUI.     Everything seemed like it was working….     I applaud Cisco for improving the Mobility Express GUI – it was more simple than some home Linksys offerings in the beginning, this is a step in the right direction.

Let me outline my environment….

  1.  I live in a rural area – there is ZERO wireless noise here, and I control the spectrum pretty well.  I don’t deploy stuff without considering the impact.
  2. I have about a dozen client devices
  3. For all my testing – I kicked everyone off 5ghz, and ran on just a single AP.   Nothing else was in the air – I confirmed this using a spectrum analyzer.

Running the latest bit me

Until I had a problem with my Macbook Air (Early 2014 Model).  If you go and look, many people complain about Apple Macbook Air’s and wireless issues – so many different opinions, some blame Apple, some say replace your “router” or access point but I couldn’t find any kind of real problem.

Not a surprise.  I ran 8.5.103 – and I was having weird problems.    All of my clients were fine except my Macbook Air – as long as it was on 2.4ghz, it was fine – but bump it up to 5ghz, and as soon as traffic started flowing – the AP would simply start ignoring the client.  Client thought it was associated, AP saw it as associated — but no traffic moved.     It would sometimes come back, sometimes not, if I bounced client adapter – it would come right back.  2.4 was solid.

Doing what I always tell my clients – run the “Gold Star” release in this case 8.3.122 – So I put that version in, and let the APs upgrade.    Everything seemed better now – connectivity was solid.      After my findings below, I went back to test 8.5.103 again…

AVC Hurting Performance

So being that it was “working”  I switched to performance testing.   I run a iPERF3 server on my QNAP here at home – confirming performance I was getting 995mb/sec from my wired desktop to the NAS…  Ok we are good to test.

My Macbook Air was connected with the following…

Performance Signal Strength: -53 dBm

Signal Quality: 43 dB

Connection Speed: 867 Mbps

Channel Width: 80 MHz

Capabilities 802.11ac (5GHz) Spatial Stream: 2

Time for a test…

Connecting to host QNAP, port 5201
[ 4] local port 56551 connected to QNAP port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 15.5 MBytes 130 Mbits/sec
[ 4] 1.00-2.00 sec 16.1 MBytes 135 Mbits/sec
[ 4] 2.00-3.00 sec 16.0 MBytes 134 Mbits/sec
[ 4] 3.00-4.00 sec 15.9 MBytes 133 Mbits/sec
[ 4] 4.00-5.00 sec 16.1 MBytes 135 Mbits/sec
[ 4] 5.00-6.00 sec 15.8 MBytes 133 Mbits/sec
[ 4] 6.00-7.00 sec 15.8 MBytes 132 Mbits/sec

Ok this isn’t right…  Something isn’t working….   So I contacted my good friend @wifijanitor – Steve to bounce some ideas off him.    We quickly got to “It’s all configured correctly”
So I started disabling this and that 802.11(insert feature here) and everything one by one. Problem remained.
Finally, I disabled AVC – Application Visibility and Control…

[ 4] 71.00-72.00 sec 46.4 MBytes 390 Mbits/sec

[ 4] 72.00-73.00 sec 46.3 MBytes 389 Mbits/sec

Well look at this…    The only thing we could figure out is that the AP must be getting hammered by the AVC…   So, I investigated that….

AP CPU with AVC Enabled

Whoa, that is 100%…  This is with my iPerf, i’m getting 140-150 mbit. Ok, let’s try with it disabled.
Screen Shot 2017-09-13 at 9.46.09 PM

AP CPU with AVC Disabled – Heavy Load with iPerf

Whoa…  That’s not a good thing…    That means even the performance i’m getting now is probably being hampered by the CPU on board…   Close to 400mbit throughput, and the CPU is high.  According to the system it is nothing but packet process.  There has to be  a choke point…    I wonder what would happen if I had more CPU – i’m not able to clear up any more CPU, everything (I think) is disabled.
Screen Shot 2017-09-13 at 9.46.09 PM

AP CPU with AVC Disabled – 100 Mbit Stream

Ok so i’m trying to prove my theory…  This is AVC Disabled, 100MB Stream using iPerf.  About 30% CPU utilization…

Screen Shot 2017-09-13 at 10.17.18 PM

AP CPU with AVC Enabled – 100 Mbit Stream

Now I re-enable AVC and run the exact same 100mbit stream.   wow ok we are looking at 75%-ish cpu.    Clearly AVC is causing a CPU bump – that has to be my problem at higher speeds.

Screen Shot 2017-09-13 at 10.12.25 PM

Conclusions and Recommendations

– With AVC running in FlexConnect mode, the AP is responsible for the nBAR engine, which is limited compared to what you get in a real WLC.    If you need/want AVC – plan on installing a full WLC, between the limited AVC capability (well document) in FlexConnect mode, and the un forseen performance issues I have seen (not well documented)  It shouldn’t be used in Mobility Express or FlexConnect installs.
– Running the latest code can bite you (I knew this!)
– Always validate your installations, not just for connectivity, but for performance
– If you are using Mobility Express – Learn the CLi, because there are just some things you cannot do in the GUI.
– I did go back to the latest 8.5 release to see if AVC was the cause of my 5GHZ issues in 8.5.103 – but it was not.

Cisco releases Spark and Jabber Interop

Good news the Cisco Spark and Cisco Jabber interoperability is up and running and working.   People using Jabber 11.9+ can now see and chat with people on Cisco Spark, but one to one only.   Soon spark spaces will also be supported.  There is some presence sharing between the platforms.


The Jabber Platform does not support access to Spark Spaces, ad-hoc group chats, the share of attachments or screen sharing across platforms at this time.  So 1 to 1 chat is all users get for now.   Details of new features in this link.

A few deployment requirements….

Jabber must be CLOUD DEPLOYED – so no on premise deployments are currently supported.  You will need to contact Cisco to get your system added to the Cisco Spark platform identity service.

This is a long time coming feature many clients have been asking for as customer embrace both platforms, however, most clients looking to migrate are probably using an on site implementation of Jabber today.  Hopefully, Cisco will support on premise migration strategies soon.

Cisco dCloud Team Releases SD-Access V2

The Cisco dCloud team has released SD-Access V2 lab which includes DNA Center.

Due to the dCloud environment being so popular you may need to wait until later this week to get your hand on it, but the good news is, it delivers.    Many have been asking about getting their hands on DNA Center.     This is a BETA – so following the lab guide is advisable or things may not work – keep in mind it isn’t actually programming real switches in the back end.

2017-07-11 15_20_06-4D_SD_Access_v2 (1).pdf

Well it is here, and you get to setup a new network, deploy SSID’s, and build policy.   Right now this is just a DNA Center demo walk through.   You will get the change to design, provision and build policy in the live demo.   DNA Assurance – NDP or Network Data Platform is not available at this time.

The team was quick to get this demo in our hands, so go out there and get your hands on DNA and see how intuitive you think it is.



Cisco Announces “The Network. Intuitive.”

With content courtesy of Cisco Systems

Last year I broke down the Cisco DNA – Digital Network Architecture in an article called “Beyond Marchitecture”, because quite frankly, it was a ton of marketing with little substance.

This year at Cisco Live! 2017, Cisco has done this the right way.   With a new campaign, backed by the technical prowess we expect from Cisco and launched with all the big names, and big programs we expect.  This was well thought out, and if this is what Chuck Robbins is going to bring to the table of Cisco Systems – there should be some big things ahead.

In a series of interviews with different business units, it was revealed that the “Handcuffs are off” and departments have been given the ability to innovate, collaborate and tear down the silos.  This new program demonstrates that.

The Network.  Intuitive.

2017-07-05 11_36_44-DNA for CL Vegas.pdf - Adobe Reader

First get past the grammar related issues of the new DNA Campaign, and realize that is it not “The Network Intuitive” it is “The Network. Intuitive.”  – punctuation matters here

The key to understanding “The Network.  Intuitive.” is in two powerful words.


As announced by Chuck Robbins in the Cisco Live keynote, they want you to power your network with business intent.   No more programming VLANs, or setting up routing, but truly going into a unified console and telling it what you want to do.

“A computer will do what you tell it to do, that may be totally different from what you had in mind” — Quote Unknown

The idea that “Machine A” can talk to “Server B” and “User Y” and talk to “System X” without worrying about the underlying infrastructure is where they are going.

This is a construct, not a product, but unlike DNA-2016, there is a strong technical basis for this idea.


Intent does not do you any good, unless you have context in your network.   We need to understand, who is where, and understand what they are before we can set our intent against that object.

Chicken before the egg syndrome a little bit, how do we secure, route and prioritize our network, if we do not know what this traffic, who they are and what they are trying to do.  Today context generally comes from things like IP Addresses and subnets.    In DNA-2017, this context come from Cisco ISE.

The Network. Intuitive.  InfoGraphic.

2017-07-05 14_38_42-DNA for CL Vegas.pdf - Adobe Reader

The latest info-graphic from Cisco really does provide a good overview of this new architecture.

The underlying technology for this new intuitive network technology is SD-Access – Software-Defined Access.  This of “ACI – Application Centric Infrastructure” but now it is user centric – make our decisions and policies and apply them to users, and where they are is unimportant.

SD-Access Building Blocks


I want to help build the SD-Access story for you, so you can understand how this technology comes together.  Like like years DNA announcement, SD-Access is a reference architecture, but there are bespoke technologies around it.

Transport Layer – Network

At the very basic transport layer, SD-Access relies on a few switch options that are available today.      Supported on Catalyst 9K, 3650, 3850, 4500E, 6500/6800 and Nexus 7K.  Wireless options are 3800, 2800, 1560 and controllers 8540, 5520 and 3504.

The new one to this party is the Catalyst 9000, developed by the team at Cisco with the new DopplerD series CPU with tons of power and supporting ETA – Encrypted Traffic Analytics.    Please see my future blog post on the Catalyst 9000 series.

These devices do all the transport and implementation of policy in the background of SD-Access and move the bits around your network


Understanding the Campus Fabric

The underlay network will transport your traffic from place to place, this is what makes up your campus fabric.   True virtual networking to the endpoints through encapsulation, not just through VLANs anymore.    The idea is we want to segregate the forwarding plane, from the services plane, why should our physical network dictate how traffic flows around our network, but how can we add capabilities without massive complexity.

2017-07-09 07_46_09-(48) TechWiseTV_ A Deeper Look at Software-Defined Access - YouTube

If you want me to sit here and admit that this is as easy as the old VLANs and IP addresses in your network – it simply is not.   However the security, control and simplicity once it is implemented is worth it.  The automation and contextual data you will receive.

The transport does not need to be complex, by using an overlay, we can deliver features through the overlay, and the underlay network, the hardware does not need to be complex.

LISP – Location Identity Separation Protocol – Layer 3

This bring together location and identity.    Think of the old way for a moment, we know switch port, and IP address or subnet, and we have a weak idea of the context of a user, who and where they are.  LISP takes the IP and Location and segregates them so that IP and Location are not tied anymore.

LISP is like DNS for packets,  when a switch needs to forward packets from place to place, LISP identifies to the network device locations and the routes required using a map server or resolver.   This could be an IOS device or a virtual machine somewhere.   LISP allows a device to live in any place on the network.  Getting in and out of the LISP environment is via a tunnel router or “XTR”.

This is what provides mobility of devices around your network, even if a user moves to another building or another floor, the IP address of that user does not change – they just move from place to place and the map system handles where that user is

VXLAN – Layer 2

Wait, why is VXLAN showing up in the access layer?   Well, LISP is really a layer 3 technology, it ensures that packets can route, but what if we have users across multiple layer 3 areas that need layer 2 connectivity?   What about multicast and broadcast traffic.

VXLAN provides the transport of our layer 2 traffic across our campus fabric.

Transporting Policy with Cisco TrustSEC

We can now add contextual information into the VXLAN headers through “SGT” or scale-able group tags.   We need to use TrustSEC so that we can apply policies against objects but not based on their IP, but their identity.     Instead of using the IP address, we use the SGT – tag to tell the rest of the network who owns this packet so we can make decisions based on security.  SGT is applied by ISE and then access lists and rules are applied against security groups, users are placed in those groups within ISE.

Identity Layer – Context

This is where the context comes in.   ISE – Identity Services Engine is used to create network identity for objects, users and systems.    I know what some of you are thinking “Oh no – ISE”.   Have you taken a look at ISE 2.1+ ?  They have vastly improved the experience.    There is no question that adding ISE will complicate your life, but it is the contextual engine that provides the data you need to secure your network.   There is no avoiding ISE anymore, you will need to have it in your life, and your network.


There are benefits here, once ISE is implemented, all of your network devices start to see things are user activity, firewalls show users names not systems, you can start deploying policy against groups of objects and network authentication becomes very easy.   Your wireless network becomes easier to manage from a security perspective.

Interface Layer – Intent

This is the real veggies.   DNA Centre is the new package for the APIC-EM platform.    This is Cisco’s single pane of glass attempt by Cisco so make a UI front end for your network, the intent is a single pane of glass for your ENTIRE network.


This is where your contextual groups from ISE like users and servers will meet up with the policy you want to create.   There is no denying the interface is a little “Meraki” like, clearly they borrowed some design concepts.    All of the complex components of SD-Access meet here in DNA Centre, and are then pushed out to the rest of your network.   The automation from DNA Centre will automate everything for you.  From dealing with ISE to programming those Catalyst switches.   This is the automation layer.  Set what intent you want, and automation will turn that into action down on your hardware layer. Worrying about all this VXLAN and LISP stuff?  No worries, DNA Centre will help you here.

2017-07-09 07_56_42-(48) Cisco SD-Access - Campus Fabric with DNA Center Automation & Assurance with

NDP – Network Data Platform

No shortage of data about our network, we have NetFlow and Syslog and any number of tools to deliver data.   In the coming months as we get a better look into the new Network Data Platform, we will learn how this will help correlate network data and provide analytics.   This is where the old “Proverbial lead into gold” promise is supposed to deliver.   For me this is a wait and see approach, right now there just isn’t enough data out there, for now that is all I have to say.  This is still very early.


More to come in future posts about Catalyst 9000 and DNA Centre, NDP and ETA.



With content courtesy of Cisco Systems


Cisco Live! Returns to Hot Breakfast!

Last year I wrote this blog about the breakfast offerings at Cisco Live!, outlining the importance of a good breakfast on learning comprehension.   I made sure that this information received wide distribution, and many of you helped with your retweets to the team @CiscoLive, and as a result it did become something considered this year.

I have been advised by the Cisco Live team that hot breakfast sandwiches have been added to the menu for Monday through Thursday!    This is amazing news.    I have to thank in particular Kathleen Mudge @KathleenMudge  for helping spread the word at Cisco Live offices.

“Food is like a pharmaceutical compound that affects the brain,” – ULCA Professor of Neurosurgery and Physiological Science Fernando Gómez-Pinilla.

Short term memory and auditory attention are higher when a breakfast offered with protein as opposed to refined carbohydrates is offered, no more sugar crash, and power through your day.

For me personally, this is important, as I have recently embarked on a low-carb Ketogenic lifestyle.  More options give us better ability to learn and interact.

This is a great win for all delegates, and for the social media community as a whole.



Cisco Live! on a Budget – 2017

Last year I wrote a great article on “Making your Case for Cisco Live” – Click Here – that article was all about how to get your boss to pay for Cisco Live, and why Cisco Live is a great value.    If you have not read that – go back and read that.   I even provide some tips on how to get free passes if you are a Cisco customer, or how to show your boss that Cisco Live! is cheaper than traditional training.

Why Cisco Live?

First, I want to talk about WHY you need to get to Cisco Live US – #CLUS.   For your career, for your job, for YOU.

Cisco Live! has some great tips on “Why” attend, I will not list them all – CLICK HERE – and I will show you all the right reasons.

There’s Never Been a Better Time – to go to Cisco Live and find out what you have been missing.

  • Breakout Sessions, Content, Content, Content…
  • DevNet
  • Seminars
  • Walk In Labs
  • World of Solutions

The bottom line is – there is more to do at Cisco Live, than you have time for, and you really do need to think about, and plan how to get the most out of your week.

I am budget constrained!

No problem, if getting your boss to pay is a problem, or you need to go on a budget (still, get the boss to pay, you need to really make your case!) this will give you tips on how to get the most – for the least at Cisco Live!

A full trip to Cisco Live! for the full conference experience is going to cost you close to $5000 USD – if you get the full conference pass.   That is the bottom line,  between airline tickets, the $2300+ full conference pass and hotel – you are approaching some big bucks.   Don’t let this get you down – you are still going – and for much less.

The Explorer + Social Pass – The Hidden Gem

I am sure the event does not want everyone figuring this out – The Explorer Pass is the best value – and I will show you how to save yourself $1900 right now – and still experience it all.     Yes, all of it.

$249 – Miss (Almost) Nothing.

For the price of “Explorer + Social Pass” which is only $249,  the only things that you are missing at Cisco Live! is the following….

  • Cisco Live T-Shirt (Trust me, you will go home with enough T-Shirts!)
  • Cisco Live Bag (If you ask around, tons of people give their away you could get one)
  • Breakfast and Lunch (Read my blog HERE about breakfast – not a big deal IMO)
  • Breakout Sessions (I will address this)
  • Your attendance does not count towards NetVet status
  • No Free Certification Exam

If you want to save an extra $150, you could get only the “Explorer” pass, but then you miss out on the “Social” part of Cisco Live, and I DO NOT recommend this,  there have been enough BLOGS out there about why Cisco Live! is all about SOCIAL.

What do you get?

DevNet Zone

Are you a developer?  Do you want to be?  Are you getting interested in the new SDN, SDWAN, XML, REST-API – are you trying to catch up in this new software defined programmatic world we are in?   Then DevNet zone is for you – you could literally hang here all week, there are tons of activities and learning opportunities.      This area should be called “Industry Shift Zone”  because this is where you will see what really is up and coming, and new ways of thinking.    Not to be missed

World of Solutions

This is where everyone goes for free stuff – but – this is where you go to learn from everyone who sells complimentary products – and they don’t tend to only send marketing people but real engineering types.     Here is the secret – Cisco has over 30% of the floor space in World of Solutions.   Lots of the content you see in breakouts is also duplicated here, and you can go one-on-one with a lot of the product teams.     I try and track down those hard to find Cisco engineering types on specific technologies, get some answers and learn about things.     Another cool trick, is if you want to integration product A + B – perhaps you want to link ACI with ASA Firewall – goto the ACI booth, and then drag that person over to the ASA booth (or vice versa) and then have a conversation – ok be nice about it, but you get the idea.   WORLD OF SOLUTIONS IS WORTH THE PRICE OF ADMISSION ALONE.   You could spend all week in here.

Customer Appreciation Event (CAE)

It is a concert, it is a great time, and you get to see/hang/learn and collaborate with like minded people – the nerd knobs never stop, and the CAE is a great place to go to network.


You still get access to keynotes, and there is nothing more inspiring than listening to some of these amazing speakers – live – in person.  If you don’t make it into the hall, don’t worry it is simulcasted all over the event.

Breakouts via Cisco Live! 365 Access

Didn’t I just say you don’t get breakouts?  That’s right – live – you don’t – but who says that you need to see them LIVE.   With access to Cisco Live 365 online – you can see almost every single breakout – online.     “But what if I have questions”  well, there is a good chance someone will ask it.

Here is another tip – go ahead and watch the breakouts you WOULD have seen – at Cisco Live Europe on Cisco Live 365 – before the event,  now you are ahead of the game.   Once you reach the event, you can use access to World of Solutions or DevNet to go ask questions.


No question, this can get expensive, if you stay at the Mandalay (Assuming you get a room) you are $490/Night+ – CRAZY.      If you stay just 2 doors down at the Excalibur, rooms as I write this are $69, and it is walking distance, or take the tram.    There are rooms for as low as $49 a night – and if you read my blog from last year, you won’t be in your room much anyway.


This is where it becomes difficult, because travel is always a challenge – and I don’t know where you are coming from, but you need to get “fancy”


If you are in the western half the USA – you have Friday night till Sunday night to get there, so you don’t lose much of your work day, so driving might be an option for you.    Don’t worry about parking, if you are a member of any M-Life hotel program (free sign up) parking is free, or maxes out at $30 (if you “lose” your ticket) and there are many other free parking offers.

If you drove from Chicago and back, it would cost you about $270 in fuel in an average car, leave the F-150 at home, and grab that Toyota Echo.

Even if you are coming from as far as Florida or Chicago – this drive is doable and can anyone say ROAD TRIP!?


I am not an “American Flight Expert” as I am Canadian but Google Flights, Travelocity, tons of other sites give you the ability to find reasonable flight options.   As I look right now you can get flights that run from $300-600 – and if you play with your dates you can reduce it a bit, remember to consider it might be worth staying an extra night on either end to bring the flight cost down.

The Sub – $1500 Live Trip

You can do it,  $249 for your ticket, $276 for your room, $600 for your flight – a little spending money for food – YOU CAN DO THIS – for less than $1500.

So what are you waiting for – CLICK HERE NOW – see what you would miss out on, and sign up now.


Cisco DNA Series: The QoS Evolution

This is a game changer, and this will be a long blog post.   Cisco is flipping the script on QoS.   Quality of Service – will now become Quality of Experience.   This isn’t a marketing term either.   Come along for a ride as I explain.

First some references, the amazing team at Tech Field Day – and the Cisco Team who presented at Tech Field Day Xtra at Cisco Live this year provided so much insight.   As I talk about this, I will provide some links to videos, or specific parts in that presentation.  Some of my graphics have been pulled from that content.   Tim Szigeti is an amazing knowledgeable professional a true leader in the field, and Ramit Kanda provides an amazing demo on this great new technology.

A history lesson…

QoS…  Since the day I took the Cisco CVOICE course, I was learning about protocols and methods of qualities of service.   The construct is simple – we need important stuff to be first.   Quickly this became a topic even the top network professionals – CCIE’s couldn’t handle.

Cisco Enterprise has a Vision..  “Transform our customers’ businesses through powerful yet simple networks” — powerful..  yes..  simple.. no so far…

As networks became constricted in bandwidth (mostly in the WAN) we needed a way to constrain less important traffic.    The start of QoS was in the VoIP world – as people like me (hard core telephony guys from the TDM days) started to work on VoIP, we wanted circuit switched performance over packet switched networks.   Zero packet drops, little jitter and delay.

We started with ToS (Type of Service) – a small field in the IPV4 header that gave us some bits we could set.   3 bits should be enough for anyone — yeah right, just like “640KB should be enough for anyone”.   For most enterprises 8 classes is enough – but for service providers, not so much.

Then there was vendors who treated TOS and DSCP bits differently, or put them into different queues and treated them differently

QoS is second only to routing in the network when it comes to adoption – but how many customers are deploying it properly.  Stay with me – we have new tools for you.

“It takes [us] 4 months and $1M to push a QoS Change… ”  says a Wall Street Financial company.

“It took us 3 months to deploy a 2 line ACL change across 10K devices, which slowed down onboarding of our Jabber application” – says a Cisco Network Architect

QoS is Too Hard

“With QOS – the #1 TAC case report – is missing or incorrect classification and marking” – says Tim Szigeti – Cisco Systems

In a recent group of CCIE’s, and some others who I also respect greatly for their knowledge they all agreed “QoS is too difficult” – just get more bandwidth.   Let me provide some illustration.  This is the way a 2P6Q3T router would classify these categories into queues.

As I go across my network – each device I have has a different QoS architecture

QOS Devices

Let me save you – don’t bother reading the below graphic – you get the point.  Can you, as a professional, trap and trace a packet as it flows across the network to ensure it is getting the treatment you want?   Can you design how to deploy a new application into this many different queuing mechanisms?  Do you even want to?

QOS Arch

What if I wanted to provide QoS for all 1400 applications that a network device supported?

1622+ lines

Here is a hint you don’t want to do that.

“We have done more to advance QoS technology in the last year, than in the last 10” says Tim Szigeti from Cisco Systems.

So Cisco made it better, — but this is still too much


Cisco Validated of Design – Classification, Marking, and best practices – 2 lines of code.   This is a huge day for QoS design.   This will be consistent across ROUTERS AND SWITCHES – all products, all lines.   So even if you are doing this in the CLI this is good news.    Cisco is moving to a single design in hardware as well in the future.  5 Queuing structures will be the future – but still only a single reference design.    Why can they not create a single structure?   Cost.    However now it has a reference design.



More Bandwidth Does Not Solve It!

HOLD THAT THOUGHT – No, more bandwidth does not solve QoS problems.   It might sound like it does on the surface – lets dig down a bit

“Bandwidth and Utilization is not an accurate way of assessing if there is a QoS Problem” – says Tim Szigeti of Cisco Systems

  • Security  – As a construct, QoS has a place, we can limit risky traffic, questionable traffic or scavenger traffic so that it cannot overwhelm our network and shut us down, and stop the speed of attacks
  • Cost – You cannot simply add bandwidth forever – your costs would simply continue to go up and up.   On that note, until now, it has been cheaper to deploy more bandwidth than configure QoS – in some situations, but that does not address the security concern or….
  • Buffers – That’s right, buffers.  Micro bursts – even with the highest performance switching ASIC – at 1% port utilization, with a micro burst we could see traffic being dropped.


Cisco DNA – Automation

If you recall in my recent article we talked about automation being at the heart of DNA.  If we want to make things simpler, automation is the only answer.


Wait a second – isn’t this SDN?   No this is automation!  Most SDN solutions – including Cisco’s own ACI – include forklift.

Cisco APIC-EM for QoS works with existing networks (brownfield!) – You can even abort the installation APIC-EM EasyQOS at anytime.  So if you deploy EasyQOS as I am about to show you – but decide after you do not like it – you can remove it – even if you made other network changes later, it tracks every single change and will set back exactly what it changed to QOS and QOS only.

“People that are really serious about software should build their own hardware” (Alan Kay – 1982) that is why Cisco developed the UADP (Unified Access Data Plane – Code Name Doppler) and the QFP (QuantumFlow Processor – Code Name Yoda)

This is all about controlling and automating that high performance hardware and pushing that configuration in a consistent way down to the network


Wait a second ago did you not say many of the queue architectures are different?  How do you address that?


EasyQOS – The APIC-EM Secret Weapon for Quality of Experience

Why is this important – the idea is simply this.   EasyQoS will allow you to program BUSINESS INTENT in your network.   You tell the EasyQoS application in APIC-EM how you want traffic to be treated, classified and prioritized.   The APIC will figure out how to apply that business intent – against all of the various QoS architectures in the routing and switching platforms that you have.


QoE via EasyQOS – How It Works

It goes without saying – this is an APIC-EM app.  So – go and get APIC-EM installed, and then come back.

The key architectural thing you need to understand is – 3 policy constructs are used here, to abstract 12 classes.   You will see that in a minute.

Step 1:  Create a scope

For your devices, create a scope in the APIC-EM for your devices, and then add the appropriate devices to the scope.

EasyQOS Step 1

Step 2:  Define Applications

Within EasyQOS there is 1300+ applications that are pre-defined, plus you can define your own applications based on a variety of factors.

Each application there is a traffic class.

You really want to create “Favourites” here, within the interface you can “star” and mark your applications as favourites, this is a good way to track which apps you are actually creating policies for.


Step 3:  Define Policy

We need to apply these applications to a policy,  within the policy we have classes of traffic – but think of this as business intent – not QoS.

There are three basic classes.     You simply drag and drop each application into each policy.


  • Business Relevant – This has 10 classes within it based on the application, but do not worry, the APIC will automatically define the business relevant apps to an appropriate class.  This is all under the covers
  • Default – Traffic you don’t really care about, this is your Best Effort class
  • Business Irrelevant –  This is your scavenger class

Step 3: Apply Policy

The policy uses various types of connections, today it uses SSH – and YES you can validate the commands before they are sent.


Any interface changes are detected by SNMP, or through polling every 30 minutes in case you change things by hand.   The changes are sent out immediately.

If during the provisioning you realise something is wrong, or something fails – the APIC tracks every transaction on every device.   You can abort a provisioning half way through – and it will back out each individual change.



Operational Features

Now we have this running.   We have some other cool tools that make our life easier.


The first is a history engine, any changes will be tracked so you can see the changes in the policy over time – so if you make changes, then realise you had an adverse affect, a simple fix is to hit “Rollback” — keep in mind, this could be 500 devices on the network.   The old way you spend a month making QOS changes – only to realise those changes are detrimental – you spend a month removing them.   In APIC you can make, and rollback these types of changes in literally minutes.   Huge cost and time savings here.


Dynamic QoS

This one is pretty crazy sounding, but for VoIP and Video, we cannot always track these by application, they are encrypted or dynamic.


So the way this works is – Jabber or Lync sends a call setup – the APIC is informed of this call, and the APIC sends a NEW QoS policy — for just that call — to all the network devices in the path.

If you are reading this and thinking “So you are telling me my QoS Config is going to be modified every time someone makes a call” — Yes that is exactly what I am saying.    I am not sure I am on board with this idea – that is a lot of dynamic network changes.  Cisco says “it works!”

Show Me The Money – Path Flow Analysis

This is the most compelling part of APIC-EM EasyQOS.   Bar None – Hands Down – Mic Drop.

You can perform Path Flow Analysis, on every device – instantly.

  • Including interface stats
  • QOS Stats
  • ACL Rules blocking traffic
  • Interface Stats

Step 1: Input the path trace data


Step 2:  Flow Visibility

Prepare to be blown away.   Here is the application flow.  It even looks inside CAPWAP tunnels.   If you had to do this by hand you have to do this per flow, in every single device.   To set this up alone would take you hours, then analyze the data, then remove that config.

The APIC-EM does all of this for you – in seconds.


Device Health, performance stats, packet loss, DSCP values, Jitter, even routing protocol information.  Router CPU level, Memory use.   If you are troubleshooting a network – this is literally gold.   “All hail the packet – for it runs on the network”  did Denise Fishburne herself call someone up and help them build this?   They should call this the APIC-EM Network Detective!


Here is a great example of an ACL block – imagine if you had 200-300 ACL’s on this device, finding the one that is causing problems would take you forever.



Even Asymmetric Flows.  Every device, every hop.   Even if you didn’t use EasyQOS this is worth the time to deploy APIC-EM.


Watch the last few minutes of our video from Tech Field Day and be BLOWN AWAY.   A room of CCIE’s clapping tells you how amazing this is.


Prove it – with Validation of Experience VoE


The functional architecture of the validation of experience is an analytics engine.   I would like to put a caveat on this discussion – this is still a bit of a proof of concept discussion.  There is limited actual capability that you can deploy at this moment – but this is the functional way this will work.

Functional Layer 1 – Instrumentation

Collect all the right things, no silent drops in hardware – collect all the relevant metrics.  Right down to the application layer if we can, as an example – Jabber.  This means not just network information, but application level metrics like video or audio frame drops.  If we want to monitor experience – we need to go all the way to layer 7


Functional Layer 2 – On-Device Analytics

We may not need to collect and return everything, but some of these are critical.  So we need to analyse them on the device, decide what is critical and then return that.


Functional Layer 3 – Telemetry

Get the critcal information off the device – we don’t want that data sitting there, we need to collect it to the analytics platform. (Cisco is still working on the analytics platform).  SNMP/MIB is simply not enough.


Functional Layer 4 – Real-Time Monitoring

We need to get alerts.  Real-Time, not in an hour.  If we make a change, and we cause a negative affect to the network, we need to know now.  Real-time monitoring of application experience and performance.


Functional Layer 5 – Scalable Storage and Efficient Retrieval

Store these analytics somewhere, with an interface to access this data.  Scaleable storage – even in the cloud.  All the information from all of the devices in the same location.  This is key, without a complete picture, from all devices and applications in the network – we cannot validate or analyze the true experience of the user.


Functional Layer 6 – Analytics

Correlation of data now results in information about network quality.  We can identify where problems are in the network or applications.


Functional Layer 7 – Troubleshooting

Now can identify the root cause of problems with the network.   Remember the quote from earlier – the #1 QOS TAC ticket is incorrect classifcation and marking.


Functional Layer 8 – Self Remediation\Troubleshooting

The holy grail – find the root cause – and fix it.



Summary – Justin’s Opinion

So, after all of that – what do I think about this.    Game changer.   The troubleshooting tools save hours and hours of time, one of my colleagues mentioned “Mean Time to Innocence” MTTI – how long it takes to prove, it wasn’t the network at fault.   With path flow analysis like this, we can prove the network out in seconds.

The ability for us to take BUSINESS INTENT and map it to technology in an intelligent way that is automated is how this will program the network to “Intrinsically know what the business needs, and then just does it” — that is delivering on the promise of the marchitecture.

QoS has been way too difficult for way too long, we NEED this type of tool, the cool part is that REST-API’s are all published, so other vendors are already starting to take advantage of EasyQOS in their own applications.   I cannot wait to see what comes out of Cisco DevNET.   Just imagine the packet analysis and tracing tools that could use the troubleshooting engine in interesting ways.

We are not fully there, or fully baked yet.  VoE is still a bit conceptual.   What is the holy grail for me would be the following

  1. Program Business Intent via EasyQOS – Quality of Experience
  2. Monitor my network for experience, provide validation of experience alerts.
  3. When problems occur either automatically fix them – or recommend changes.

We are not far from this – the team at Cisco says “it’s in the pipeline”

My recommendation – if you are not up to speed with APIC-EM – you better start, because networks have finally burst the bounds of our brains when it comes to understanding everything that is going on – so you need this automation in order to tackle these complex network and application needs.


Tech Field Day Extra – 2016 – Cisco APIC-EM Controller Discussion



Tech Field Day Extra – 2016 – Cisco Validation of Experience with Tim Szigeti


Tech Field Day Extra – 2016 – APIC-EM EasyQoS Demo



Cisco DNA Series: Beyond Marchitecture

Remark:  After posting, I actually had additional thoughts – so I have added them in here.  That’s right, I edit after publishing.

At Cisco Live! 2016, “DNA” was everywhere.   The Digital Network Architecture. Clearly a focus by Cisco Systems.    As someone who received multiple briefings before the big event I kept fighting to get past the marketing.   Even the day of, and throughout Cisco Live, I still struggled to understand what was under the actual hood of DNA.   Finally a light bulb went on.

I want you to stay with me here –  Cisco DNA is like OSI – it is a MODEL.   Most customers will not deploy ALL of the DNA features or architectures.   Some might use one, two or all of them.    You don’t need to use API’s and programming languages to accomplish this, and it isn’t about automating things you don’t need automated.

I am inherently a technical person.   However as you move forward in your career, it is about looking further out, and looking at the 50,000 FT view.   You really do need to look at the bigger picture.

So let me express my dis-content with marchitecture.


“The Network Intrinsically Understands What Needs To Be Done”  — Ok this is what I have a problem with.  You are making this seem WAY easier than it is.  This is a dis-service to the entire IT Industry when you make things sound that easy to CEO’s who fund our IT departments and teams.     They expect autonomic networks that create software, fancy mobile apps and automatically nuclear bomb hackers that attempt to get into my software.   It is not that simple.    This also takes away from the very smart CCIE and other professionals who take YOUR business requirements and transform those into a functioning network.   No network “Intrinsically understands” unless someone tells it how to.   That’s way too Sky Net.

“It’s like turning proverbial lead into gold” — Really?   Come on.

“It is your very own blueprint to success”


This isn’t a 50,000 ft level – this is looking down on Earth from Mars.   If our intention is to create some kind of over arching architecture (say that 5 times fast) that actually functions like Sky Net, then we really do need to go that far back.    The business and “C” level types are going to love this – it sounds amazing.  However the technical people really do see the marchitecture.  So let me the technical people start to drill down.

It starts with the blueprint – that you see above.   For the “C” level types, Cisco is claiming 85% faster network provisioning, 79% reduction in network install costs, 2X software value, 100X faster threat detection and 80% more energy savings and reduced maintenance costs.

IT departments see “Great, so my budget is getting cut, and i’m going to be forced to do more – with less”  Well, yes, but that’s assuming you are RUNNING the Cisco DNA architecture.  This could actually be a way to modernise your infrastructure with promises like that – but be careful, promising 79% reductions in installation costs for hardware might be a bit hasty.  Once you spend the money and don’t return the future savings you could find yourself on LinkedIN Jobs.  Read the fine print (and trust me, Cisco is careful about putting little superscript numbers over every one of those claims)

Even Cisco’s own content on the model is more whiteboard and less hands on.


The DNA architecture and model is more about outcomes, than technologies and products – but somehow we need to get from the promise to production

It really is all about APIC-EM


It is amazing how APIC-EM started as a little platform to do some automation and now an entire architecture has almost been built upon it.

APIC-EM is the automation platform surrounding Cisco DNA.  New services are being developed for this right now.

Not everyone is an SDN believer, infact some think SDN is still an unproven, non standarized technology.  Many are betting on automation and not software definition – some bet on both.    If you are a network professional without coding skills (like me), APIC-EM will seem a little more intuitive.

Cisco Plug and Play,  EasyQOS and iWAN App are the big key points in the DNA portfolio we get from APIC-EM.    Coming soon will be my article on EasyQOS, all I can say is – it will change how you think about QOS, a technology many simply gave up on and said “just get more bandwidth, it’s easier”

More on EasyQOS in my next blog post…   However the key message is that Cisco is moving from QoS to QoE – it is about Quality of Experience – that’s not a marketing term either, in DNA, we tell the system what quality we want for various applications – and QoS is automatically configured for that.  More on that later.


NFV – Not Just For Service Providers

Enterprise “NFV” aims to take out physical Routers, Firewalls, Accelerators and Wireless LAN Controllers in the branch.   The idea is centralised management and deployment with everything virtual in the branch.    This can be run either on a UCS C220 server, or on top of a ISR4000 with UCS-E blade.


Most of the content you will see online from Cisco — is like the above, very abstract.  However we can get more into the meat and potatoes of Enterprise NFV from our friends at TechFieldDay.    Here is an actual demo of NFV deployment with some good questions from the delegates


Security at Heart

TrustSec, StealthWatch and ISE are all the key security products at play in DNA,  I know entire customers who went down the ISE path – and cancelled projects from complexity, so while high security, flexibility and reduced operating costs might be the end result of DNA – security isn’t cheap, and getting there will not be either.    These products can have a long installation cycle / process.

Getting from Promise to Production with DNA Readiness Model

This is where we have an issue, before we can have an elastic multi domain secure flexible network – we need to deploy the tools for DNA.    As Rod Soderbery of Cisco says “Adopting Cisco DNA is a Journey”  – that is for sure, this will not be an overnight change for any organization.


They call it a journey,  start with base automation, move to policy based services on APIC like iWAN and EasyQOS, and then add your more advanced security, think ISE,  more software control and then Digital Services.   Each is a step in the journey to DNA.   I don’t know many organisations that are even close.

Marketing The End State To Start Conversations

This is the problem for IT organisations – “Digital Services” – see that end green bubble, that’s how this is being sold to the “C” level types – they don’t understand the blue bubbles but we all know a lot of work has to be done to reach those trans-formative “Digital Services”

The good news at least for Cisco is that on all the news of DNA and the hype, the stock hit an all time high. If this does nothing more than start the conversations about next generation infrastructure,  next generation firewalls and security products, or maybe the entire DNA architecture then this will be good for Cisco.