//Network Device Telemetry at Scale

Platform Engineering

Network Device Telemetry at Scale

M Mentat CollectiveDec 15, 20228 min readShare

Introduction

Detecting and visualizing common errors and misconfigurations across your network

Galaxy-style Mentat image used as the hero for the telemetry post

All too often technology managers believe that if you use the best of breed tools nothing can go wrong. I mean it makes sense, right? Pick the best, buy the best, implement the best and you should have a fairly easy job managing your infrastructure. That logically does follow and makes a ton of sense on the surface. However, this is far from a tangible reality.

To mitigate failures it is important to approach the full lifecycle of any system with the mindset of anything that can go wrong will go wrong. Werner Vogels, CTO at Amazon, said it well: failures are a given and everything will eventually fail over time, from routers to disks to memory to transient network errors.

So we plan for failure. But what does that practically mean? It means building systems that embrace failure as a natural occurrence and keep running even when the house is on fire. It also means being able to manage the affected pieces without taking the whole system down.

At this point some of you are probably thinking, these are great ideas but honestly I do not know where or how to start. Fair. The first thing that you need to do when planning for failure is visibility. You need to be able to see what is going on before you can create a solution. You cannot fix what you cannot see.

The Problem

One of our clients approached us to see if we could create some automation to help fix a slew of network issues. For years the client had experienced slow network performance, frequent outages, and every time they fixed one issue another one cropped up.

The client was in the unenviable position of fighting fires 24/7 and never locating and remediating the root cause of the network issues.

Solution Tech Stack

Azure DevOps — The Orchestration Engine.
Azure Key Vault — The Secret Keeper.
Azure Kubernetes — The Work Horse.
Elasticsearch — The Storage Engine.
Kibana — The Visualizer.
Ansible — The Configuration Code.
Terraform — The Builder.

The Plan

Before we could hope to create network automation solutions we needed visibility, we needed telemetry. We needed data insights before creating code. Without eyes on data and tools that enable trend analysis and machine learning, we would end up in the same position as the client, always writing automation code to fix random issues.

We selected Azure DevOps as the orchestration engine because it was natively Azure Active Directory integrated and the client was already a heavy Azure user. Azure Key Vault became the home for secrets. Azure Kubernetes Service became the scalable work engine for pipeline jobs. Ansible handled the actual network device connections and data collection. Elasticsearch and Kibana formed the landing zone and visualization layer for telemetry.

Getting Things Wired Up

The pipeline is separated into variables, parameters, resources, the AKS pool where work is executed, and the release stages. Those release stages contain both dynamic tfvars generation and reusable pipeline templates.

Once the orchestration components were functional, we deployed Elasticsearch into an on-prem OpenShift cluster. Because Elastic can get unwieldy quickly at scale, we chose Kubernetes to get faster deployment, scaling, and infrastructure-as-code benefits. We also codified the index schema, Kibana dashboards, backup configurations, ILM policies, and users and groups.

Terraform release pipeline
# Terraform release pipeline
variables:
  - ${{ if or(eq(parameters.environment, 'dev'), eq(parameters.environment, 'tst')) }}:
      - group: terraform-variables-dev
  - ${{ if eq(parameters.environment, 'stg') }}:
      - group: terraform-variables-stg
  - ${{ if eq(parameters.environment, 'prd') }}:
      - group: terraform-variables-prod
  - name: download_tf_vars
    value: 'true'
  - name: tf_directory
    value: $(System.DefaultWorkingDirectory)/$(agency)/$(provider)/${{ parameters.environment }}/$(tf_module)
  - name: TF_VAR_TF_STORAGE_ACCOUNT_NAME
    value: terraform${{ parameters.environment }}
  - name: TF_VAR_TF_STORAGE_KEY_SUFFIX
    value: -$(tag)-$(environment)-${{ parameters.statefile_suffix }}
  - name: provider
    value: azure
  - name: environment
    value: ${{ parameters.environment }}
  - name: tf_module
    value: map-base
 
parameters:
  - name: subscription
    type: string
    default: azure-core-infrastructure
    values:
      - azure-core-infrastructure
      - azure-services
  - name: environment
    displayName: environment
    type: string
    default: dev
    values:
      - dev
      - tst
      - stg
      - prd
  - name: statefile_suffix
    displayName: Identifier of the statefile.
    type: string
    default: '1'
  - name: destroy_validation
    displayName: Type DestroyMe (case sensitive) if intending to run the destroy stage on a PRD environment.
    type: string
    default: N/A
 
resources:
  repositories:
    - repository: templates
      type: git
      name: platform/pipeline-templates
      ref: master
 
pool: $(pool)
 
# Release Stages
stages:
  - stage: Variables
    jobs:
      - job:
        steps:
          - template: ../common-pipeline-yamls/terraform-tfvars-azure.yaml
          - task: PublishPipelineArtifact@1
            inputs:
              targetPath: $(tf_directory)/terraform.tfvars
              artifactName: terraform_parameter_vars_$(tf_module)
  - template: terragrunt/terragrunt-preflight-check.yaml@templates
  - template: terragrunt/terragrunt-plan.yaml@templates
  - template: terragrunt/terragrunt-cost-estimate.yaml@templates
  - template: terragrunt/terragrunt-apply.yaml@templates

From Data Landing Zone to Telemetry

And, just like everything else we code up, we used pipelines and our orchestration engine to run rapid deployments of Elastic into OpenShift. If you are interested in deploying Elasticsearch into Kubernetes, check out the Helm charts here https://github.com/elastic/helm-charts.

Hrm maybe we will do a blog just on this topic this year. Anyway, back to the solution.

Now that our automation services were deployed and we had a data landing zone in Elasticsearch, we moved onto Ansible. We used the following repos as a base and then forked the code to meet the specific needs of the client:

https://github.com/CiscoTestAutomation/pyats
https://github.com/CiscoTestAutomation/genieparser
https://github.com/clay584/parse_genie

Main Playbook

Our main playbook file ended up looking like this:

We call in the roles we forked.
Pass in the Elasticsearch username and password secretly.
Read and show interface data.
Push that data to Elasticsearch.
Network telemetry playbook
- name: Get Network Structured Data
 hosts: "{{ target_hosts | default(omit) }}"
 gather_facts: false
 connection: network_cli
 vars:
   elastic_user: "{{ lookup('env', 'ELASTICSEARCH_USERNAME') }}"
   elastic_pass: "{{ lookup('env', 'ELASTICSEARCH_PASSWORD') }}"
 tasks:
 
   - name: Read in Pyats Role
     include_role:
       name: ansible-pyats
 
   - name: Read in Parse Genie Role
     include_role:
       name: parse_genie
 
   - name: Execute Show Interfaces - IOS
     ios_command:
       commands: show interfaces
     register: cli_output_ios
     when: ansible_network_os != 'nxos'
 
   - name: Read Interfaces - IOS
     set_fact:
       parse_output_ios: "{{ cli_output_ios['stdout'][0] | parse_genie(command='show interfaces', os=ansible_network_os) }}"
     when: ansible_network_os != 'nxos'
 
   - name: Push to Elasticsearch - IOS
     uri:
       url: "{{ elastic_uri }}/{{ el_index }}/_doc"
       body: "{{ {'@timestamp': '%Y-%m-%dT%H:%M:%S' | strftime ,'DEVICE': inventory_hostname, 'PORT': item.key ,'PORT_DESCRIPTION': item.value.description | default(''), 'CRCS': item.value.counters.in_crc_errors, 'COUNTERS_CLEARED': item.value.counters.last_clear, 'DATE': '%m/%d/%Y' | strftime } | to_json }}"
       body_format: json
       method: POST
       user: "{{ elastic_user }}"
       status_code:
         - 201
       password: "{{ elastic_pass }}"
       validate_certs: "{{ validate_certs }}"
     when:
       - ansible_network_os != 'nxos'
       - item.value.counters is defined
       - item.value.counters.in_crc_errors is defined
       - item.value.counters.in_crc_errors != 0
     loop: "{{ parse_output_ios | dict2items }}"
     loop_control:
       label: "{{ item.key }}"
     delegate_to: localhost
become: false

Outcomes

Significantly reduced network outages and bordered on elimination because the network teams could finally see and track issues within and between device types.
Greater visibility of network issues in near real time, allowing the client to become proactive instead of reactive.
Increased performance as more underlying issues were identified and remediated.
Faster identification of network misconfigurations and hardware defects, including batches of physically defective devices that would have been nearly impossible to spot without telemetry.
New misconfigurations are now identified in near real time and resolved with automation.
Improved quality of life for all client teams: fewer all-day or multi-day conference calls, more sleep, and less stress.
Network Device Telemetry at Scale

Key takeaway

Ready to automate smarter?

Before you automate network remediation, you need visibility. Telemetry turns endless firefighting into trend analysis, root cause detection, and proactive action.