Platform Engineering
Network Device Telemetry at Scale
Introduction
Detecting and visualizing common errors and misconfigurations across your network

All too often technology managers believe that if you use the best of breed tools nothing can go wrong. I mean it makes sense, right? Pick the best, buy the best, implement the best and you should have a fairly easy job managing your infrastructure. That logically does follow and makes a ton of sense on the surface. However, this is far from a tangible reality.
To mitigate failures it is important to approach the full lifecycle of any system with the mindset of anything that can go wrong will go wrong. Werner Vogels, CTO at Amazon, said it well: failures are a given and everything will eventually fail over time, from routers to disks to memory to transient network errors.
So we plan for failure. But what does that practically mean? It means building systems that embrace failure as a natural occurrence and keep running even when the house is on fire. It also means being able to manage the affected pieces without taking the whole system down.
At this point some of you are probably thinking, these are great ideas but honestly I do not know where or how to start. Fair. The first thing that you need to do when planning for failure is visibility. You need to be able to see what is going on before you can create a solution. You cannot fix what you cannot see.
The Problem
One of our clients approached us to see if we could create some automation to help fix a slew of network issues. For years the client had experienced slow network performance, frequent outages, and every time they fixed one issue another one cropped up.
The client was in the unenviable position of fighting fires 24/7 and never locating and remediating the root cause of the network issues.
Solution Tech Stack
The Plan
Before we could hope to create network automation solutions we needed visibility, we needed telemetry. We needed data insights before creating code. Without eyes on data and tools that enable trend analysis and machine learning, we would end up in the same position as the client, always writing automation code to fix random issues.
We selected Azure DevOps as the orchestration engine because it was natively Azure Active Directory integrated and the client was already a heavy Azure user. Azure Key Vault became the home for secrets. Azure Kubernetes Service became the scalable work engine for pipeline jobs. Ansible handled the actual network device connections and data collection. Elasticsearch and Kibana formed the landing zone and visualization layer for telemetry.
Getting Things Wired Up
The pipeline is separated into variables, parameters, resources, the AKS pool where work is executed, and the release stages. Those release stages contain both dynamic tfvars generation and reusable pipeline templates.
Once the orchestration components were functional, we deployed Elasticsearch into an on-prem OpenShift cluster. Because Elastic can get unwieldy quickly at scale, we chose Kubernetes to get faster deployment, scaling, and infrastructure-as-code benefits. We also codified the index schema, Kibana dashboards, backup configurations, ILM policies, and users and groups.
# Terraform release pipeline
variables:
- ${{ if or(eq(parameters.environment, 'dev'), eq(parameters.environment, 'tst')) }}:
- group: terraform-variables-dev
- ${{ if eq(parameters.environment, 'stg') }}:
- group: terraform-variables-stg
- ${{ if eq(parameters.environment, 'prd') }}:
- group: terraform-variables-prod
- name: download_tf_vars
value: 'true'
- name: tf_directory
value: $(System.DefaultWorkingDirectory)/$(agency)/$(provider)/${{ parameters.environment }}/$(tf_module)
- name: TF_VAR_TF_STORAGE_ACCOUNT_NAME
value: terraform${{ parameters.environment }}
- name: TF_VAR_TF_STORAGE_KEY_SUFFIX
value: -$(tag)-$(environment)-${{ parameters.statefile_suffix }}
- name: provider
value: azure
- name: environment
value: ${{ parameters.environment }}
- name: tf_module
value: map-base
parameters:
- name: subscription
type: string
default: azure-core-infrastructure
values:
- azure-core-infrastructure
- azure-services
- name: environment
displayName: environment
type: string
default: dev
values:
- dev
- tst
- stg
- prd
- name: statefile_suffix
displayName: Identifier of the statefile.
type: string
default: '1'
- name: destroy_validation
displayName: Type DestroyMe (case sensitive) if intending to run the destroy stage on a PRD environment.
type: string
default: N/A
resources:
repositories:
- repository: templates
type: git
name: platform/pipeline-templates
ref: master
pool: $(pool)
# Release Stages
stages:
- stage: Variables
jobs:
- job:
steps:
- template: ../common-pipeline-yamls/terraform-tfvars-azure.yaml
- task: PublishPipelineArtifact@1
inputs:
targetPath: $(tf_directory)/terraform.tfvars
artifactName: terraform_parameter_vars_$(tf_module)
- template: terragrunt/terragrunt-preflight-check.yaml@templates
- template: terragrunt/terragrunt-plan.yaml@templates
- template: terragrunt/terragrunt-cost-estimate.yaml@templates
- template: terragrunt/terragrunt-apply.yaml@templatesFrom Data Landing Zone to Telemetry
And, just like everything else we code up, we used pipelines and our orchestration engine to run rapid deployments of Elastic into OpenShift. If you are interested in deploying Elasticsearch into Kubernetes, check out the Helm charts here https://github.com/elastic/helm-charts.
Hrm maybe we will do a blog just on this topic this year. Anyway, back to the solution.
Now that our automation services were deployed and we had a data landing zone in Elasticsearch, we moved onto Ansible. We used the following repos as a base and then forked the code to meet the specific needs of the client:
https://github.com/CiscoTestAutomation/pyats
https://github.com/CiscoTestAutomation/genieparser
https://github.com/clay584/parse_genie
Main Playbook
Our main playbook file ended up looking like this:
- name: Get Network Structured Data
hosts: "{{ target_hosts | default(omit) }}"
gather_facts: false
connection: network_cli
vars:
elastic_user: "{{ lookup('env', 'ELASTICSEARCH_USERNAME') }}"
elastic_pass: "{{ lookup('env', 'ELASTICSEARCH_PASSWORD') }}"
tasks:
- name: Read in Pyats Role
include_role:
name: ansible-pyats
- name: Read in Parse Genie Role
include_role:
name: parse_genie
- name: Execute Show Interfaces - IOS
ios_command:
commands: show interfaces
register: cli_output_ios
when: ansible_network_os != 'nxos'
- name: Read Interfaces - IOS
set_fact:
parse_output_ios: "{{ cli_output_ios['stdout'][0] | parse_genie(command='show interfaces', os=ansible_network_os) }}"
when: ansible_network_os != 'nxos'
- name: Push to Elasticsearch - IOS
uri:
url: "{{ elastic_uri }}/{{ el_index }}/_doc"
body: "{{ {'@timestamp': '%Y-%m-%dT%H:%M:%S' | strftime ,'DEVICE': inventory_hostname, 'PORT': item.key ,'PORT_DESCRIPTION': item.value.description | default(''), 'CRCS': item.value.counters.in_crc_errors, 'COUNTERS_CLEARED': item.value.counters.last_clear, 'DATE': '%m/%d/%Y' | strftime } | to_json }}"
body_format: json
method: POST
user: "{{ elastic_user }}"
status_code:
- 201
password: "{{ elastic_pass }}"
validate_certs: "{{ validate_certs }}"
when:
- ansible_network_os != 'nxos'
- item.value.counters is defined
- item.value.counters.in_crc_errors is defined
- item.value.counters.in_crc_errors != 0
loop: "{{ parse_output_ios | dict2items }}"
loop_control:
label: "{{ item.key }}"
delegate_to: localhost
become: falseOutcomes

Key takeaway
Ready to automate smarter?
Before you automate network remediation, you need visibility. Telemetry turns endless firefighting into trend analysis, root cause detection, and proactive action.