ACI Advanced Monitoring and Troubleshooting

ACI Advanced Monitoring and Troubleshooting provides a solid conceptual foundation and in-depth technical knowledge for monitoring and troubleshooting virtually any problem encountered during testing, deployment, or operation of Cisco Application Centric Infrastructure (ACI) infrastructure. Authored by leading ACI support experts at Cisco, it covers all students will need to learn to keep an ACI deployment working optimally. Coverage includes:

• Core ACI concepts and components, including Nexus 9000 Series platforms, APIC controllers, and protocols
• In-depth insight into ACI’s policy model
• ACI fabric design options: single and multiple data centers, stretched vs. multiple fabrics, and multi-pod/multi-site
• Automation, orchestration, and the cloud in ACI environments
• ACI topology and hardware/software specifications
• End host and network connectivity
• VMM integration
• Network management configuration, including SNMP, AAA, and SPAN
• Monitoring ACI fabrics and health
• Getting immediate results through the NX-OS command line interface
• Troubleshooting use cases: fabric discovery, APIC, management access, contracts, external connectivity, leaf/spine connectivity, end-host connectivity, VMM problems, ACI multi-pod/multi-site problems, and more

• Comprehensive, insider guidance for optimizing ACI in any nextgen datacenter environment
• Easy, step-by-step guidance for building your own ACI fabric
• Walks through detailed real-world ACI use cases, including failure scenarios with proven remedial actions
• Reviews management, monitoring, automation, and orchestration for software defined datacenters
• By a team of Cisco experts who’ve been helping enterprise clients succeed with ACI since its launch

Foreword by Yusuf Bhaiji     xxviii

Foreword by Ronak Desai     xxix

Introduction     xxx

PART I:  INTRODUCTION TO ACI

Chapter 1  Fundamental Functions and Components of Cisco ACI     1

ACI Building Blocks     8

    Hardware Specifications     8

ACI Key Concepts     14

    Control Plane     15

    Data Plane     17

    VXLAN     17

    Tenant     18

    VRF     19

    Application Profile     20

    Endpoint Group     21

    Contracts     22

    Bridge Domain     24

    External Routed or Bridged Network     25

Summary     26

Review Key Topics     26

Review Questions     27

Chapter 2  Introduction to the ACI Policy Model     31

Key Characteristics of the Policy Model     32

    Management Information Tree (MIT)     33

    Benefits of a Policy Model     37

Logical Constructs     37

Tenant Objects     38

VRF Objects     39

Application Profile Objects     40

Endpoint Group Objects     41

Bridge Domain and Subnet Objects     43

    Bridge Domain Options     45

Contract Objects     46

    Labels, Filters, and Aliases     48

    Contract Inheritance     49

    Contract Preferred Groups     49

    vzAny     50

Outside Network Objects     51

Physical Construct     52

    Access Policies     52

    Switch Policies     53

    Interface Policies     54

    Global Policies     55

Managed Object Relationships and Policy Resolution     57

Tags     58

Default Policies     58

How a Policy Model Helps in Diagnosis     60

Summary     63

Review Key Topics     63

Review Questions     64

Chapter 3  ACI Command-Line Interfaces     67

APIC CLIs     68

    NX-OS–Style CLI     68

    Bash CLI     74

ACI Fabric Switch CLIs     78

    iBash CLI     78

    VSH CLI     81

    VSH_LC CLI     83

Summary     84

Reference     84

Chapter 4  ACI Fabric Design Options     85

Physical Design     85

    Single- Versus Multiple-Fabric Design     87

    Multi-Pod     97

    Multi-Site     116

    Remote Leaf     131

    Hardware and Software Support     134

    ACI Multi-Pod and Remote Leaf Integration     143

Logical Design     149

    Design 1: Container-as-a-Service Using the OpenShift Platform and Calico CNI     149

Design 2: Vendor-Based ERP/SAP Hana Design with ACI     165

Design 3: vBrick Digital Media Engine Design with ACI     175

Summary     180

Review Key Topics     181

Review Questions     181

Chapter 5  End Host and Network Connectivity     185

End Host Connectivity     185

    VLAN Pool     186

    Domain     186

    Attachable Access Entity Profiles (AAEPs)     186

    Switch Policies     187

    Interface Policies     188

    Virtual Port Channel (VPC)     191

    Port Channel     197

    Access Port     201

    Best Practices in Configuring Access Policies     206

    Compute and Storage Connectivity     207

    L4/L7 Service Device Connectivity     210

Network Connectivity     213

    Connecting an External Bridge Network     213

    Connecting an External Routed Network     218

Diagnosing Connectivity Problems     242

Summary     245

Review Questions     245

Chapter 6  VMM Integration     249

Virtual Machine Manager (VMM)     249

    VMM Domain Policy Model     250

    VMM Domain Components     250

    VMM Domains     250

    VMM Domain VLAN Pool Association     252

VMware Integration     257

    Prerequisites for VMM Integration with AVS or VDS     257

    Guidelines and Limitations for VMM Integration with AVS or VDS     257

    ACI VMM Integration Workflow     258

    Publishing EPGs to a VMM Domain     258

    Connecting Virtual Machines to the Endpoint Group Port Groups on vCenter     259

    Verifying VMM Integration with the AVS or VDS     259

Microsoft SCVMM Integration     260

    Mapping ACI and SCVMM Constructs     261

    Mapping Multiple SCVMMs to an APIC     262

    Verifying That the OpFlex Certificate Is Deployed for a Connection from the SCVMM to the APIC     262

    Verifying VMM Deployment from the APIC to the SCVMM     263

OpenStack Integration     263

    Extending OpFlex to the Compute Node     264

    ACI with OpenStack Physical Architecture     264

    OpFlex Software Architecture     265

    OpenStack Logical Topology     265

    Mapping OpenStack and ACI Constructs     266

Kubernetes Integration     272

    Planning for Kubernetes Integration     272

    Prerequisites for Integrating Kubernetes with Cisco ACI     273

    Provisioning Cisco ACI to Work with Kubernetes     274

    Preparing the Kubernetes Nodes     277

    Installing Kubernetes and Cisco ACI Containers     279

    Verifying the Kubernetes Integration     280

OpenShift Integration     281

    Planning for OpenShift Integration     282

    Prerequisites for Integrating OpenShift with Cisco ACI     283

    Provisioning Cisco ACI to Work with OpenShift     284

    Preparing the OpenShift Nodes     287

    Installing OpenShift and Cisco ACI Containers     290

    Updating the OpenShift Router to Use the ACI Fabric     291

    Verifying the OpenShift Integration     291

VMM Integration with ACI at Multiple Locations     292

    Multi-Site     292

    Remote Leaf     295

Summary     298

Chapter 7  L4/L7 Service Integration     299

Service Insertion     299

The Service Graph     300

    Managed Mode Versus Un-Managed Mode     301

    L4–L7 Integration Use Cases     302

    How Contracts Work in ACI     303

    The Shadow EPG     306

    Configuring the Service Graph     307

    Service Graph Design and Deployment Options     312

Policy-Based Redirect (PBR)     322

    PBR Design Considerations     323

    PBR Design Scenarios     324

    Configuring the PBR Service Graph     325

    Service Node Health Check     326

    Common Issues in the PBR Service Graph     328

L4/L7 Service Integration in Multi-Pod and Multi-Site     332

    Multi-Pod     332

    Multi-Site     338

Review Questions     342

Chapter 8  Automation and Orchestration     343

The Difference Between Automation and Orchestration     343

    Benefits of Automation and Orchestration     344

REST API     349

Automating Tasks Using the Native REST API: JSON and XML     351

    API Inspector     351

    Object (Save As)     353

    Visore (Object Store Browser)     355

    MOQuery     357

    Automation Use Cases     364

Automating Tasks Using Ansible     372

    Ansible Support in ACI     375

    Installing Ansible and Ensuring a Secure Connection     378

    APIC Authentication in Ansible     382

    Automation Use Cases     384

Orchestration Through UCS Director     392

    Management Through Cisco UCS Director     392

    Automation and Orchestration with Cisco UCS Director     393

    Automation Use Cases     395

Summary     402

Review Questions     402

PART II:  MONITORING AND MANAGEMENT BEST PRACTICES

Chapter 9  Monitoring ACI Fabric     405

Importance of Monitoring     405

Faults and Health Scores     407

Faults     407

Health Scores     411

ACI Internal Monitoring Tools     415

    SNMP     415

    Syslog     420

    NetFlow     426

ACI External Monitoring Tools     430

    Network Insights     430

    Network Assurance Engine     437

    Tetration     453

Monitoring Through the REST API     473

    Monitoring an APIC     475

Monitoring Leafs and Spines     482

    Monitoring Applications     499

Summary     505

Review Questions     506

Chapter 10  Network Management and Monitoring Configuration     509

Out-of-Band Management     509

    Creating Static Management Addresses     510

    Creating the Management Contract     510

    Choosing the Node Management EPG     513

    Creating an External Management Entity EPG     513

    Verifying the OOB Management Configuration     515

In-Band Management     517

    Creating a Management Contract     517

    Creating Leaf Interface Access Policies for APIC INB Management     518

    Creating Access Policies for the Border Leaf(s) Connected to L3Out     520

    Creating INB Management External Routed Networks (L3Out)     522

    Creating External Management EPGs     524

    Creating an INB BD with a Subnet     527

    Configuring the Node Management EPG     529

    Creating Static Management Addresses     530

    Verifying the INB Management Configuration     530

AAA     533

    Configuring Cisco Secure ACS     533

    Configuring Cisco ISE     542

    Configuring AAA in ACI     547

    Recovering with the Local Fallback User     550

    Verifying the AAA Configuration     550

Syslog     551

    Verifying the Syslog Configuration and Functionality     555

SNMP     556

    Verifying the SNMP Configuration and Functionality     562

SPAN     566

    Access SPAN     567

    Fabric SPAN     571

    Tenant SPAN     572

    Ensuring Visibility and Troubleshooting SPAN     575

    Verifying the SPAN Configuration and Functionality     576

NetFlow     577

    NetFlow with Access Policies     580

    NetFlow with Tenant Policies     582

    Verifying the NetFlow Configuration and Functionality     585

Summary     587

PART III:  ADVANCED FORWARDING AND TROUBLESHOOTING TECHNIQUES

Chapter 11  ACI Topology     589

Physical Topology     589

APIC Initial Setup     593

Fabric Access Policies     595

    Switch Profiles, Switch Policies, and Interface Profiles     595

    Interface Policies and Policy Groups     596

    Pools, Domains, and AAEPs     597

VMM Domain Configuration     601

    VMM Topology     601

Hardware and Software Specifications     603

Logical Layout of EPGs, BDs, VRF Instances, and Contracts     605

    L3Out Logical Layout     606

Summary     608

Review Key Topics     608

References     609

Chapter 12  Bits and Bytes of ACI Forwarding     611

Limitations of Traditional Networks and the Evolution of Overlay Networks     611

High-Level VXLAN Overview     613

IS-IS, TEP Addressing, and the ACI Underlay     615

    IS-IS and TEP Addressing     615

    FTags and the MDT     618

Endpoint Learning in ACI     626

    Endpoint Learning in a Layer 2–Only Bridge Domain     627

    Endpoint Learning in a Layer 3–Enabled Bridge Domain     635

    Fabric Glean     640

    Remote Endpoint Learning     641

    Endpoint Mobility     645

    Anycast Gateway     647

    Virtual Port Channels in ACI     649

Routing in ACI     651

    Static or Dynamic Routes     651

    Learning External Routes in the ACI Fabric     656

    Transit Routing     659

Policy Enforcement     661

    Shared Services     664

    L3Out Flags     668

Quality of Service (QoS) in ACI     669

    Externally Set DSCP and CoS Markings     671

CoS Preservation in ACI     672

Multi-Pod     674

Multi-Site     680

Remote Leaf     684

Forwarding Scenarios     686

    ARP Flooding     686

    Layer 2 Known Unicast     688

    ARP Optimization     690

    Layer 2 Unknown Unicast Proxy     690

    L3 Policy Enforcement When Going to L3Out     693

    L3 Policy Enforcement for External Traffic Coming into the Fabric     695

Route Leaking/Shared Services     695

    Consumer to Provider     695

    Provider to Consumer     698

Multi-Pod Forwarding Examples     698

    ARP Flooding     700

    Layer 3 Proxy Flow     700

Multi-Site Forwarding Examples     703

    ARP Flooding     703

    Layer 3 Proxy Flow     705

Remote Leaf     707

    ARP Flooding     707

    Layer 3 Proxy Flow     710

Summary     713

Review Key Topics     713

References     714

Review Questions     714

Chapter 13  Troubleshooting Techniques     717

General Troubleshooting     717

    Faults, Events, and Audits     718

    moquery     722

    iCurl     724

    Visore     726

Infrastructure Troubleshooting     727

    APIC Cluster Troubleshooting     727

    Fabric Node Troubleshooting     734

How to Verify Physical- and Platform-Related Issues     737

    Counters     737

    CPU Packet Captures     743

    SPAN     748

Troubleshooting Endpoint Connectivity     751

    Endpoint Tracker and Log Files     752

    Enhanced Endpoint Tracker (EPT) App     756

    Rogue Endpoint Detection     758

Troubleshooting Contract-Related Issues     759

    Verifying Policy Deny Drops     764

Embedded Logic Analyzer Module (ELAM)     765

Summary     769

Review Key Topics     769

Review Questions     769

Chapter 14  The ACI Visibility & Troubleshooting Tool     771

Visibility & Troubleshooting Tool Overview     771

Faults Tab     772

Drop/Stats Tab     773

    Ingress/Egress Buffer Drop Packets     774

    Ingress Error Drop Packets Periodic     774

    Storm Control     774

    Ingress Forward Drop Packets     775

    Ingress Load Balancer Drop Packets     776

Contract Drops Tab     777

    Contracts     777

    Contract Considerations     778

Events and Audits Tab     779

Traceroute Tab     780

Atomic Counter Tab     782

Latency Tab     785

SPAN Tab     786

Network Insights Resources (NIR) Overview     787

Summary     790

Chapter 15  Troubleshooting Use Cases     791

Troubleshooting Fabric Discovery: Leaf Discovery     792

Troubleshooting APIC Controllers and Clusters: Clustering     795

Troubleshooting Management Access: Out-of-Band EPG     799

Troubleshooting Contracts: Traffic Not Traversing a Firewall as Expected     801

Troubleshooting Contracts: Contract Directionality     804

Troubleshooting End Host Connectivity: Layer 2 Traffic Flow Through ACI     807

Troubleshooting External Layer 2 Connectivity: Broken Layer 2 Traffic Flow Through ACI     812

Troubleshooting External Layer 3 Connectivity: Broken Layer 3 Traffic Flow Through ACI     814

Troubleshooting External Layer 3 Connectivity: Unexpected Layer 3 Traffic Flow Through ACI     816

Troubleshooting Leaf and Spine Connectivity: Leaf Issue     821

Troubleshooting VMM Domains: VMM Controller Offline     826

Troubleshooting VMM Domains: VM Connectivity Issue After Deploying the VMM Domain     829

Troubleshooting L4–L7: Deploying an L4–L7 Device     832

Troubleshooting L4–L7: Control Protocols Stop Working After Service Graph Deployment     834

Troubleshooting Multi-Pod: BUM Traffic Not Reaching Remote Pods     837

Troubleshooting Multi-Pod: Remote L3Out Not Reachable     839

Troubleshooting Multi-Site: Using Consistency Checker to Verify State at Each Site     841

Troubleshooting Programmability Issues: JSON Script Generates Error     844

Troubleshooting Multicast Issues: PIM Sparse Mode Any-Source Multicast (ASM)     846

Summary     860

Appendix A  Answers to Chapter Review Questions     861

Index     873

Advanced real-world Cisco Application Centric Infrastructure (ACI) monitoring and troubleshooting

This expert guide and reference will help you confidently deploy, support, monitor, and troubleshoot ACI fabrics and components. It is also designed to help you prepare for your Cisco DCACIA (300-630) exam, earning Cisco Certified Specialist–ACI Advanced Implementation certification and credit toward CCNP Data Center certification if you choose.

Authored by three leading Cisco ACI experts, it combines a solid conceptual foundation, in-depth technical knowledge, and practical techniques. It also contains proven features to help exam candidates prepare, including review questions in most chapters, and Key Topic icons highlighting concepts covered on the exam.

The authors thoroughly introduce ACI functions, components, policies, command-line interfaces, connectivity, fabric design, virtualization and service integration, automation, orchestration, and more. Next, they introduce best practices for monitoring and management, including the use of faults, health scores, tools, the REST API, in-band and out-of-band management techniques, and monitoring protocols. Proven configurations are provided, with steps for verification. Finally, they present advanced forwarding and troubleshooting techniques for maximizing ACI performance and value.

ACI Advanced Monitoring and Troubleshooting is an indispensable resource for every data center architect, engineer, developer, network or virtualization administrator, and operations team member working in ACI environments.

• Understand Cisco ACI core functions, components, and protocols
• Apply the ACI Policy-Based Object Model to develop overall application frameworks
• Use command-line interfaces to manage and monitor Cisco ACI systems
• Master proven options for ACI physical and logical fabric design
• Establish connectivity for compute, storage, and service devices, switches, and routers
• Gain visibility into virtualization layers through VMM, and integrate hypervisors from multiple vendors
• Seamlessly integrate Layer 4 to Layer 7 services such as load balancing and firewalling
• Automate and orchestrate for fast deployment with the REST API, scripting, and Ansible
• Minimize downtime and maximize ROI through more effective monitoring and configuration
• Thoroughly master concepts and techniques for advanced ACI and VXLAN forwarding
• Build deep practical expertise for quickly troubleshooting critical events
• Gain quick visibility into traffic flows and streamline problem isolation with the ACI Visibility & Troubleshooting Tool
• Walk through multiple real-world troubleshooting scenarios step-by-step
• Forewords written by Yusuf Bhaiji, Director of Certifications, Cisco Systems; and Ronak Desai, VP of Engineering for the Data Center Networking Business Unit, Cisco Systems.

This book is part of the Networking Technology Series from Cisco Press, which offers networking professionals valuable information for constructing efficient networks, understanding new technologies, and building successful careers.

Sadiq Memon, CCIE No. 47508, is a Lead Solutions Integration Architect (Automotive) with Cisco Customer Experience (CX). He has over 30 years of diversified experience in information technology with specialization and expertise in data center and enterprise networking. Sadiq joined Cisco in 2007, and as a Cisco veteran of over 13 years, he has worked with various large enterprise customers, including automotive, financials, manufacturing, and government in designing, implementing, and supporting end-to-end architectures and solutions. Sadiq was part of the Cisco Advanced Services Tiger Team during the early ACI incubation period. He has published a series of short videos covering ACI configuration on YouTube and has presented ACI/Cloud-related topics at Cisco Live! Sadiq was the technical editor for the Cisco Press book Deploying ACI and possesses multiple IT industry certifications from leading companies such as Cisco (CCIE, CCNA), VMware (VCP-DCV), Microsoft, and Citrix. Sadiq holds a bachelor’s degree in computer systems engineering from NED University of Engineering & Technology, Karachi, Pakistan.

Joseph Ristaino, CCIE No. 41799, is a Technical Leader with the ACI Escalation Team in RTP, North Carolina. He joined Cisco in 2011 after graduating from Wentworth Institute of Technology with a bachelor’s degree in computer networking. Joseph started with Cisco on the Server Virtualization TAC team, specializing in UCS and virtualization technologies. He has in-depth knowledge of compute/networking technologies and has been supporting customers for over eight years as they implement and manage data center deployments around the globe. Joseph now works closely with the ACI Technical Support teams to provide assistance on critical customer issues that go unsolved and has been working on ACI since its inception in 2014. Joseph lives with his wife in Durham, North Carolina.

Carlo Schmidt, CCIE No. 41842, is a Data Center Solutions Architect. He works with global enterprises, designing their next-generation data centers. Carlo started at Cisco in 2011, on the Data Center Switching TAC team. In that role, he focused on Nexus platforms and technologies such as FCoE, fabric path, and OTV. In 2016, he migrated to the ACI TAC team, where he specialized in customer problem resolution as well as improving product usability. In 2019 Carlo decided to take his knowledge and lessons learned from his eight years in Cisco TAC to a presales role as a Solutions Architect. Carlo is based out of Research Triangle Park, North Carolina.