Doctor - Fault Management

Transcription

1 OPNFV Summit 2015 Doctor - Fault Management Gerald Kunzmann, DOCOMO Carlos Goncalves, NEC Ryota Mibu, NEC 1

2 Doctor Overview Goal Build fault management and maintenance framework Approach Identify requirement Gap Analysis Implementation work in Upstream (OpenStack) Integration and testing Status Initial Requirement study, architecture design, Gap analysis : Done Collaborative Development: On-going (3 merged Blueprints in OpenStack Liberty) Standardization Sync: On-going (by NFV member efforts, joint meeting) 2

3 Doctor Members At project creation (Dec 2014) NTT DOCOMO, Sprint NEC, Nokia, Ericsson, Huawei, ClearPath Network, Cisco Now (Oct 2015) 2x NTT DOCOMO, Sprint, AT&T, Telecom Italia, KDDI NEC, Nokia, Ericsson, Huawei, ClearPath Network, Cisco Cloudbase Solutions, Spirent, Intel, ZTE 3

4 Assumption of VNF (NFV Application) Telco Applications basically deployed in active-standby or active-active fashion App state will be switched when failure occurred App (Active) App (Standby) App and App Manager (VNFM) cannot detect HW failures directly VM Machine VM Machine 4

5 Use Case 1: Fault management Consumer C1 Consumer C2 Consumer C3 4. Switch to SBY configuration OpenStack Northbound Interface 5. Instruction (VM ID) 3. FaultNotification (VM ID, Fault ID) 2. Inform the Consumer? If YES, find owner of affected VMs from database VM-1 Hypervisor Hardware Server S1 VM-2 VM-7 VM-4 Hypervisor Hardware Server S2 Resource Pool Hypervisor Hardware Server S3 X 6. Execute Instruction - e.g. migrate VM 1. Fault ing - Hardware fault - Hypervisor fault - Host OS fault Resource Map Server VM mapping Server S1 VM-1, VM-2 Server S2 VM-7 Server S3 VM-4 Ownership information VM-1, VM-7 Consumer C1 VM-2 Consumer C2 VM-4 Consumer C3 Virtualized Infrastructure Manager (VIM), e.g. OpenStack 5

6 Use Case 2: Maintenance 4. Switch to SBY configuration Consumer C1 Consumer C2 Consumer C3 Administrator OpenStack Northbound Interface 5. Instruction (VM ID) 3. Maintenance Notification (VM ID) 1. Maintenance Request (Server S3) Virtualized Infrastructure Manager (VIM), e.g. OpenStack VM-1 Hypervisor Hardware Server S1 VM-2 VM-7 VM-4 6. Execute Instruction - e.g. migrate VM Hypervisor Hardware Server S2 Resource Pool Hypervisor Hardware Server S3 Resource Map 2. Which VMs are affected? Find Consumer owning the VM(s) from the database. Server VM mapping Server S1 VM-1, VM-2 Server S2 VM-7 Server S3 VM-4 Ownership information VM-1, VM-7 Consumer C1 VM-2 Consumer C2 VM-4 Consumer C3 6

7 Fault Management Sequence Detection Reaction Applications App App App VIM User and Administrator Virtualized Infrastructure Virtual Compute Virtual Storage Virtualization Layer Virtual Network Virtualized Infrastructure Manager (VIM) = OpenStack Hardware Resources Doctor Scope 7

8 Key Requirements as VIM Consistent Resource State Awareness Immediate Notification Extensible ing Fault Correlation 8

9 Doctor Architecture and Typical Scenario Application Manager 0. Set Alarm 6-. Action 5. Notify Error 4. Notify all Virtualized Infrastructure (Resource Pool) Resource Map 3. Update State 2. Find Affected 1. Raw Failure Notifier Inspector Alarm Conf. Failure Policy 9

10 Doctor OSS Map Application Manager 0. Set Alarm 6-. Action 5. Notify Error 4. Notify all Virtualized Infrastructure (Resource Pool) Nova Neutron Cinder e.g. Zabbix Resource Map 3. Update State 2. Find Affected 1. Raw Failure Notifier Ceilometer Inspector e.g. Monasca Alarm Conf. Failure Policy 10

11 Doctor OSS Development Application Manager 0. Set Alarm 6-. Action 5. Notify Error Virtualized Infrastructure (Resource Pool) State Correction Nova Neutron Cinder e.g. Zabbix 4. Notify all Resource Map 3. Update State 2. Find Affected 1. Raw Failure Event Alarm Notifier Ceilometer Inspector e.g. Monasca Alarm Conf. Failure Policy 11

12 Doctor Blueprints in Liberty Cycle Project Blueprint Spec Drafter Developer Status Ceilometer Event Alarm Evaluator Ryota Mibu (NEC) Ryota Mibu (NEC) Completed (Liberty) New nova API call to mark novacompute down Tomi Juvonen (Nokia) Roman Dobosz (Intel) Completed (Liberty) Nova Support forcing service down Get valid server state Tomi Juvonen (Nokia) Tomi Juvonen (Nokia) Carlos Goncalves (NEC) Completed (Liberty) Spec approved (Mitaka) Add notification for service status change Balazs Gibizer (Ericsson) Balazs Gibizer (Ericsson) Waiting for spec approval (Mitaka) 12

13 Doctor BP Detail: Nova Mark Nova-Compute Down ing External ing Service Client NEW API to update nova-compute service state Force-down API VM nova api Hypervisor nova compute queue nova conductor service state nova DB vswitch EXISTING (periodic update) nova scheduler BMC Host / Machine 13

14 Doctor BP Detail: Ceilometer - Event Alarm Nova Neutron Cinder Manager event stats sample notification Audit Service EXISTING (polling-based) Notificationdriven alarm evaluator NEW Shortcut (notification-based) 14

15 Doctor Southbound API Configuration Fault Messaging Admin Threshold Enable Conf. Policy Conf. Enable NFVI Inspector Notifier User Unified Event API 15

16 Next Step Done Neutron Cinder Zabbix DPDK Doctor Status Notifier Inspector Ceilometer Nova Monasca? To-Be Arch. Design Gap Analysis Blueprint Coding Integration OPNFV Release Dec 2014 Mar 2015 Sep 2015 Feb

17 Don t miss out... Doctor Fault Management Project Theater, Wednesday, 3:55 pm 4:15 pm Doctor: Failure Detection and Notification for NFV DOCOMO booth, PoC Demo Zone 17