Design and Analysis of Safety Critical Systems

Transcription

1 Design and Analysis of Safety Critical Systems Peter Seiler and Bin Hu Department of Aerospace Engineering & Mechanics University of Minnesota September 30, 2013

2 Uninhabited Aerial Systems (UAS) Flight Research (UMN UAV Lab) Public Safety (AeroVironment) Agricultural Monitoring Emergency Response (NASA/JPL) 2

3 Design Challenges for Low-Cost UAS Human Factors Guidance and Controls Navigation Modeling/System Identification Safety Critical Software 3

4 Design Challenges for Low-Cost UAS Systems Design and Reliability 4

5 Recent Policy Changes Increased reliability needed to integrate UAS into the national airspace 5

6 Outline Existing design techniques in commercial aviation Analytical redundancy is rarely used Certification issues Tools for Systems Design and Certification Motivation for model-based fault detection and isolation (FDI) Extended fault trees Stochastic false alarm and missed detection analysis Conclusions and future work 6

7 Outline Existing design techniques in commercial aviation Analytical redundancy is rarely used Certification issues Tools for Systems Design and Certification Motivation for model-based fault detection and isolation (FDI) Extended fault trees Stochastic false alarm and missed detection analysis Conclusions and future work 7

8 Boeing Dreamliner seats Length=56.7m, Wingspan=60.0m Range < 15200km, Speed< M0.89 First Composite Airliner Honeywell Flight Control Electronics Commercial Fly-by-Wire Boeing seats Length=63.7m, Wingspan=60.9m Range < 17370km, Speed< M0.89 Boeing s 1 st Fly-by-Wire Aircraft Ref: Y.C. Yeh, Triple-triple redundant 777 primary flight computer,

9 777 Primary Flight Control Surfaces [Yeh, 96] Advantages of fly-by-wire: Increased performance (e.g. reduced drag with smaller rudder), increased functionality (e.g. soft envelope protection), reduced weight, lower recurring costs, and possibility of sidesticks. Issues: Strict reliability requirements <10-9 catastrophic failures/hr No single point of failure 9

10 Classical Feedback Diagram Pilot Inputs Primary Flight Computer Actuators Sensors Reliable implementation of this classical feedback loop adds many layers of complexity. 10

11 Triplex Control System Architecture Actuators Sensors Actuator Control Electronics Each ACE votes on redundant actuator commands Pilot Inputs Column All data communicated on redundant data buses Each PFC votes on redundant sensor/pilot inputs Primary Flight Computer 11

12 777 Triple-Triple Architecture [Yeh, 96] Sensors x3 Databus x3 Triple-Triple Primary Flight Computers Actuator Electronics x4 12

13 777 Triple-Triple Architecture [Yeh, 96] Left PFC INTEL AMD MOTOROLA Sensors x3 Databus x3 Triple-Triple Primary Flight Computers Actuator Electronics x4 13

14 Main Design Requirements: Redundancy Management < 10-9 catastrophic failures per hour No single point of failure Must protect against random and common-mode failures Basic Design Techniques Hardware redundancy to protect against random failures Dissimilar hardware / software to protect against common-mode failures Voting: To choose between redundant sensor/actuator signals Encryption: To prevent data corruption by failed components Monitoring: Software/Hardware monitoring testing to detect latent faults Operating Modes: Degraded modes to deal with failures Equalization to handle unstable / marginally unstable control laws Model-based design and implementation for software 14

15 Main Design Requirements: Redundancy Management < 10-9 catastrophic failures per hour No single point of failure Must protect against random and common-mode failures Basic Design Techniques Hardware redundancy to protect against random failures Dissimilar hardware / software to protect against common-mode failures Voting: To choose between redundant sensor/actuator signals Encryption: To prevent data corruption by failed components Monitoring: Software/Hardware monitoring testing to detect latent faults Operating Modes: Degraded modes to deal with failures Equalization to handle unstable / marginally unstable control laws Model-based design and implementation for software 15

16 Outline Existing design techniques in commercial aviation Analytical redundancy is rarely used Certification issues Tools for Systems Design and Certification Motivation for model-based fault detection and isolation (FDI) Extended fault trees Stochastic false alarm and missed detection analysis Conclusions and future work 16

17 Analytical Redundancy Small UASs cannot support the weight associated with physical redundancy. Approach: Use model-based or datadriven techniques to detect faults. Parity-equation architecture (Wilsky) 17

18 Analytical Redundancy Small UASs cannot support the weight associated with physical redundancy. Research Objectives: Hardware, models, data (Freeman, Balas) Advanced filter design Tools for systems design, analysis and certification Approach: Use model-based or datadriven techniques to detect faults. Parity-equation architecture (Wilsky) 18

19 Analytical Redundancy Small UASs cannot support the weight associated with physical redundancy. Research Objectives: Hardware, models, data (Freeman, Balas) Advanced filter design Tools for systems design, analysis and certification Approach: Use model-based or datadriven techniques to detect faults. Parity-equation architecture (Wilsky) 19

20 Tools for Systems Design and Certification Diagram Reference: R. Isermann. Fault-Diagnosis Systems: An Introduction from Fault Detection to Fault Tolerance. Springer-Verlag,

21 Tools for Systems Design and Certification Why are new tools required? Example: Fault Tree Analysis Diagram Reference: R. Isermann. Fault-Diagnosis Systems: An Introduction from Fault Detection to Fault Tolerance. Springer-Verlag,

22 Fault Tree Analysis 22

23 Fault Tree Analysis Probability of hardware component failure can be estimated from field data. 23

24 Fault Tree Analysis Probability of hardware component failure can be estimated from field data. Model-based fault detection introduces new failure models (false alarms, missed detections, etc.) 24

25 Extended Fault Tree Analysis References 1. Aslund, Biteus, Frisk, Krysander, and Nielsen. Safety analysis of autonomous systems by extended fault tree analysis. IJACSP, Hu and Seiler, A Probabilistic Method for Certification of Analytically Redundant Systems, SysTol Conference, Incorporate failure modes due to false alarms and missed detections (per hour) (Enumerate time-correlated failures and apply total law of probability) 25

26 Example: Dual-Redundant Architecture s(k) Primary Sensor m ( k 1 ) Fault Detection Logic (FDI) d (k) mˆ ( k) Back-up Sensor m ( k 2 ) Switch Objective:Compute reliability of system assuming sensors have a mean-time between failure of 1000Hrs. 26

27 Failure Modes Missed Detection, M N Primary Fails Missed Detection 0 T 1 T 1 +N 0 N Time False Alarm, F N False Alarm T S Backup Fails System Failure 0 T 2 +N 0 N T 2 Time Proper Detection, D N Primary Fails T 1 Failure Detected System Failure 0 T 2 +N 0 N T S Backup Fails T 2 Time Failure Detected Primary Fails Backup Fails System Failure Early False Alarm, E N 0 T 1 T 2 +N 0 N T S T 2 Time 27

28 System Failure Rate Notation: Sensor failure per hour False alarm per hour Detection per failure Approximate system failure probability: 28

29 System Failure Rate Notation: Sensor failure per hour False alarm per hour Detection per failure Approximate system failure probability: Primary sensor fails + missed detection Failure detected + Backup sensor fails False alarm + Backup sensor fails 29

30 System Failure Rate Notation: Sensor failure per hour False alarm per hour Detection per failure Question: How can we compute these probabilities? Approximate system failure probability: Primary sensor fails + missed detection Failure detected + Backup sensor fails False alarm + Backup sensor fails 30

31 False Alarm Analysis What is the conditional probability of an alarm given that no fault has occurred? Abstraction: Discretetime uncertain linear system driven by noise. 31

32 Problem Formulation (Healthy) Dynamics for residual Simple Thresholding Objective: Assume n k is a stationary Gaussian process and assume known dynamic model for residuals. Compute the probability P N that r k > T for some k in {1,,N}. 32

33 Problem Formulation (Healthy) Dynamics for residual Simple Thresholding References 1. Glaz and Johnson. Probability inequalities for multivariate distributions with dependence structures. JASA, Hu and Seiler, Probability Bounds for False Alarm Analysis of Fault Detection Systems, Allerton, Theorem: There exist bounds γ k (k=1,,n) such that 1. γ k P N 2. γ k are monotonically non-increasing in k 3. γ k requires evaluation of k-dim. Gaussian integrals 33

34 Results: Effects of Correlation Neglecting correlations is accurate for small a False Alarm Probabilities and Bounds for N=360,000 but not for a near 1. For each (a,t), P 1 = which gives NP 1 =3.6 x 10-6 Residual Generation r = ar + n + k +1 k k f k d k Decision Logic 0 if rk T = 1 else 34

35 Worst-case False Alarm Probability Reference Hu and Seiler, Worst-Case False Alarm Analysis of Aerospace Fault Detection Systems, Submitted to ACC, Issue: Model depends on unknown (uncertain) parameters, ϵ. Objective: Compute the worst-case false alarm probability Main Result: Robust H 2 analysis results can be used to compute worstcase residual variance. This yields bounds on P N *. 35

36 Conclusions Commercial aircraft achieve high levels of reliability. Analytical redundancy is rarely used (Certification Issues) Model-based fault detection methods are an alternative that enables size, weight, power, and cost to be reduced. Tools for Systems Design and Certification Extended fault trees Stochastic false alarm and missed detection analysis Methods to validate analysis using flight test data (Hu and Seiler, 2014 AIAA) 36

37 Acknowledgments NASA Langley NRA NNX12AM55A: Analytical Validation Tools for Safety Critical Systems Under Loss-of-Control Conditions, Technical Monitor: Dr. Christine Belcastro Air Force Office of Scientific Research: Grant No. FA , "A Merged IQC/SOS Theory for Analysis of Nonlinear Control Systems, Technical Monitor: Dr. Fariba Fahroo. NSF Cyber-Physical Systems: Grant No , Embedded Fault Detection for Low-Cost, Safety-Critical Systems, Program Manager: Theodore Baker. 37

38 Backup Slides 38

39 Dual-Redundant Architecture s(k) Primary Sensor m ( k 1 ) Fault Detection Logic (FDI) d (k) mˆ ( k) Back-up Sensor m ( k 2 ) Switch Objective:Efficiently compute the probability P S,N that the system generates bad data for N 0 consecutive steps in an N-step window. 39

40 Assumptions 1. Knowledge of probabilistic performance a. Sensor failures: P[ T i =k ] where T i := failure time of sensor i b. FDI False Alarm: P[ T S N T 1 =N+1 ] c. FDI Missed Detection: P[ T S k+n 0 T 1 =k ] 2. Neglect intermittent failures 3. Neglect intermittent switching logic 4. Sensor failures and FDI logic decision are independent Sensors have no common failure modes. 40

41 Failure Modes Missed Detection, M N Primary Fails Missed Detection 0 T 1 T 1 +N 0 N Time False Alarm, F N False Alarm T S Backup Fails System Failure 0 T 2 +N 0 N T 2 Time Proper Detection, D N Primary Fails T 1 Failure Detected System Failure 0 T 2 +N 0 N T S Backup Fails T 2 Time Failure Detected Primary Fails Backup Fails System Failure Early False Alarm, E N 0 T 1 T 2 +N 0 N T S T 2 Time 41

42 System Failure Probability Apply basic probability theory: 42

43 System Failure Probability Apply basic probability theory: Knowledge of probabilistic performance a. Sensor failures: P[ T i =k ] where T i := failure time of sensor i 43

44 System Failure Probability Apply basic probability theory: Knowledge of probabilistic performance a. Sensor failures: P[ T i =k ] where T i := failure time of sensor i b. FDI False Alarm: P[ T S N T 1 =N+1 ] 44

45 System Failure Probability Apply basic probability theory: Knowledge of probabilistic performance a. Sensor failures: P[ T i =k ] where T i := failure time of sensor i b. FDI False Alarm: P[ T S N T 1 =N+1 ] c. FDI Missed Detection: P[ T S k+n 0 T 1 =k ] 45

46 Example Sensor Failures: Geometric distribution with parameter q Residual-based threshold logic m 1( k ) y(k) Fault Detection Filter r(k) Threshold, T d(k) Residual r ( k + 1) = n( k) + f ( k) Decision Logic 0 if d( k) = 1 f is an additive fault n is IID Gaussian noise, variance=σ r( k) T else 46

47 Example Per-frame false alarm probability can be easily computed For each k, r(k) is N(0,σ 2 ) : T [ d(k) = 1 No Fault] = 1 P = Pr p( r) dr F P F = 1 erf ( ) T 2σ 2 T Approximate per-hour false alarm probability x 10-3 P FA (30) = for σ = 0.25 P [ TS N T1 = N + 1] = 1 (1 P ) F N NP F P FA (N) 1 Per-frame detection probability P D can be similarly computed Time Window, N 47

48 System Failure Rate Notation: Sensor failure per hour False alarm per hour Approximate system failure probability: Detection per failure 48

49 System Failure Rate Notation: Sensor failure per hour False alarm per hour Approximate system failure probability: Detection per failure Primary sensor fails + missed detection Failure detected + Backup sensor fails False alarm + Backup sensor fails 49

50 System Failure Rate 10-3 P S,N 10-4 f/σ=1 f/σ=6 f/σ= T/σ Sensor mean time between failure = 1000hr and N= ( = 1 hour at 100Hz rate) 50

51 Correlated Residuals Example analysis assumed IID fault detection logic. Many fault-detection algorithms use dynamical models and filters that introduce correlations in the residuals. Question:How can we compute the FDI performance metrics when the residuals are correlated in time? FDI False Alarm: P[ T S N T 1 =N+1 ] FDI Missed Detection: P[ T S k+n 0 T 1 =k ] 51

52 False Alarm Analysis with Correlated Residuals Problem:Analyze the per-hour false alarm probability for a simple first-order fault detection system: Residual Generation (0<a<1) r = ar + n + k +1 k k f k f is an additive fault n is IID Gaussian noise, variance=1 Residuals are correlated in time due to filtering d Decision Logic k 0 if rk T = 1 else The N-step false alarm probabilityp N is the conditional probability that d k =1 for some 1 k N given the absence of a fault. P N = 1 T T L pr ( r1,..., rn ) dr1 T T Ldr There are N= samples per hour for a 100Hz system N 52

53 False Alarm Analysis Residuals satisfy the Markov property: P N can be expressed as an N-step iteration of 1- dimensional integrals: 53 = = = = T T N T T T T N N N N N N N N N dr r p r f P dr r r p r f r f dr r r p r f r f r f ) ( ) ( 1 ) ( ) ( ) ( ) ( ) ( ) ( 1 ) ( M k k k k f n ar r + + = +1 ( ) ( ) k k k k r r p r r r p 1 1 1,, + + = L ( ) ( ) ( ) ( ) ,, r p r r p r r p r r p k k k R = L L = T T N T T N R N dr dr r r p P L L 1 1 ),..., ( 1 This has the appearance of a power iteration A N x

54 False Alarm Probability Theorem:Let λ 1 be the maximum eigenvalue and ψ 1 the corresponding eigenfunctionof λψ = T 1 1( ) ψ 1 T x ( y) p( y x) dy Then P N cλ N 1 1 where c = 1, ψ 1 Proof This is a generalization of the matrix power iteration The convergence proof relies on the Krein-Rutmantheorem which is a generalization of the Perron-Frobenius theorem. For a=0.999 and N=360000, the approximation error is Ref: B. Hu and P. Seiler. False Alarm Analysis of Fault Detection Systems with Correlated Residuals, Submitted to IEEE TAC,