Video Fire Detection Techniques and Applications in Fire Industry

Video Fire Detection Techniques and Applications in Fire Industry Ziyou Xiong, Rodrigo E. Caballero, Hongcheng Wang, Alan M. Finn, and Pei-yuan Peng United Technologies Research Center, 411 Silver Lane, East Hartford, CT, 06109 {xiongz, caballre, wangh1, finnam, pengp@utrc.utc.com} Abstract Video fire detectors use a relatively new technology compared with traditional heat sensors, smoke sensors, or gas detectors. We first review different fire detection methods and put video fire detection into the general fire detection arena. We then review various video fire detection methods, including those for both flame detection and smoke detection. We next describe our contributions to this area of research. We conclude with our thoughts for the challenges in this field and predict promising research directions. 1 Introduction The fire industry is dedicated to fire detection, fire suppression, and fire protection. Early detection of accidental fires is important to the fire industry because it minimizes loss of human lives and property. Over the past several decades, various fire detection technologies have been developed. These technologies can be broadly grouped into four categories depending on the fire signature to be detected: heat sensors, smoke detectors, flame detectors, and gas detectors. Fire detectors using multiple sensors that combine two or more of the fire signatures have also been developed to overcome the drawbacks of single sensors in fire detection. Fire detectors are also being integrated with other building systems such as alarm systems, suppression systems, and evacuation systems in integrated building solutions. Video fire detectors (VFD), use a relatively new technology compared with traditional heat sensors, smoke sensors, flame detectors, or gas detectors. But since these detectors still detect fire signatures such as smoke or flame, they can still be put into one group (flame detector, smoke detector) or a combination of the two. Closed Circuit TV (CCTV) cameras and corresponding facilities required in the video sensor system are already standard features of many buildings. It is desirable to add fire detection capability to these cameras with minimal additional cost through changes in software and correlating results between the video system and other sensors. Also VFD systems have been developed because minimum fire de-

2 tection latency is crucial to minimizing damage and saving lives. Current nonimaging fire detectors inherently suffer from the transport delay of the combustion byproducts, i.e. heat or smoke, from the fire to the sensor. The transport may be by natural convection, may be artificially enhanced as in aspirated smoke detection systems, and in some cases where there is air stratification, transport may never occur. A VFD system does not have the transport delay of a conventional detector. Video is a volume sensor, potentially directly monitoring a large area with only a single sensor. Video fire detection may be the only possible method when smoke does not propagate in a normal manner, e.g., in tunnels and mines, and other areas with forced ventilation or air stratification, e.g., in aircraft hangars and warehouses. Video is also applicable to large, open areas where there may be no heat or smoke propagation to a fixed point, e.g., in saw mills, petrochemical refineries, and forests. A block diagram of a video fire detection system is shown in Figure 1. Figure 1. Block Diagram of a Video Fire Detection System A typical VFD system consists of one or more video cameras connected to communication equipment that provides video to processing equipment. The processing equipment contains software algorithms that recognize smoke, flames, or both, and provide alerts and alarms. The communication equipment may be analog and as simple as a coax cable, or may contain hardware such as switches and distribution amplifiers. Alternatively, the communication equipment may be entirely digital and as simple as an Ethernet cable, or may contain hardware such as routers and gateways. In principle, some of the communication equipment may be wireless. In either case, analog or digital, other video or data traffic may be carried on the same communication equipment. The system in Figure 1 is only one example of a VFD system. As with any computer system there are many possible alternative implementations including having the processing equipment and algorithms physically inside the camera housing. All of this equipment, regardless of location, is powered and, as with conventional detectors, an alert is required on loss of power. It is increasingly common with digital cameras to provide the power from the Communication Equipment over the Ethernet cable (power over Ethernet, or PoE). The VFD system may be standalone, or it might share hardware with a security system. The alarms, alerts, and video images from a VFD system may be displayed locally, and

3 are often passed across another communication network, not shown, to a remote site. The issues of communication reliability apply to this network as well. 2 Review of existing VFD techniques Fire detection by VFD is not governed by a single physical principle, e.g., temperature or temperature rate, optical obscuration, etc. Instead, a number of software algorithms detect features in the video that correspond to one or more visible characteristics of fire. For example, color, flickering, and texture have been proposed for flame detection; obscuration, optical flow, and turbulence metrics have been proposed for smoke detection. The individual features over time are combined by a decision module, e.g., neural nets, fuzzy logic, decision trees, or support vector machines, to determine if a fire is present. 2.1 Video Flame Detection Techniques The key representative video flame detection methods are summarized in the following: 1. Healy et al. [1] have used color based models to separate color image pixels into flame pixels or non-flame pixels. Flame pixels are then connected to flame regions. 2. Phillips et al. [2] and Dedeoglu et al. [3] each first use pre-trained color models to detect fire-colored pixels, and then use a temporal variation of pixel intensities to determine fire pixels. (This temporal variation feature is denoted as flickering.) Their approaches differ in color model construction and how to calculate flicking. But they share a common color plus flickering feature extraction framework. 3. Liu and Ahuja [4] first use color models to segment flame color regions. This is similar to Healy et al. s approach [1]. But they then use temporal models to classify shapes of the color regions to detect flame. 4. Privalov and Shakhutdinov [5] have chosen not to use color models, instead they first look for regions with a bright static core and a dynamic boundary. They then classify these regions into flame or non-flame regions using trained models (e.g., feed-forward neural networks). 2.2 Video Smoke Detection Techniques The key representative video smoke detection methods are summarized in the following:

4 1. Fujiwara and Terada [6] proposed to use fractal encoding concepts to extract smoke regions from an image. They used the property of selfsimilarity of smoke shapes to look for features of smoke regions in the code produced by fractal encoding of an image. 2. Kopilovic et al. [7] took advantage of irregularities in motion due to nonrigidity of smoke. They computed optical flow field using two adjacent images, and then used the entropy of the distribution of the motion directions as a key feature to differentiate smoke motion from non-smoke motion. 3. Töreyin et al. [8] extracted image features such as motion, flickering, edge-blurring to segment moving, flickering, and edge-blurring regions from video. The methods to extract these features were background subtraction, temporal wavelet transformation, and spatial wavelet transformation. 4. Vicente and Guillemant [9] extracted local motions from cluster analysis of points in a multidimensional temporal embedding space in order to track local dynamic envelopes of pixels, and then used features of the velocity distribution histogram to discriminate between smoke and various natural phenomena such as clouds and wind-tossed trees that may cause such envelopes. 5. Grech-Cini [10] used more than 20 image features, such as the percentage of image change, correlation, variance etc., extracted from both reference images and current images, and then used a rule-based or a rulefirst-bayesian-next analysis method to differentiate smoke motion from non-smoke motion. 3 Our contribution to video flame detection Almost all known video flame detection approaches, such as [1] [3], combine a subset or the entire set of 3 image features: intensity, color, and flickering using a simple AND or OR operator, all at the pixel level. Although these 3 features correlate well with our perception of key flame characteristics: flame looks bright, flame usually displays red or yellow color, and flame dances over time, they often disagree, however, in terms of deciding whether a pixel is a flame pixel. For example, a white (bright) cloud pixel is not a flame pixel; a red pixel in a waving flag is not a flame pixel, but usually threshold-based algorithms mistake a white cloud pixel or a red pixel in a waving flag for a flame pixel. This implies that a simple AND or OR binary feature fusion scheme may have a high false alarm rate. With the same set of features, approaches in Liu and Ahuja [4] and Privalov and Shakhutdinov [5] rely on pixel-level segmentation algorithms to generate flame candidate regions. These regions are recognized using machine-learning techniques thereafter. The single-feature segmentation maps are usually broken into pieces because of many thresholds in the feature space. Different segmentation

maps do not usually complement each other to produce a better multiple-feature segmentation map. The classification stage may have a high false alarm rate as well. We have developed a block-based flame detection system based on a selfsimilarity property of flames. Our algorithms are at a block-level, compared with a pixel-level mentioned above. We divide each video frame into small blocks and detect fire on each chunk of video blocks. A chunk of video blocks can consist of a contiguous number (e.g., 16) of color image blocks (e.g., of size 8x8 pixels). The shape of the blocks does not need to be square or rectangular, e.g., it can be triangular or circular. The size can be even smaller. This is based on the observation that a dancing flame in even very small video blocks can be detected by human subjects, which is illustrated in Figure 2. Note that this is not true in some other object detection tasks, e.g., face detection. Figure 3 shows this difference where if the block is taken from the wrong position or wrong scale then there is no face in the block. A traditional face detection scheme is to construct a Gaussian pyramid of images and to search exhaustively for faces on each possible pixel location on each level of the image in the pyramid, which is algorithmically time-consuming. To decide whether a chuck of video has fire, we fuse features like average intensity, average hue, and average flickering over an entire block using a fuzzy-logic based scheme. 5

6 Figure 2. Left: two frames of a fire video; Right: two corresponding image blocks from the Left. A fire can also be identified from the image blocks from the Right. Figure 3. Left: An image of a human face; Right: A block taken from the Left. Unlike Fig. 2, it is more difficult to detect a face from the Right. 4 Our contribution to video smoke detection We have started a research project to develop novel techniques for video smoke detection. The key components developed in this project are background subtraction, flickering extraction, contour initialization, and contour classification using both heuristic and empirical knowledge about smoke. In the following we will present more detail on our approach. Background Subtraction

We follow the approach of Stauffer and Grimson [11], i.e., using adaptive Gaussian Mixture Model (GMM) to approximate the background modeling process. This is because in practice multiple surfaces often appear in a particular pixel and the lighting conditions change. In this process, each time the parameters are updated, the Gaussians are evaluated to hypothesize which are most likely to be part of the background process. Pixel values that do not match one of the pixel's background Gaussians are grouped using connected component analysis as moving blobs. Flickering extraction 7 A pixel at the edge of a turbulent flame could appear and disappear several times in one second of a video sequence. This kind of temporal periodicity is commonly known as flickering. Flickering frequency of turbulent flame has shown experimentally to be around 10Hz. Flickering frequency of smoke however, could be as low as 2 ~ 3 Hz for slowly-moving smoke. The temporal periodicity can be calculated using Fast Fourier Transform (FFT), Wavelet Transform, or Mean Crossing Rate (MCR). In our system, we use Mean Crossing Rate (MCR). Contour initialization Based on our observations from experiments that a smoke flickering mask is sparse, we pick those moving blobs from the background subtraction module and check whether there is a sufficient number of flickering pixels within the blobs. Boundaries of the blobs that pass this test and a minimum size test are extracted as blob contours. Smoke classification Blobs with contours are candidates of smoke regions. Features are extracted from them and passed to a smoke classification module for further check. The features that we use are based on the work by Catrakis et al. [12, 13] in characterizing turbulent phenomena. Smoke and (non-laminar flow) flames are both turbulent phenomena. The shape complexity of turbulent phenomena may be characterized by a dimensionless edge/area or surface/volume measure. One way, then, of detecting smoke is to determine the edge length and area, or the surface area and volume, of smoke in images or video. For a single image, turbulence is determined by relating the perimeter of the candidate region to the square root of the area as

8 2 P 1/ 2 1/ 2 2 * A Where P represents the perimeter of the region and A represents the area of the region. Ω 2 is normalized such that a circle would result in Ω 2 having a value of unity. As the complexity of a shape increases (i.e., the perimeter increase with respect to the area) the value associated with Ω 2 increases. In three spatial dimensions, the shape complexity is determined by relating the surface area of the identified region to the volume of the identified region as 3 S 2 / 3 1/ 3 6 * V Where S is the surface area and V is the volume. Once again, the ratio is normalized such that a sphere would result in Ω 3 having a value of unity. As the complexity of the shape increases the value associated with Ω 3 also increases. It is easy to show that an object may have any value of Ω 2 (or Ω 3 ). For instance, a unit-area rectangle of sides x and y has area xy=1 and perimeter 2(x+y). The corresponding possible values for Ω 2 are 2, rectan gle 2 / 3 1 x x 1/ 2 which can have any real value greater than 2/ π for strictly positive x. However, the rectangle, a non-turbulent shape, has the same value of Ω 2 regardless of scale. Replacing x by x/a and y by y/a yields the same value for Ω 2. A turbulent shape has different values of Ω 2 depending on the scale. For video sequences from a single camera, both the time sequence of estimates Ω 2 or an approximation to Ω 3 may be used for detection. The shape complexity defined with respect to Ω 2 and Ω 3 provides insight into the nature of a candidate region. The turbulent nature of a region can be detected (regardless of size) by relating the extracted spatial features to one another using a power law relationship. For instance, a power law relationship relating the perimeter to the area (or the equivalent for square root surface area to the cube root of volume) is defined as 2 P c( A 1/ ) q The existence of turbulent phenomena is detected by the relation of perimeter P to area A by variable q, wherein c is a constant. Based on the study of natural rain clouds, a region may be defined as turbulent when q is approximately equal to a value of 1.35.

9 Based on the above empirical knowledge of turbulent phenomena, we use a line-fitting technique to estimate the value q from the contours of the blobs in a pre-defined time interval. One example of the scatter-plot of a sequence of smoke blobs is in Fig. 4. A value close to the empirical value of 1.35 from line-fitting in the log domain suggests the existence of turbulence within the time interval. Figure 4. Scatter plot of Perimeter vs. Area of an exemplar smoke sequence Experimental results We use the dataset that is publicly available at http://signal.ee.bilkent.edu.tr/visifire/demo/smokeclips/ for experiments. This dataset has been used by Dedeoglu et al. [3] and can potentially be used to compare different algorithms. Sample images showing the detected smoke regions are presented in Fig. 5. We have made the following observations: 1. An entire smoke region might be split into multiple smaller smoke regions due to different degree of flickering (turbulence) associated with different spreading speed of smoke particles. 2. Outward boundaries of smoke are less prone to miss-detection than the source regions of smoke. This is because the periphery displays more flickering (turbulence) than the core regions.

10 Figure 5. Sample images showing the detected smoke regions Although no false alarms are issued in videos that do not have smoke, shown in Fig. 6, there are false alarms in some of the smoke video clips.

11 Figure 6. Snapshots of the video clips without smoke 4 Limit of Performance The ability of a VFD system to detect fire depends on the physical characteristics of the fire, e.g., its size, motion, color, transparency, duration; the visible environment, especially the background which includes color, texture, illumination, people or objects in motion, contrast, etc. The physical features of fire are detected in the video signal by software algorithms when the fire is in the field of view. As a result, there are clear limitations on detection. As with any fire detector, a VFD system may not be able to detect a fire that is too small. In the case of VFD, however, too small is measured with respect to the size in the video image. The size in the image will depend on the size of the fire, the distance from the camera, and the optical characteristics of the lens. A physically very small fire may still be detected provided that the camera has a suf-

12 ficiently long lens and it is pointed at the fire. Conversely, a relatively large fire might remain undetected if a very wide angle lens is used. A VFD system may not be able to detect a fire against a background similar in intensity or color. For example, it may not be able to detect a transparent blue flame, e.g., from an alcohol fire, especially against a blue background. A VFD system may not be able to detect grey smoke against a cloudy sky or black smoke against the night sky. A VFD system may not be able to detect laminar flames or flames on moving vehicles, depending on the features being used for detection. A VFD system may not be able to detect a fire when that fire is being artificially moved as a fire on a moving vehicle. The ability to detect such a moving fire will depend on the speed of movement, the software detection features, and the camera shutter speed. Video cameras often have a shutter speed of 1/60th of a second. If a fire is moved significantly while the shutter is open, then the visible image will be smeared out much the same as if the camera had been moving and the fire was stationary. The smearing of the image may remove shape or motion characteristics used for detection. A VFD system may have a false detection when naturally occurring situations mimic the features used for fire detection. For instance, certain combinations of motion and color (sunlight or moonlight reflecting on waves in water, rescue orange clothing on people who are moving, fall foliage moving in the wind) may cause false alarms. False alarms are distinct from nuisance alarms. A nuisance alarm occurs when there are legitimate flames or smoke-like clouds in the field of view. Steam from power plants, exhaust from vehicles, even fog blown by the wind may be detected as smoke depending on the software algorithms. Similarly, there are also certain circumstances of controlled or expected fire that may cause a nuisance alarm. For instance, if the VFD camera is aimed at a television that shows a fire, an alarm may be raised. The flames or reflection of flames from the flare stack at a petrochemical plant may be detected as a fire.

13 5 Challenges in Testing Video Fire Detection At its present state of maturity, Video Fire Detection (VFD) will likely have much higher probability of false alarm (Pfa) and probability of missed detection (Pmd) than conventional detectors. A testing methodology will have to consider how these characteristics will be tested and how the consumer will be educated to understand the limitations of VFD. Since the sources of false alarm depend on the algorithms used to detect fire, testing must rely on a sufficiently rich set of scenarios that contain a wide variety of false alarm sources. VFD has as its primary goal to detect fire with very low latency. To test against large and/or fully developed fires as in some existing standards will not allow testing of this characteristic. Of course, testing latency, particularly on very small fires, leads to the difficult problem of defining exactly when a fire starts. The issue of fire size is equally problematic to define. A testing methodology will need some specification of detection latency once a fire subtends, and maintains, a certain area (number of pixels on target in the camera s field of view). VFD is not a single hardware device as are conventional detectors. VFD is a complex system of sensors (visible spectrum cameras which may or may not have near-ir capability and which may have various lenses), communication, computing hardware, and, principally, software. A certification procedure that requires testing of every combination of specific camera, lens, communication medium, processor, and software would be prohibitively burdensome both for a testing agency and for manufacturers. Further, there is currently very rapid progress in these technology areas. It is not unreasonable to expect that specific components will become obsolete in less than one year. This implies the necessity of separating the source of the video signals and tests for its adequacy from the tests of the functional performance of the software algorithms. A VFD system will have to be self-checking, especially with regards to video quality. The video being analyzed must have sufficient illumination, correct color (when color is required), proper field of view and focus, and not be obscured by other objects or environmental effects. A testing methodology will certainly need to verify that once working, the system can't be changed arbitrarily (particularly the software) and doesn't degrade with time, e.g., the lens does not become obscured. A VFD system may, in principle, share hardware with other systems. It is especially desirable to share existing security cameras and communication infrastructure. If sharing is allowed, then non-interference testing will be needed to ensure that the VFD system has priority and will properly perform its functions despite

14 any other activity by coexisting systems. The sharing of Internet Protocol (IP) communication is particularly problematic in this regard. IP does not currently support prioritization and guaranteed quality of service (QoS). Finally, users may want to use VFD in either indoor or outdoor use. This presents a problem with the range of variation that might occur in the background. While there is no way to test all possible scenarios, a testing methodology will have to have some approach to knowing that the system will work even if the background is different than any that were explicitly tested. 5 References [1] G. Healey, D. Slater, T. Lin, B. Drda, and D. Goedeke, A system for realtime fire detection, Proceedings of International Conference on Computer Vision and Pattern Recognition, pages 605 606, 1993. [2] W. Phillips III, M. Shah, and N. V. Lobo, Flame recognition in video, Pattern Recognition Letters, vol. 23(1-3), pp. 319 327, 2002. [3] Dedeoglu, Y., Toreyin, B.U., Gudukbay, U., and Enis Cetin, A., "Real-Time Fire and Flame Detection in Video, IEEE 30th International Conference on Acoustics, Speech and Signal Processing (ICASSP 2005), March 18-23, 2005, Philadelphia, PA., pp. 669-672. [4] Liu, C-B., and Ahuja, N., "Vision Based Fire Detection", Pattern Recognition, 17 th International Conference on (ICPR 04), Volume 4, pp. 134-137, August 23-26, 2004, Cambridge UK. [5] G. Privalov and A. Z. Shakhutdinov, Fire suppression system and method, US Patent No. US6975225B2. [6] N. Fujiwara and K. Terada, Extraction of a smoke region using fractal coding, IEEE International Symposium on Communications and Information Technology, 2004, ISCIT 2004, Volume 2, 26-29 Oct. 2004, Page(s):659-662. [7] I. Kopilovic, B. Vagvolgyi, and T. Sziranyi, Application of panoramic annular lens for motion analysis tasks: surveillance and smoke detection, Proceedings of 15th International Conference on Pattern Recognition, 2000, Volume 4, 3-7 Sept. 2000 Page(s):714-717. [8] B. U. Toreyin, Y. Dedeoglu, and A. E. Cetin, Wavelet based real-time smoke detection in video, in EUSIPCO 05, 2005. [9] J. Vicente, and P. Guillemant, An image processing technique for automatically detecting forest fire, International Journal of Thermal Sciences Volume 41, Issue 12, December 2002, Pages 1113-1120. [10] H. J. Grech-Cini, Smoke Detection, US Patent No. US6844818B2. [11] C. Stauffer and W.E.L. Grimson: "Adaptive Background Mixture Models for Real-Time Tracking", Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1999.

[12] H. J. Catrakis, R. C. Aguirre, J. Ruiz-Plancarte, and R. D. Thayne, "Shape complexity of whole-field three-dimensional space-time fluid interfaces in turbulence", Physics of Fluids, vol. 14, iss. no. 11, p. 3891-3898. [13] Catrakis, H.J., and Dimotakis, P.E., "Shape Complexity in Turbulence," Phys. Rev. Lett., v.80, n.5, 2 Feb 1998, pp. 968-971 aeronautics.eng.uci.edu/catrakis_d.1998.pdf 15