State machine analysis of sensor data from dynamic processes

Abstract

A state machine model analyzes sensor data from dynamic processes at a facility to identify the actual processes that were performed at the facility during a period of interest for the purpose of remote facility inspection. An inspector can further input the expected operations into the state machine model and compare the expected, or declared, processes to the actual processes to identify undeclared processes at the facility. The state machine analysis enables the generation of knowledge about the state of the facility at all levels, from location of physical objects to complex operational concepts. Therefore, the state machine method and apparatus may benefit any agency or business with sensored facilities that stores or manipulates expensive, dangerous, or controlled materials or information.

Claims

We claim:

1. A method for remote facility inspection, comprising the steps of:

a) gathering raw sensor data from a plurality of dynamic processes with a plurality of sensors during a period of interest;

b) converting the raw sensor data into a plurality of events;

c) advancing a state machine with the plurality of events from an initial state through a plurality of object state transitions to a final state, Wwerein the object state transitions comprise a plurality of transitioning steps whereby a destination at the intersection of each of a plurality of object states with each of the plurality of events within the state machine instructs an object to transition from a first state to a second state when an event happens and the object is in the object state defined by the destination;

d) grouping the plurality of object state transitions into a plurality of actual processes performed at the facility during the period of interest;

e) declaring a plurality of declared processes that an inspector expects to happen at the facility during the period of interest; and

f) comparing the plurality of actual processes to the plurality of declared processes, thereby identifying undeclared processes performed at the facility during the period of interest.

2. The method of claim 1, wherein the converting step further comprises identifying a significant transition point in the raw sensor data as an event indicating a change in a process state of the facility.

3. An apparatus for remote facility inspection, comprising:

a) a processor;

b) storage accessible from the processor;

c) an input system accessible from the processor;

d) an output system accessible from the processor;

e) a plurality of sensors for gathering raw sensor data from a plurality of dynamic processes;

f) means for using facility characteristics from the input system and the processor to define a plurality of event rules in the storage;

g) means for using facility characteristics from the input system and the processor to construct a state machine in the storage;

h) means for converting the raw sensor data into a plurality of events according to the event rules;

i) means for advancing the state machine with the plurality of events from an initial state through a plurality of object state transitions to a final state, the advancing means comprising a matrix further comprising a plurality of object states and the plurality of events and having a destination at the intersection of each of the plurality of object states with each of the plurality of events, wherein the destination instructs an object to transition from a first state to a second state when the an event happens and the object is in the object state defined by the destination;

j) means for grouping the object state transitions into a plurality of actual processes performed at the facility during the period of interest;

k) means for using the processor and the output system to communicate the actual processes;

l) means for using expected operations from the input system and the processor to construct a plurality of declared processes in the storage; and

m) means for comparing the plurality of actual processes to the plurality of declared processes to identify undeclared processes performed at the facility during the period of interest.

4. The apparatus of claim 3, wherein the destination is modeled with a destination description language.

5. The apparatus of claim 3, wherein the event rules identify significant transition points in the raw sensor data as a change in a process state of the facility.

Description

BACKGROUND OF THE INVENTION

The present invention is directed to the analysis of raw sensor data from dynamic processes at a facility for the purpose of remote facility monitoring and inspection, and more particularly to a method and apparatus for identifying the actual processes that were performed at the facility during a period of interest using a state machine model and comparing the actual processes identified to the expected processes declared by an inspector.

The use of sensor systems for monitoring and tracking the status of high value assets and processes has proven to be less costly and less intrusive than the on-site human inspections that they are intended to replace. Such systems can help to minimize the need for costly material inventories and human exposure to hazardous materials. In general, such a remote monitoring system may be of benefit to any agency or business with sensored facilities that stores or manipulates expensive, dangerous, or controlled materials or information of any kind.

Typical government applications for such sensor systems include nuclear material handling sites with sensored operations (for example, weapons facilities that fabricate, transport, or store nuclear material), facilities secured with detection sensors where controlled, expensive, or classified materials must be accessed and stored, and facilities that must safely handle expensive, dangerous, or controlled materials of any kind (conventional weapons, high explosives, experimental reactors, etc.)

Industrial applications for such sensor systems include facilities with sensored operations (for example, nuclear power plants, pharmaceutical companies, and chemical manufacturers), sites that must monitor access to expensive or one-of-a-kind objects (satellite components, fine-line lithography equipment, etc.), and plants that must electronically monitor complicated, human-error prone operations for safety or efficiency (power plants, computer chip producers, etc.)

In the realm of international inspections, such a sensor system can save inspectors from numerous trips to foreign facilities. In addition, facility inspectors can automate the current data inspection/verification process, allowing them to concentrate on process abnormalities, and saving them from unnecessary attention to normal processing.

Such monitoring systems, however, present a classic information overload problem to an inspector trying to analyze the resulting sensor data. These data are typically so voluminous and contain information at such a low level that the significance of any single reading (e.g., a door open event) is not obvious. Sophisticated, automated techniques are needed to identify and extract expected processes in the data and isolate and characterize the remaining patterns that may be due to undeclared or abnormal activities. A key issue from an operational perspective is that it is not feasible to expect a human inspector to manually perform all of the required analysis reliably.

The data gathered by monitoring systems come from a wide variety of sensors including discrete state sensors (e.g., breakbeams), analog sensors that measure continuous physical quantities (e.g., tank levels or temperatures), and sensors that measure spectra (e.g., chemical photo analyzers and gamma radiation spectra). Analysis of this data requires extracting, correlating, and classifying patterns in the sensor data and interpreting them in terms of the allowed activities at the monitored facility. In most situations, it is not obvious how to combine the discrete, analog, and spectrum sensor data in ways to draw useful conclusions about the dynamic processes being monitored.

There are many factors that make sensor data analysis both difficult and labor intensive:

the processes being monitored can have a tremendous degree of variability (e.g., activities in the process may not always be performed in the same order);

many of the sensors provide only minimal information, indicating activity but not conveying sufficient information to reliably classify the nature of the activity;

interpretation of the raw sensor data is facility- and process-specific, requiring a high degree of human training;

the data may be incomplete (e.g., two objects may pass through a breakbeam with only one resultant trigger);

there may be "noise" from background activities, either expected or unexpected (e.g., two activities occurring simultaneously);

the interpretation of the raw sensor data may depend on the current or past states of multiple sensors or subtle timing differences between events coming from multiple sensors;

sensors tend to drift away from their calibration points; and

measurements of continuous physical quantities have inherent uncertainty.

All of these factors must be accounted for in assessing how well the events from many sensors correlate with expected, normal behavior.

Traditional systems typically monitor static situations. For a storage facility, for example, a simple monitoring system that checks sensor values against static set points is quite adequate. However, for facilities with dynamic processes, it is no longer sufficient simply to check periodically the readings of each sensor against a fixed threshold value. In particular, analysis of data from dynamic processes differs from static facilities in important ways:

interpretation of the data from a sensor that is within threshold often depends on knowing the current status (state) of other objects or processes in the facility;

verification of correct operation of a facility often requires knowledge about correct sequencing of processes and sub-processes;

correct identification of what processes have occurred can require knowledge of the relative timing between sensor events; and

correct assessment of the current status of a facility can require knowledge of the status of the facility at any time in the past.

There remains a need for facilities with dynamic processes for an inspector to be able to detect situations where knowledge about the combined states of multiple sensors is required to make judgements about possible diversion, safety, security, sensor system integrity, sensor data quality, or other, more abstract concepts.

SUMMARY OF THE INVENTION

The present invention, hereinafter referred to as Knowledge Generation (KG), solves the problem of information overload at facilities having multiple dynamic processes with an automated data analysis engine that runs a state machine model of the processes and sensors at the facility.

The invention comprises a method that analyzes the raw sensor data, advancing the state machine through a series of object state transitions, thereby converting and combining the outputs from many sensors into operator domain level information, or actual processes, that occurred at the facility during the period of interest. The method further compares the actual processes identified against a set of expected processes declared by an inspector (these would normally be the legal or allowed operations), and then presents the differences between processes that actually occurred and those that were declared by the inspector in the process level domain.

The present invention also comprises an apparatus for performing the KG analysis. A processor connects with an input/output system and storage. A sensor array monitors the dynamic processes at the facility. The apparatus also comprises means for inputting the raw sensor data and the facility characteristics into the processor to define event rules in the storage, converting the raw sensor data into events with the event rules, advancing the state machine through a series of object state transitions using with the events, and grouping the transitions into the actual processes. The apparatus further comprises means for inputting expected operations into the processor to construct a process declarations file and comparing the declared processes to the actual processes to identify undeclared processes that occurred at the facility during the period of interest. This information is output to the inspector through a user interface.

The difference between KG and existing analysis systems is the use of finite state machines to model not just the facility sensor system, but also user objects and processes. KG is the only known automated method to monitor dynamic processes, extract the actual processes that occurred from the raw sensor data, compare the actual processes against a set of processes declared to represent truth by the inspector, and explicitly detect undeclared or covert processes.

KG can perform the analysis of static facilities, but it can also dynamically track and check the states of dynamic processes where material is transformed into different states or forms or transported to different locations. KG can track processes at a much higher level than traditional systems since it assembles actual macro-processes out of the raw sensor data. The finite state machines can also handle multiple simultaneous processes, tracking not only the sensor states but also the states of the process objects and higher level user concepts (safety, security, probability of diversion, etc., sometimes called "goals") and dynamically and continuously track states and values for these concepts during facility operations. Since KG has knowledge of the state of the system at all levels of the user's system, it can more intelligently report to the inspector what caused any deviation from normal and the impact of the abnormality on the facility goals.

Although the KG system analysis engine was written with international nuclear material safeguards, nonproliferation, and transparency in mind, there is no information about any particular facility in the analysis engine software. Examples of where KG can be applied include anywhere it is important to:

check for sensor threshold errors;

check for inconsistent readings between redundant sensors;

check for timing between sensors events;

track the locations of objects affected by the processes at the facility;

track the states of objects at the facility;

track the health of the monitoring system itself;

track the integrity of the sensor data;

verify that only the declared processes occurred;

identify processes which occurred which were unexpected;

identify processes which were started but not completed;

identify processes which were performed out of order;

detect intrusion into or exit from a secured area;

verify that safe or proper operating procedures have been followed; or

evaluate other concepts such as integrity, reliability, appropriate use, etc.

Thus, a primary advantage of KG is the ability to generate knowledge about the state of the facility at all levels. Because the user processes are modeled in state machines, information from multiple sensors within the sensor system can be combined to form knowledge about the facility. That is, data points are combined into events; events are combined into activities; activities are combined into processes. These activities might further be taken into a higher-level state machine to monitor the safety of operations of the facility. The ability to raise data points to higher levels of knowledge is only limited by the creativity of the site expert who is designing the state machines and the quality of the sensor data available.

Another advantage of KG is the ability to analyze dynamic processes. Traditional data analysis of sensored systems has concentrated on static systems where the report from any sensor is subjected to a threshold check, and an alarm is raised if the threshold is exceeded. Modeling a facility as a state machine enables the extraction of information from sensor data streams that depend not only on the state of one sensor, but also on the sensor's past states, and the current and past states of any or all other sensors in the system. KG dynamically tracks the state of the system and can compare readings from multiple sensors, check timing between events, make comparisons against historical data, etc. Since KG combines the data from many sensors and tracks the state of the entire system, it can make intelligent decisions about the state of the entire system.

Another advantage of KG is the ability to handle multiple simultaneous processes. While human inspectors typically can examine a data set for one well-defined characteristic, KG can accurately and reliably track many performance/safety/security measures. This ability to monitor multiple dimensions of the process can vastly increase the confidence of facility inspectors that process steps are being monitored and performed correctly. In addition to monitoring multiple metrics simultaneously, KG can track complex processes which may require critical times, critical values, and critical sequences, any or all of which may be revealed only by simultaneously comparing the data streams from multiple sensor.

A further advantage of KG is the ability to extract useful information from an imperfectly sensored system. Imperfectly sensored means that sensors may not function perfectly or monitor the best feature of the system to capture the best information. This advantage is accomplished by providing alternate paths through the facility state machine. This provision has proved to be crucial, for example, when the sensor system in a facility was designed to optimize detection of security violations, but the inspector later needs to make conclusions about the safe usage of equipment.

Another advantage of KG is the ability to dispatch system functions or equipment to gain more information about a detected abnormality. As an example, KG has been used to issue a command to a robot to travel to a remote area to test a sensor that appeared to be failing. It could similarly be used to dispatch a robot to investigate evidence of diversionary activities (take pictures, detect motion, etc.).

Furthermore, KG has the ability to reconstruct the history of a process. Since state machines are deterministic, it is possible to reconstruct the path that a process has taken to arrive at its current state. This ability to track back through the history of a process makes the task of determining where an abnormal process started to fail much easier.

A further advantage of KG is the ability to quickly produce repeatable results so that the description of the facility processes can be closely "tuned" to accept normal processes and reject abnormal processes that barely fall outside normal operations. The difference between a normal process and one which has been minutely perturbed (via out of order steps, covert activities, small timing differences, etc.) will often appear visually identical to a human inspector. The increased discrimination ability enabled by KG means that human inspectors are required only for the analysis of a small number of processes--those that are determined by the analysis engine to fall outside the normal operational limits. Thus, human inspectors can examine much more closely the few abnormal operations detected by KG.

A further advantage of KG is the ability to perform constrained process level comparisons. The inspector declarations about what is expected to happen might be constrained by a number of different conditions that occur in real processes. For example, a process may be declared to have explicit time constraints. Declared processes may also be constrained to occur either before or after parts of other declared processes (relative timing). Declared processes may also be constrained by number (optional processes).

Finally, KG can extract what processes actually occurred without a set of declared processes from the inspector. In this application, the comparison between actual and declared is omitted, and the inspector is presented with a list of what processes actually happened. Since the sensor data from a remote power plant is typically examined by hand, the ability to extract what processes actually occurred from raw data is a vast improvement in speed, repeatability, fineness of the analysis, and accuracy, compared to manual analysis.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and form part of the specification, illustrate the present invention and, together with the description, explain the invention. In the drawings, like elements are referred to by like numbers.

FIG. 1 is a block diagram of the Knowledge Generation analysis system.

FIG. 2 is a schematic diagram of sensored facility upon which a KG analysis of a barrel retrieval and transportation simulation was performed.

FIG. 3 shows the relationship of the high-level state machines in the simulation.

DETAILED DESCRIPTION OF THE INVENTION

1. Definitions

It is understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.

The term "activity" refers to a significant action, as defined by a site expert. It represents the successful completion of one or more events. Activities provide a convenient way for the state machine designer to group events, generally into human inspector discernable blocks. Intuitively, an activity should be visible to an inspector at the facility; that is, an inspector should be able to tell by inspection whether or not an activity has occurred. The state machine will track the starts and ends of activities in the same way that it tracks the starts and ends of processes. An example of this structure, depicting the activities and processes required to drive a car to work, is shown schematically in Table 1.

    TABLE 1
    Drive Car to Work            // The root process
    Start Car                    // A process which can be declared
        Put foot on brake        // An activity
        Put key in ignition      // Another activity
        Turn key to on           // Another activity
    Drive Car                    // Another declarable process
        As required, put in reverse // May need to back out of the
                                 driveway
        Accelerate               // Assume we have an automatic shift
        As required, stop        // Need to allow some observance of
                                 traffic signals
    Stop Car                     // Another declarable process
        Shift to Park            // An activity
        Turn key to off          // An activity

    TABLE 2
    State.backslash.Event GrabEvent          ReleaseEvent
    InTray             MovingToTurntable  Error=BundleDropped
    MovingToTurntable  Error=DoubleGrab   OnTurntable
    OnTurntable        MovingToFlask      Etc.

    TABLE 3
    SENSORS
    Door1.vertline.Door1 Balanced Magnetic Switch 1 (BMS1) Data.vertline.Center
     door
    along East
     wall.vertline.DISCRETE.vertline.0.vertline.Closed.vertline.4.vertline.Open
    .vertline.
    Door1-SOH.vertline.Door1 Balanced Magnetic Switch 1 (BMS1)
     State-of-health.vertline.
    Center door along East
     wall.vertline.DISCRETE.vertline.0.vertline.Report.vertline.

                TABLE 4
                DESTINATION PeopleMotion Door1Open
                CHANGE PeopleMotionCounter +1
                // PRINT PeopleMotionCounter

    TABLE 5
    // DESTINATIONS for object BBObject (BreakBeam Object)
    // Process CenterPillarMotion - moves BBObject from staging area
    // to center.
    // Sends CenterPillarEvent
    //
    // This model is a bit complicated because we do not want south pillar
    // events to move the PillarObject when it is in the staging area.
    // We do this by not creating the BBObject until we are ready
    // to use it, and we destroy it as soon as we detect the motion
    // we expected. This is a desentization in the model but it is
    // necessary because of the huge amount of noise
    // (caused by human motion) in the data.
    DESTINATION QuiescentStaging Breakbeam2Broken
    // The flags in the following MESSAGE command instruction the state
    // machine engine to flag the "start" of the process
    CenterPillarMotion.
    MESSAGE *SN *FF *TP *PCenterPillarMotion *AS Motion between
    center pillars start.
    // The SEND command signals a higher level process that something has
    happened.
    SEND CenterPillarEvent
    // Start a 60 second timer to make sure this is really the AGV.
    START WaitForBB3Timer 60 BB3NotHitBeforeTimeout
    NEXTSTATE QuiescentCenterOut
    // We got BB2, but BB3 did not trigger fast enough to make us
    // believe we really have the AGV - must be human motion.
    DESTINATION QuiescentCenterOut BB3NotHitBeforeTimeout
    MESSAGE *SN *FF *TP *PCenterPillarMotion *AI Center pillar motion
    not confirmed by south pillar motion.
    // We got faked out by noise. Send the BBObject (ostensibly the Agv)
    // back to the staging area.
    NEXTSTATE QuiescentStaging

    TABLE 6
    07-Sep-99 1:30:28 PM, 1, Motion4   // Motion 4 sensor reports
    "Motion"
    07-Sep-99 1:30:30 PM, 0, Breakbeam2-SOH
    07-Sep-99 1:30:38 PM, 1, T1
    07-Sep-99 1:30:40 PM, 0, T1
    07-Sep-99 1:30:50 PM, 4, Breakbeam1
    07-Sep-99 1:30:51 PM, 0, Breakbeam1
    07-Sep-99 1:30:53 PM, 4, Door2
    07-Sep-99 1:30:58 PM, 0, Door2
    07-Sep-99 1:31:12 PM, 0, Motion4   // Motion 4 sensor reports
    "No Motion"

    TABLE 7
    SourceToDestination
        SourcePalletRetrieve
              AgvToRacks
                    CenterPillarMotion
                    SouthPillarMotion
              PalletInMotion
              AgvReturnsToInitial
                    SouthPillarMotion
                    CenterPillarMotion
        TransportToDestination
    //  ItelStorePallet
    //        PalletInMotion
    //        AgvToRacks
    //              CenterPillarMotion
    //              SouthPillarMotion
    //        AgvReturnsToInitial
    //              SouthPillarMotion
    //              CenterPillarMotion
        DestinationPalletStore
              PalletInMotion
              CenterPillarMotion // The AGV goes to the racks.
              CenterPillarMotion // The AGV returns to its staging
    location.

• Inventors
  Cook, William R.; Brabson, John M.; Deland, Sharon M.;
• Assignee
  Sandia Corporation (Albuquerque, NM)
• Published
  Dec-23-2003
• Current US Classes:
  700/27
  700/65
  702/182
  702/188
  703/17
  703/22
  705/11
  705/7
  717/104
• Application #
  844028
• International Classes
  G05B 019/18
• Field of Search
  700/27 700/65 700/9 700/86-88 700/103 703/17 703/22 707/103 717/104 702/182 702/188 705/7 705/11
• Examiner
  Khatri; Anil
• Agent
  Bieg; Kevin W.
• US Patent References:
  4141006
  5394347
  5485600
  5586066
  5761090
  6408262
  6434512
  6477439