There are two types of anomaly. The first is in data all the time and can be considered to be noise or expected features in the data. The second type, that’s the subject of this post, is something new that has happened that isn’t usually in the data. An example is a new type of motor vibration brought on due to a physical fault in a motor.
Traditional supervised learning has humans label data so as to direct the machine learning to narrow the model to classify specific known outcomes. Failure cases are hard to include as they don’t usually occur and all possible failure cases usually can’t be sampled as they might not even be known. Also, a failure case usually happens very rarely so usually gets filtered out as noise.
Unsupervised learning, on the other hand, looks for features that are patterns in the data. For the motor this might be different speeds, the normal vibration of the sensor and typical acceleration and deceleration patterns. These are output as features. Each feature, typically one of hundreds, highlights a pattern in the data that occurs over an arbitrary timeframe (long, short or in between). Each feature also has a numerical value indicating how strong the feature is at a particular time. Using analysis, the collection of feature values can be deemed to be typical or a atypical of past feature sets. In this way we provide for unexpected anomaly detection.
You might ask why we don’t do the same analysis on the input sensor data rather than the unsupervised features. The problem is that the sensor data represents just an instant in time and one, for example, x,y,z acceleration data sample, isn’t enough to determine it is typical. Unsupervised features encapsulate lots of x,y,z acceleration data samples that form a pattern and we can determine if a pattern, or more specifically groups of patterns, is typical or atypical over time.