← Back to Documentations

Signature Description Parameters 
#include <DataFrame/DataFrameMLVisitors.h>

template<typename T, typename I = unsigned long,
         std::size_t A = 0>
struct AnomalyDetectByLOFVisitor;

// -------------------------------------

template<typename T, typename I = unsigned long,
         std::size_t A = 0>
using and_lof_v = AnomalyDetectByLOFVisitor<T, I, A>;
This is a "single action visitor", meaning it is passed the whole data vector in one call and you must use the single_act_visit() interface.

Local outlier factor (LOF) is an algorithm used for Unsupervised outlier detection. It produces an anomaly score that represents data points which are outliers in the data set. It does this by measuring the local density deviation of a given data point with respect to the data points near it. Local density is determined by estimating distances between data points that are neighbors (k-nearest neighbors). So for each data point, local density can be calculated. By comparing these we can check which data points have similar densities and which have a lesser density than its neighbors. The ones with the lesser densities are considered as the outliers. Firstly, k-distances are distances between points that are calculated for each point to determine their k-nearest neighbors. The 2nd closest point is said to be the 2nd nearest neighbor to the point.
LOF is unlike other outlier detection algorithms. It doesn't work on proximity of data, but it works on density. For that reason, it may not work for all datasets (see code sample below) and it may need some experimenting. To do effective LOF application consider the following points:
  1. You must know the nature of your data
  2. You must understand the LOF algorithm well
  3. You must fine tune the parameters k and threshold

The result is a vector of indices to the original data that were deemed outliers.
using distance_func = std::function<double(const T &x, const T &y)>;

AnomalyDetectByLOFVisitor(
    size_type k,
    value_type threshold,
    normalization_type norm_type = normalization_type::none,
    distance_func &&f = [](const value_type &x, const value_type &y) -> double  {
                            return (std::fabs(x - y));
                        }));

k: K neighborhood for knn operation
threshold: The threshold for LOF score above which it is considred an anomaly
norm_type: Normalization type. the default is no normalization
f: Function to calculate distance between two datapoints
T: Column data type
I: Index type
A: Memory alignment boundary for vectors. Default is system default alignment
 
static void test_AnomalyDetectByLOFVisitor()  {

    std::cout << "\nTesting AnomalyDetectByLOFVisitor{ } ..." << std::endl;

    StrDataFrame    ibm;

    try  {
        ibm.read("IBM.csv", io_format::csv2);
    }
    catch (const DataFrameError &ex)  {
        std::cout << ex.what() << std::endl;
        ::exit(-1);
    }
    ibm.get_column<double>("IBM_Adj_Close")[502] = 800.0;
    ibm.get_column<double>("IBM_Adj_Close")[1001] = 900.0;
    ibm.get_column<double>("IBM_Adj_Close")[2002] = 450.0;
    ibm.get_column<double>("IBM_Adj_Close")[5000] = 1.5;
    ibm.get_column<double>("IBM_Adj_Close")[5030] = 20.0;

    and_lof_v<double, std::string>  anomaly { 10, 2.0 };

    ibm.single_act_visit<double>("IBM_Adj_Close", anomaly);

    const std::vector<std::size_t>  result = { 502, 1001, 2002, 5000 };

    assert((anomaly.get_result() == result));

    constexpr std::size_t   item_cnt = 1024;
    MyStdDataFrame          df2;

    df2.load_index(MyStdDataFrame::gen_sequence_index(0, item_cnt, 1));

    std::vector<double>   sine_col;

    sine_col.reserve(item_cnt);
    for (std::size_t i = 0; i < item_cnt; ++i)  {
        sine_col.push_back(std::sin(2.0 * M_PI * i / 20.0)); // Base sine wave
        if (i % 30 == 0)  sine_col.back() += 10.0;  // Inject anomalies
    }
    df2.load_column("sine col", std::move(sine_col));

    and_lof_v<double>   anomaly2 { 10, 1.5 };

    // It doesn't work for sine wave data. All the scores are 1.
    //
    df2.single_act_visit<double>("sine col", anomaly2);
    assert(anomaly2.get_result().empty());
}

C++ DataFrame