Compass is an open source project based on OPPO’s internal big data diagnostic platformwhich can be used to diagnose big data tasks running on scheduling platforms such as DolphinScheduler and Airflow.

Compass Core Functions

Compass currently supports the following functions and features:

• Non-invasive, instant diagnosis, no need to modify the existing scheduling platform, you can experience the diagnosis effect.

• Supports a variety of mainstream scheduling platforms, such as DolphinScheduler, Airflow or self-developed, etc.

• Support multi-version Spark, Hadoop 2.x and 3.x task log diagnosis and analysis.

• Support abnormal diagnosis of workflow layer to identify various failures and baseline time-consuming abnormal problems.

• Support engine layer exception diagnosis, including 14 types of exceptions such as data skew, large table scan, memory waste, etc.

• It supports the writing of various log matching rules and the adjustment of abnormal thresholds, and can be optimized according to actual scenarios.

Overview of diagnostic types supported by Compass:

Compass Technical Architecture

Compass is mainly composed of task metadata module for synchronous workflow layer, synchronous Yarn/Spark App metadata module, associated workflow layer/engine layer App metadata module, workflow task anomaly detection module, engine layer anomaly detection module, and Portal display module .

Overall Architecture Diagram

The overall architecture is divided into 3 layers:

• The first layer is to interface with external systems, including scheduler, Yarn, HistoryServer, HDFS and other systems, to synchronize metadata, cluster status, operating environment status, logs, etc. until diagnostic system analysis;

• The second layer is the architecture layer, including data collection, metadata association & model standardization, anomaly detection, and diagnostic Portal module;

• The third layer is the basic component layer, including MySQL, Elasticsearch, Kafka, Redis and other components.

Specific module process stages:

(1)data collection stage: Synchronize workflow metadata such as users, DAGs, jobs, and execution records from the scheduling system to the diagnostic system; regularly synchronize metadata from Yarn ResourceManager and Spark HistoryServer App to the diagnostic system, mark the storage path of job running indicators, and make preparations for subsequent data processing stages Base;

(2)Data Association & Model Standardization Phase: Associate the workflow execution records collected step by step, Spark App, Yarn App, cluster operating environment configuration and other data through the ApplicationID medium. At this time, the metadata of the workflow layer and the engine layer have been associated, and the data standard model is obtained (user, dag, task, application, clusterConfig, time);

(3)Workflow layer & engine layer anomaly detection stage: So far, the data standard model has been obtained, and the Workflow anomaly detection process is further aimed at the standard model. At the same time, the platform maintains a set of data governance knowledge base that has been accumulated for many years, and the knowledge base is loaded into the standard model. Through heuristic rules, the index data of the standard model, Logs are simultaneously excavated for exceptions, combined with the status of the cluster and the state of the running environment, and analyzed to obtain abnormal results at the workflow layer and engine layer;

(4)business view: Store and analyze data, provide user task overview, workflow layer task diagnosis, engine layer job Application diagnosis, workflow layer displays exceptions caused by scheduler execution tasks, such as task failure, loopback task, baseline deviation task, etc., calculation The engine layer shows the time consumption, resource usage, and runtime issues caused by Spark job execution.