CloudCanal Data Process is an open source data processor for CloudCanal SDK, and the native official SDK only has some documentation.

This project is a custom code processor initiated by officials and jointly promoted by the community. This project brings together CloudCanal data processing plug-ins to achieve the goal of data custom transformation.

Plug-in description

• wide-table : Plug-in for wide table data processing, mainly including combined processing of fact tables and single-dimensional tables
• data-transform : General data conversion plug-ins, such as changing operations, adding additional fields, cleaning and backfilling data
• data-gather : Data aggregation plug-in, real-time aggregation of sub-database sub-table, vertical split, and remote data
• data-compare : Data comparison plug-in, business reconciliation according to changes in source data
• business-alert : Business alarm plug-in, make corresponding alarms according to data change trends

Instructions for use

• Install CloudCanal and create a data migration synchronization task
• Perform adaptive transformation of the CloudCanalProcessor implementation class (such as: WideTableProcessorV2_simple) that needs to be used
• Modify it in src/main/resources/META-INF/cloudcanal/plugin.properties under the subproject to the class that needs to be used
• under subproject mvn -Dtest -DfailIfNoTests=false -Dmaven.javadoc.skip=true -Dmaven.compile.fork=true clean package Pack
• CloudCanal console creation task (Reference case article), and uploadSubproject jar package under target (eg: wide-table-1.0.0-SNAPSHOT.jar)

CloudCanal

related resources