README

A synchronization plugin that runs ETL jobs between two heterogeneous data sources.

Each job reads data from an origin layer, compares it with a target layer, and applies the necessary CRUD operations to keep them in sync.

flowchart TB
    subgraph ACME_ETL
        JOB_NAME_01([JOB_NAME_01])
        JOB_NAME_02([JOB_NAME_02])
    end
    subgraph DATA_SOURCE
        ORIGIN_LAYER[(ORIGIN_LAYER<p>postgresql, oracle, mysql, api http, ldap</p>)]
        TARGET_LAYER[(TARGET_LAYER<p>postgresql, oracle, mysql, api http, ldap</p>)]
    end
    ORIGIN_LAYER --> ACME_ETL --> TARGET_LAYER
    CLIENT --GET http://cname/api/v1/synch:job:name:01--> JOB_NAME_01
    CLIENT --GET http://cname/api/v1/synch:job:name:02--> JOB_NAME_02

ETL Pipeline

For each job, the plugin executes the following pipeline:

// Hook executed before anything else
$this->preProcess();

// Fetch data from the origin data source
$originDatas = $this->extractOriginEntity();

// Fetch data from the target data source
$targetDatas = $this->extractTargetEntity();

// Normalize and index origin data to make it comparable with target
$originDatas = $this->transformOriginToTarget($originDatas);

// Normalize and index target data
$targetDatas = $this->transformTarget($targetDatas);

// Diff origin vs target to generate CRUD operations
$generatedRequest = $this->generateRequestAfterCompare($originDatas, $targetDatas);

// Execute generated requests against the target layer (HTTP, SQL, or LDAP)

Plugin implementation: ACMEPluginETL.php

Known Limitations

• Only the SQL and HTTP layers can be configured independently between origin and target. The LDAP layer is shared — you cannot define a custom LDAP connection in addition to the default one.

Documentation

Monitoring

When the ACME metrics layer is enabled in cfg.ini, the plugin automatically pushes runtime metrics to the configured metrics database.

A Grafana dashboard definition is available for visualization (requires plugin version >= v1.6.5-alpha).