README

A comprehensive, modular data import workflow engine for Laravel 12. Provides a full pipeline system for downloading, reading, filtering, mapping, transforming, and processing data from external sources — with execution tracking, scheduling, and a built-in dashboard.

Namespace: Elaitech\Import
Requires: PHP 8.4+ · Laravel 12 · elaitech/data-mapper

📖 Table of Contents

• Installation
• Architecture
• Pipeline System
• Models
• Services
• Enums
• Configuration
• Extending
• Testing
• License

🚀 Installation

As a local Composer package (recommended)

In your root composer.json, add the package as a path repository:

{
    "repositories": [
        {
            "type": "path",
            "url": "./packages/import",
            "options": { "symlink": true }
        }
    ],
    "require": {
        "elaitech/import": "@dev"
    }
}

Then install:

composer update elaitech/import

Publishing Assets

# Publish configuration
php artisan vendor:publish --tag=import-config

# Publish migrations (optional — they auto-load)
php artisan vendor:publish --tag=import-migrations

# Run migrations
php artisan migrate

🏗 Architecture

src/
├── ImportServiceProvider.php         # Package bootstrap — registers all sub-providers
├── Helpers.php                       # Global helper functions
│
├── Models/                           # Eloquent models (5)
│   ├── ImportPipeline.php            # Main pipeline definition
│   ├── ImportPipelineConfig.php      # Per-step JSON configuration
│   ├── ImportPipelineExecution.php   # Execution run records
│   ├── ImportPipelineLog.php         # Per-stage log entries
│   └── ImportPipelineTemplate.php    # Reusable pipeline templates
│
├── Contracts/                        # Top-level interfaces
│   ├── Repositories/                 # Repository contracts
│   └── Services/                     # Service contracts
│
├── Enums/                            # Backed enums (6)
│   ├── ImportPipelineStatus.php      # pending, running, completed, failed, cancelled
│   ├── ImportPipelineFrequency.php   # once, daily, weekly, monthly
│   ├── ImportPipelineStep.php        # 8 stepper steps
│   ├── PipelineStage.php            # Pipeline execution stages
│   ├── PipelineStatus.php           # active, inactive, needs_configuration
│   └── ImageDownloadMode.php        # Image download strategies
│
└── Services/
    ├── Core/                         # Shared infrastructure
    │   ├── Abstracts/                # Base abstract classes
    │   ├── Contracts/                # Shared interfaces (ServiceInterface, FactoryInterface)
    │   ├── DTOs/                     # Shared data objects (FilterConfigurationData, etc.)
    │   ├── Exceptions/               # Custom exceptions (Filter, Reader, Downloader, etc.)
    │   ├── Operators/                # Operator definitions
    │   ├── Registry/                 # Service registries
    │   ├── Cache/                    # Caching strategies
    │   ├── Configuration/            # Configuration handling
    │   └── Traits/                   # HasOptions, ServiceTrait, etc.
    │
    ├── Downloader/                   # Data source downloaders
    │   ├── Abstracts/                # AbstractDownloader base
    │   ├── Contracts/                # DownloaderInterface
    │   ├── Factories/                # DownloaderFactory
    │   └── Implementations/
    │       ├── HttpDownloader.php    # HTTP/HTTPS downloads
    │       ├── FtpDownloader.php     # FTP downloads
    │       └── SftpDownloader.php    # SFTP downloads
    │
    ├── Reader/                       # Data format readers
    │   ├── Abstracts/                # AbstractReader base
    │   ├── Contracts/                # ReaderInterface
    │   ├── Factories/                # ReaderFactory
    │   └── Implementations/
    │       ├── CsvReader.php         # CSV parsing
    │       ├── JsonReader.php        # JSON parsing
    │       ├── XmlReader.php         # XML parsing
    │       └── YamlReader.php        # YAML parsing
    │
    ├── Filter/                       # Data filtering engine
    │   ├── Abstracts/                # AbstractFilterOperator (Template Method)
    │   ├── Contracts/                # FilterInterface, OperatorRegistryInterface, etc.
    │   ├── Extractors/               # DotNotationValueExtractor
    │   ├── Registry/                 # OperatorRegistry
    │   ├── Validators/               # FilterValidator
    │   └── Implementations/          # 17 filter operators
    │
    ├── Pipeline/                     # Pipeline orchestration
    │   ├── Contracts/                # PipelineExecutionServiceInterface, etc.
    │   ├── DTOs/                     # PipelineContext, StageResult, etc.
    │   ├── Factories/                # ImportPipelineConfigFactory
    │   ├── Orchestrators/            # Pipeline orchestrator
    │   ├── Pipes/                    # 7 sequential pipes (see below)
    │   ├── Services/                 # ExecutionService, SchedulingService, TestDataService
    │   ├── ValueObjects/             # Pipeline value objects
    │   └── Implementations/          # Concrete implementations
    │
    ├── Prepare/                      # Data preparation
    │   ├── Contracts/                # PrepareInterface
    │   └── Services/                 # Preparation services
    │
    ├── ImageDownloader/              # Image download handling
    │
    ├── ImportDashboard/              # Dashboard service & repository
    │   ├── ImportDashboardService.php
    │   └── ImportPipelineRepository.php
    │
    ├── Jobs/                         # Queue jobs (2)
    │
    └── Providers/                    # Sub-service providers (5)
        ├── DownloaderServiceProvider.php
        ├── ReaderServiceProvider.php
        ├── FilterServiceProvider.php
        ├── PrepareServiceProvider.php
        └── PipelineServiceProvider.php

🔄 Pipeline System

Each import pipeline consists of 7 sequential pipes that process data in order:

┌─────────────┐    ┌──────────┐    ┌────────────┐    ┌──────────┐
│  1. Download │───▶│ 2. Read  │───▶│ 3. Filter  │───▶│  4. Map  │
└─────────────┘    └──────────┘    └────────────┘    └──────────┘
                                                          │
┌─────────────┐    ┌──────────┐    ┌────────────────┐     │
│   7. Save   │◀───│6. Prepare│◀───│5. Images Prep  │◀────┘
└─────────────┘    └──────────┘    └────────────────┘

Pipes

Pipeline Configuration Steps (Stepper)

Pipelines are configured through an 8-step stepper wizard:

📊 Models

ImportPipeline

The core model representing an import pipeline definition.

Relationships: config(), executions(), logs(), creator(), updater()
Scopes: active(), scheduled()
Accessors: status (computed: active / inactive / needs_configuration)

ImportPipelineConfig

Stores per-step JSON configuration. Each pipeline has multiple config entries (one per stepper step).

ImportPipelineExecution

Tracks execution runs with status, timing, and result data.

Helper Methods: markAsRunning(), markAsCompleted(), markAsFailed(), markAsCancelled(), addLog()

ImportPipelineLog

Per-stage log entries with level, message, and context.

ImportPipelineTemplate

Reusable pipeline configuration templates for quick setup.

⚙️ Services

Downloader

Reader

Filter (17 Operators)

Built using the Template Method pattern via AbstractFilterOperator:

Features:

• Dot-notation field access for nested data
• AND/OR logical grouping of rules
• Case-sensitive/insensitive matching
• Extensible via AbstractFilterOperator base class

Pipeline Services

Dashboard Services

📋 Enums

ImportPipelineStatus

pending · running · completed · failed · cancelled

Methods: getLabel(), getDescription(), getColor(), isActive(), isFinished(), isSuccessful(), isFailed(), isCancelled()

ImportPipelineFrequency

once · daily · weekly · monthly

Methods: getLabel(), getDescription(), getOptions()

ImportPipelineStep

basic-info · downloader-config · reader-config · filter-config · mapper-config · images-prepare-config · prepare-config · preview

Methods: title(), description(), route(), order()

PipelineStage

Represents the runtime execution stage of a pipeline.

PipelineStatus

active · inactive · needs_configuration

ImageDownloadMode

Strategies for handling image downloads during pipeline execution.

⚙️ Configuration

The package publishes config/import-pipelines.php:

php artisan vendor:publish --tag=import-config

🔧 Extending

Adding a Custom Downloader

1. Create a class extending the abstract downloader
2. Implement the required methods
3. Register in DownloaderServiceProvider

Adding a Custom Reader

1. Extend the abstract reader
2. Implement parsing logic
3. Register in ReaderServiceProvider

Adding a Custom Filter Operator

use Elaitech\Import\Services\Filter\Abstracts\AbstractFilterOperator;

final class CustomOperator extends AbstractFilterOperator
{
    public function getName(): string { return 'custom'; }
    public function getLabel(): string { return 'Custom Operator'; }
    public function getDescription(): string { return 'My custom filter logic'; }
    public function supportsValueType(mixed $value): bool { return true; }

    protected function doApply(mixed $dataValue, mixed $filterValue, array $options): bool
    {
        // Your logic here
        return $dataValue === $filterValue;
    }
}

Register it in FilterServiceProvider:

$registry->register(new CustomOperator());

Implementing Persistence (ResultSaverInterface)

SavePipe does not persist anything itself — it resolves the class named by config('import-pipelines.save.using') and calls it. Point that config at your own saver:

// config/import-pipelines.php
'save' => ['using' => \App\Import\ProductResultSaver::class],

// App\Import\ProductResultSaver
use Elaitech\Import\Services\Core\Contracts\ResultSaverInterface;
use Elaitech\Import\Services\Pipeline\DTOs\PipelinePassable;
use Elaitech\Import\Services\Pipeline\DTOs\SaveResultData;

final class ProductResultSaver implements ResultSaverInterface
{
    public function save(PipelinePassable $passable, string|int $targetId): SaveResultData
    {
        $rows = $passable->prepareResult->preparedData; // final mapped rows
        // create/update your models, then return counts:
        return new SaveResultData(createdCount: /* … */, updatedCount: /* … */, totalProcessed: count($rows));
    }
}

Category resolution and value normalization belong upstream: use a Prepare-step resolver (config('import-pipelines.prepare.using'), a ResolverInterface) and the mapper's value_mapping, respectively — keep the saver dumb.

🧪 Testing

# From the package directory
./vendor/bin/phpunit

# From the root project
php artisan test

📦 Dependencies

📄 License

MIT — see LICENSE for details.