🚀 Qwati Syntra Veloq Ingestion Engine 💝

A high-velocity, predictive people-analytics and marketing ingestion framework. The system combines multi-variate regression mapping with synthetic data generation pipelines, managed through range-coded concurrent queues capable of scaling up to 18,000 operations per second.

📊 AI-Driven Digital Marketing & Ad Ingestion System

A modular, high-throughput digital marketing processing platform built with Python, Streamlit, and MySQL. The system leverages a multi-tier, range-coded task queue architecture designed to ingest client advertisements, dynamically filter target audience segments using intelligent data layers, and orchestrate large-scale email campaigns.

🚀 Performance Milestone

Bulk Ingestion Throughput: Achieved 18,000 operations/sec benchmark speed during stress testing across 100 enterprise corporate structures and 1,000 relational employee records per organization.

🎯 Core Product Features

1. Target Audience Ad Ingestion

Clients can submit active marketing advertisements along with specific target customer volume requirements directly through the system gateway terminal.

2. Intelligent AI Target Filtering

The central orchestration layer evaluates submitted company profiles, operational sectors, and live market trends. It dynamically identifies and extracts the highest-converting customer, employee, or corporate leads that match the client's advertising criteria.

3. Automated Ad Distribution Pipelines

Once target demographics are isolated, the system pushes execution jobs to background worker lanes. These lanes handle automated email dispatchers that deliver customized advertisements to the filtered target audience.

🏗️ Architectural Topology Map

The application platform is engineered as a decoupled, multi-tier microservice network to maintain stability under high volume loads:

+--------------------+      +--------------------+      +---------------------+

| Client UI Terminal | ---> | RS Ingestion Layer | ---> | Task Manager Engine |
+--------------------+      |    (Port 7000)     |      |    (Lanes 1 & 2)    |
         ^                  +--------------------+      +---------------------+

         |                            |                            |
         |                            v                            v
+--------------------+      +--------------------+      +---------------------+

|   Streamlit Flow   |      | Read/Write Logic   |      |      AI / Dev       |
|      Monitors      | <--- |      Layer         | <--- |       Modules       |
+--------------------+      +--------------------+      +---------------------+
                                      |
                                      v
                            +--------------------+

                            |   MySQL Physical   |
                            |      Storage       |
                            +--------------------+

Directory Asset Layout

gui/: Streamlit operational portals comprising a User Terminal, Live Multi-Station Flow Monitor, and an Administrative Maintenance Console.
rs_layer/: The network ingestion gateway. Runs a thread-safe TCP socket server (port 7000) handling high-speed task transformation, client request normalization, and dual-lane priority execution loops.
task_manager/: A distributed worker network that scales horizontally into synchronous fast lanes (Lane 1) or asynchronous background batch processing lanes (Lane 2).
modules/: Decoupled core business micro-engines running task mutations, data generation utilities, and campaign calculation steps.
read_write_layer/: The logical validation engine. Features cache-mapped relational indexing and bulk array queries to maximize disk write throughput.
storage/: The persistence manager, handling transactional database setup, indexing, and physical table initializations.
scripts/: Automation batch files providing cross-platform process orchestration wrappers.

🧬 Network Pipeline & Task Range Matrix

The infrastructure classifies and routes transactional operations securely based on strict command code ranges:

Task Code Range	Target Subsystem Path	Processing Behavior
`1 - 100`	Internal Controls	Administrative API endpoints reserved strictly for internal station syncs and task worker pops.
`101 - 200`	Admin / Developer	Routed to the slow background queue block. Treats `Priority 2` commands as top-priority items and pushes them to index zero.
`201 - 300`	Fast Ingestion Lane	Dedicated to high-speed client interface queries. Premium requests block socket connections synchronously for up to 15 seconds.
`301 - 400`	Slow Processing Lane	Reserved for long-running email campaign distributions and heavy background analytical processing operations.

🚀 Installation & Setup Lifecycle

1. Workspace Prerequisites

Ensure your infrastructure environment runs Python 3.10+ and a running MySQL Server instance.

2. Clone and Install Package Boundaries

# Clone repository
git clone https://github.com
cd advertisement-official

# Install system dependencies and register the local package in editable mode
pip install -r requirements.txt
pip install -e .

3. Establish Configurations

Copy the environment template layout to a localized environment file:

cp .env.example .env

Open your newly created .env file and insert your personal MySQL credentials (DB_USER, DB_PASSWORD).

🚀 Launching the Platform

Use the centralized automation shortcuts located in your scripts/ directory, or execute processes natively from the root folder:

Initialize Database Schemas: bash python storage/storage.py
Start the Network Queue (RS Layer): bash python rs_layer/rs_layer.py
Deploy Task Orchestrators: ```bash # Open terminal A for Fast Lane Workers (Ad Requests) python task_manager/task_manager.py 1

# Open terminal B for Slow Lane Workers (AI Filtering & Mailers) python task_manager/task_manager.py 2 4. **Open User Interfaces**:bash streamlit run gui/developer_console.py ```

🚀 Distributed Task Manager System (DTMS)

A high-concurrency, modular task orchestration system based on a "Head Chef" (TM) architecture. This project manages tasks across multiple gateways, sorting by priority and delegating to specialized worker modules.

🏗️ System Architecture

Architecture Diagram

graph TD
    %% Define Nodes
    gateway_1["Gateway 1"]
    rs_layer["RS Layer"]
    tm_orchestration["TM Orchestration"]

    %% Define Connections
    gateway_1 --> rs_layer
    rs_layer --> tm_orchestration

1. RS Layer (Receiving & Sending)

The gateway interface that manages communication between users and the core logic. * QI (Input Queue): Tasks are assigned a Task_ID and sorted by Priority. * QO (Output Queue): Holds results matched by Task_ID for delivery back to clients.

2. TM (The Head Chef) Orchestration

The central brain of the system. * Monitoring: Constantly polls the QI for pending tasks. * Task Identification: Inspects the Task_Type to determine the execution path. * Delegation: Routes tasks to specialized modules: * Simple Modules: Basic operations (Math, Queries). * AI Modules: Machine Learning and Data Analysis. * Workers: Heavy-duty tasks (Ad Distribution, Updates).

3. R&W Layer (Read & Write)

The exclusive data access point. * Provides a secure interface for CRUD (Create, Read, Update, Delete) operations. * Acts as a buffer between the Logic Layer and the Physical Storage.

4. Storage Block

The foundation of the system. * Physical Database: Manages tables and data persistence. * Maintenance: Handles database creation, indexing, and Data Isolation.

📋 Task Object Schema (Stage-1)

Field	Description	Example
Task ID	Unique identifier	`T001`
User ID	Submitter identifier	`U123`
Task Type	Category of task	`Ad_Distribution`
Payload	Execution data	`{"ad_text": "Sale!", "target": 500}`
Priority	Urgency level	`Normal`, `High`
Status	Lifecycle state	`Pending` → `Running` → `Completed`
Timestamp	Creation time	`2026-04-04 00:10 IST`
Result	Final outcome	`Ad sent to 500 people`
Error Log	Failure details	`Null` or `Error: Timeout`

🔄 Execution Pipeline

Ingestion: Client requests enter through Gateways (G1-G5).
Queuing: The RS Layer wraps data into the Task Schema and pushes it to QI.
Processing: The TM (Head Chef) pulls tasks and delegates to the correct Worker Module.
Data Ops: The R&W Layer logs progress and fetches necessary data from the Storage Block.
Completion: Results flow back through the TM to the RS Layer (QO).
Delivery: The RS Layer matches the Task_ID and pushes the final result to the user.

Note: Only the TM and specific authorized modules have access to the R&W Layer to ensure data integrity.