Benchmarking Suite

📊 Benchmarking, Performance Evaluation & Data Generation Suite📌 Project OverviewThe benchmarking/ directory houses the programmatic stress-testing frameworks, mock telemetry scripts, and transaction ingestion clients for the platform. This coordinated engine generates massive relational database mockups that strictly conform to business rule invariants and streams them into active networking nodes at peak speed. This architecture allows developers to evaluate structural data layer capacities, trace queue processing speeds, and verify real-time performance thresholds under high-throughput data insertion scenarios.🏗️ Architecture & Component Mapping LayoutThe benchmarking infrastructure consists of three tightly coupled components working in absolute synchronization:textbenchmarking/ ├── data_generator.py <-- Simulation Engine (Generates domain-valid data rows) ├── push_test.py <-- Connection Tester (Verifies single-task socket bounds) └── push_to_rs.py <-- Master Seeding Pipeline (Coordinates multi-phase batch ingestion) Use code with caution.1. Ingestion Communication & Protocol InvariantsEvery automated testing utility in this package connects to the backend network hub using high-efficiency, non-blocking TCP socket streams (localhost:7000) protected by explicit execution boundaries (s.settimeout(10)).Decoupled Schema Processing: Scripts serialize transaction details as flat, optimized JSON objects. This architectural decision offloads the schema-wrapping task exclusively to the RS Layer on port 7000, allowing the client to push mass data packets sequentially without losing processing cycles to complex local memory assembly steps.High-Precision Telemetry Monitoring: Telemetry loops use high-precision monotonic CPU performance counters (time.perf_counter()) to encapsulate network pipeline injection intervals, isolating actual physical wire processing times from background CPU thread noise.🚀 Performance MilestoneBulk Ingestion Throughput: Achieved a blistering processing speed of 18,000 operations per second during bulk stress-testing runs initializing 100 enterprise corporate structures containing 1,000 relational employee records each.🧬 Subsystem Functional Reference Matrix1. Master Seeding Pipeline (push_to_rs.py)Coordinates multi-phase bulk dataset insertions to initialize or stress-test the storage infrastructure under heavy workloads.Relational Phase Sequencing: Pushes company master datasets first to establish the core corporate relational profiles. It introduces a 1-second system hold (time.sleep(1)) to ensure rows commit cleanly before streaming child entities.Array Segment Chunking Loop: Segregates massive lists (such as a 100,000-person global pool) into highly optimized packets of 250 records per batch. This protects the socket from memory degradation or buffer overflows.Administrative Overrides: Automatically forces developer ranking (priority: "2") and target operational codes (task_type: "101") on every packet, forcing workers to execute tasks at index zero.2. Relational Simulation Engine (data_generator.py)Generates logical corporate profiles and staff arrays based on precise operational requirements. It accepts size variables (n = number of companies; m = number of employees per company) to compile clean mock objects.Mathematical Age/Experience Invariant Rules: Implements strict logical guards when building personnel records to prevent broken data profiles:(\text{Maximum\ Experience}=\max (0,\text{Age}-19))This calculation ensures an employee's years of experience mathematically aligns with a minimum working age threshold of 18, preventing transactional database constraint errors in your read_write_layer.py.3. Connection Pumper (push_test.py)Provides a lightweight network utility designed to rapidly pump flat test messages over TCP sockets directly into the core queues to measure network boundaries.🛠️ Automated Seeding & Benchmarking LifecycleWhen launching the master pipeline loading sequence (push_to_rs.py), the platform executes the following operational phases:[System Init Triggered] ➔ Captures Start Time: time.perf_counter() │ ▼ [data_generator Engine] ➔ Generates 100 Companies & 100,000 Global Employee Records │ ▼ (Phase 1: Ingestion Initialization) [Push Corporate Rows] ➔ Dispatches 100 Corporate Parent Records via call_rs() │ ▼ (1-Second Strategic Processing Delay) [Transaction Commits] ➔ Ensures MySQL Tables write corporate identifiers safely │ ▼ (Phase 2: Micro-Batch Stream Routing) [Loop Array Slices] ➔ Segments Personnel Records into 250-record chunks │ ▼ (Sequential Batch Pipeline Delivery) [Batch 1 Injected] ➔ [Batch 2 Injected] ➔ [Batch 3 Injected] ... ➔ Loops to End │ ▼ [Telemetry Concluded] ➔ Captures End Time ➔ Compiles Average Execution Performance 🚀 Execution & Performance Diagnostics1. Pre-Execution SetupConfirm your benchmarking directory structure includes the required core scripts:textbenchmarking/ ├── README.md ├── data_generator.py ├── push_test.py └── push_to_rs.py Use code with caution.2. Running the High-Throughput Seeding TestExecute the master pipeline routine from your project root folder to run the seeding simulation:bashpython benchmarking/push_to_rs.py Use code with caution.Console Performance Telemetry Output:text🚀 Starting Data Generation... 📦 Pushing 100 Companies... RS Response: {"task_id": 1001, "status": "Queued"} 👥 Pushing 100000 Employees in batches... RS Response: {"task_id": 1002, "status": "Queued"} ✅ Sent batch 1 RS Response: {"task_id": 1003, "status": "Queued"} ✅ Sent batch 2 ... ⏱️ Total Pipeline Execution Time: 5.5412 seconds 📊 Processed 100 companies and 100000 employees.