Best fit
- Use when large data ingestion, backfill, export, ETL, warehouse loading, manifest catch-up, or table synchronization needs to become much faster while preserving data correctness.
affaan-m/ECC
data-throughput-accelerator
Use when large data ingestion, backfill, export, ETL, warehouse loading, manifest catch-up, or table synchronization needs to become much faster while preserving data correctness.
76Collecting
See how to use itView GitHub source
npx skills add https://github.com/affaan-m/ECC --skill "skills/data-throughput-accelerator"Automated source guide
Source checked Jul 28, 2026·Refresh due Oct 26, 2026
Reorganized from the pinned upstream SKILL.md
Turn data-throughput-accelerator's source instructions into a guide you can follow
According to the pinned SKILL.md from affaan-m/ECC: Use this skill when the bottleneck is moving, transforming, or saving lots of data. The goal is not just speed. The goal is faster correct data landing in the right place with proof.
npx skills add https://github.com/affaan-m/ECC --skill "skills/data-throughput-accelerator"Bring this context
- A concrete task that matches the documented purpose of data-throughput-accelerator.
- The files, examples, or context the task depends on.
- Your constraints, target environment, and definition of done.
Expected outputs
- Use a hard accounting block:
Key source sections
Read data-throughput-accelerator through these 5 source sections
Sections are extracted automatically from the pinned SKILL.md and link back to the source.
Workflow
1. Read the current source, target, and manifest contracts. 2. Measure backlog: external files, manifest rows, raw rows, derived rows, min/max timestamps, and unprocessed counts. 3. Run a safe catch-up or sample benchmark. 4. Compare variants: batch size, worker count, warehouse…
Read the current source, target, and manifest contracts.Measure backlog: external files, manifest rows, raw rows, derived rows,Run a safe catch-up or sample benchmark.
First Distinction
Separate these before optimizing:
source extraction speed;network transfer speed;warehouse/load speed;
Fast Path Heuristics
Move compute to where the data already is.
Move compute to where the data already is.Prefer warehouse-native scans, joins, and appends for large landed files.Use manifests or checkpoints so completed files/partitions are skipped.
Accounting Output
Use a hard accounting block:
Use a hard accounting block:
Guardrails
Do not delete raw data to make a metric look better.
Do not delete raw data to make a metric look better.Do not skip failed files silently.Do not mix historical backfill status with live-tail freshness.
SkillSignal prompt templates
Provide the task, context, and acceptance criteria
These prompts were written by SkillSignal from the source structure; they are not upstream text.
Task-start prompt
Confirm source fit, inputs, and outputs before acting.
Use data-throughput-accelerator to help me with: [specific task]. Context: [files, data, or background]. Constraints: [environment, scope, and prohibited actions]. Before acting, check the pinned SKILL.md and explain which sections apply, what inputs are still missing, and what you will deliver.
Source-guided execution
Make the Agent explicitly follow the key extracted sections.
Apply the pinned data-throughput-accelerator source to [task]. Pay particular attention to these source sections: “Workflow”, “First Distinction”, “Fast Path Heuristics”, “Accounting Output”, “Guardrails”. Preserve the important decision at each step. Mark facts not covered by the source as “needs confirmation” instead of inventing them. Then verify the result against my acceptance criteria: [criteria].
Result-review prompt
Check omissions, permissions, and source drift before delivery.
Review the current data-throughput-accelerator result: (1) does it satisfy the original task; (2) were any applicable steps or limits in the pinned SKILL.md missed; (3) did it perform any unauthorized file, command, network, or data action; and (4) which conclusions remain unverified? List issues first, then fix only what the source or user authorization supports.
Output checklist
Verify each item before delivery
The task matches the purpose documented in the SKILL.md.
The source section “Workflow” has been checked.
The source section “First Distinction” has been checked.
The source section “Fast Path Heuristics” has been checked.
The source section “Accounting Output” has been checked.
Inputs, constraints, and acceptance criteria are explicit.
Unverified facts, compatibility, and outcome claims are clearly marked.
Any file, command, network, or data action has been reviewed.
Choose a different workflow
When another Skill is the better fit
design-review
Use when the user says "review the design", "check the UI", or wants a comprehensive UI/UX review. Uses a 7-phase methodology covering interaction, responsiveness, accessibility, and more.
A separate implementation from event4u-app/agent-config; compare its source, maintenance signals, and permission requirements.
Open source detaildask
Distributed computing for larger-than-RAM pandas/NumPy workflows. Use when you need to scale existing pandas/NumPy code beyond memory or across clusters. Best for parallel file processing, distributed ML, integration with existing pandas code. For out-of-core analytics on single machine use vaex; for in-memory speed use polars.
A separate implementation from K-Dense-AI/scientific-agent-skills; compare its source, maintenance signals, and permission requirements.
Open source detailmedchem
Medicinal chemistry filters for compound triage. Apply drug-likeness rules (Lipinski, Veber, CNS), structural alert catalogs (PAINS, NIBR, ChEMBL), complexity metrics, and the medchem query language for library filtering.
A separate implementation from K-Dense-AI/scientific-agent-skills; compare its source, maintenance signals, and permission requirements.
Open source detailFAQ
What does data-throughput-accelerator do?
Use this skill when the bottleneck is moving, transforming, or saving lots of data. The goal is not just speed. The goal is faster correct data landing in the right place with proof.
How do I start using data-throughput-accelerator?
The catalog detected this source-specific install command: npx skills add https://github.com/affaan-m/ECC --skill "skills/data-throughput-accelerator". Inspect the command and pinned source before running it.
Which Agent platforms does it declare?
No dedicated Agent platform is declared in the pinned source record.
Repository stars
234,327
Repository forks
35,711
Quality
76/100
Source repository last pushed
Quality breakdown
Based on traceable docs and repository signals; stars are not treated as quality.
Documentation24/30
Specificity15/25
Maintenance20/20
Trust signals17/25
Compare before choosing
Related Agent Skills and source variants
These links are selected from shared tasks, functions, stacks, platforms, and same-name variants. Compare the source owner, documentation, permissions, and maintenance signals.
design-review by event4u-app
Use when the user says "review the design", "check the UI", or wants a comprehensive UI/UX review. Uses a 7-phase methodology covering interaction, responsiveness, accessibility, and more.
dask by k-dense-ai
Distributed computing for larger-than-RAM pandas/NumPy workflows. Use when you need to scale existing pandas/NumPy code beyond memory or across clusters. Best for parallel file processing, distributed ML, integration with existing pandas code. For out-of-core analytics on single machine use vaex; for in-memory speed use polars.
medchem by k-dense-ai
Medicinal chemistry filters for compound triage. Apply drug-likeness rules (Lipinski, Veber, CNS), structural alert catalogs (PAINS, NIBR, ChEMBL), complexity metrics, and the medchem query language for library filtering.
neurokit2 by k-dense-ai
Use NeuroKit2 to build or audit reproducible research workflows for physiological time-series preprocessing, event/interval analysis, multimodal alignment, variability, and complexity. Trigger when code imports neurokit2 or needs its current APIs, schemas, and method-aware validation—not for diagnosis or device validation.
higgsfield-generate by MoizIbnYousaf
Use when the user wants to generate an image or video via Higgsfield AI. Covers 30+ models: Soul V2, Seedance 2.0, Kling 3.0, Veo 3.1, GPT Image 2, Nano Banana 2. Also covers Marketing Studio — branded ad video/image with avatars and products. Use whenever: "generate an image", "make a video", "animate this photo", "image-to-video", "img2vid", "edit this image with AI", "produce a clip", "create an ad", "make a UGC video", "marketing video", "brand video", "TV spot", "import product from URL", "
View original Skill.mdThis page is parsed directly from the repository SKILL.md without editorial rewriting. Collected: Jul 28, 2026 · about 2 min
Data Throughput Accelerator
Use this skill when the bottleneck is moving, transforming, or saving lots of data. The goal is not just speed. The goal is faster correct data landing in the right place with proof.
First Distinction
Separate these before optimizing:
- source extraction speed;
- network transfer speed;
- warehouse/load speed;
- transform speed;
- serving-table freshness;
- live tail growth while the job runs.
A pipeline can be "fast" and still appear behind if new data arrives faster than the final catch-up window.
Fast Path Heuristics
- Move compute to where the data already is.
- Prefer warehouse-native scans, joins, and appends for large landed files.
- Use manifests or checkpoints so completed files/partitions are skipped.
- Use partitioning and clustering that match the read and append pattern.
- Batch small files, requests, and writes.
- Make writes idempotent through unique keys, manifests, or replaceable staging.
- Keep raw, derived, and serving tables separately accountable.
Workflow
- Read the current source, target, and manifest contracts.
- Measure backlog: external files, manifest rows, raw rows, derived rows, min/max timestamps, and unprocessed counts.
- Run a safe catch-up or sample benchmark.
- Compare variants: batch size, worker count, warehouse SQL, file grouping, staging shape, and manifest update method.
- Promote only the fastest path that keeps counts and timestamps coherent.
- Codify the path as a CLI, scheduled job, workflow, or runbook.
- Rerun final accounting after the codified path executes.
Accounting Output
Use a hard accounting block:
Data throughput result:
- Source files discovered: 294
- Files processed this run: 294
- Raw rows added: 9,683,598
- Derived rows added: 8,917,585
- Remaining tail: 24 files at readback time
- Runtime: 38.7s
- Correctness gate: manifest counts and table max timestamps match
Guardrails
- Do not delete raw data to make a metric look better.
- Do not skip failed files silently.
- Do not mix historical backfill status with live-tail freshness.
- Do not call a pipeline complete until the target tables and manifest agree.
- For finance, healthcare, regulated, or customer-impacting data, preserve replay evidence and approval gates.
- Source repo
- affaan-m/ECC
- Skill path
- skills/data-throughput-accelerator/SKILL.md
- Commit SHA
- 4e973d3eaf92
- Repository license
- MIT
- Data collected