Skip to content

Add big batch sample dataset ingestion and benchmark tool - #48

Open
tuannx wants to merge 1 commit into
IQSS:masterfrom
tuannx:big-batch-dataset
Open

tuannx wants to merge 1 commit into
IQSS:masterfrom
tuannx:big-batch-dataset

Conversation

@tuannx

@tuannx tuannx commented Sep 21, 2026

Copy link
Copy Markdown

What this PR does / why we need it:

Adds create_sample_big_batch_dataset.py for generating and ingesting large batches of sample files (e.g. hundreds or thousands of files) for Dataverse performance testing and benchmarking.

Motivation & Context:

Currently, create_sample_custom_dataset.py allows generating N files into the "Dataverse performance test dataset", but relies on CairoSVG (which requires system C libraries like cairo-2) and generates large PNG files on disk. For evaluating large-scale dataset ingestion (such as testing batch uploads of 500, 1,000, or 10,000+ files as discussed in community reports and IQSS/dataverse#12728), generating thousands of heavy image files is impractical.

create_sample_big_batch_dataset.py provides:

  1. Lightweight file generation: Quickly creates dummy text or minimal 1x1 image payloads without heavy external C-library dependencies.
  2. Batch upload pacing: Ingests files in configurable batch sizes with inter-batch delay (replicating tools like DVUploader or bulk migration pipelines).
  3. Latency & throughput metrics: Reports real-time speed, min/max/average latency, and compares the first 10% vs last 10% of files to evaluate ingestion scaling.
  4. Local file mode: Supports --generate-only to write files to disk for consumption by create_sample_data.py.

Related issues/PRs:

- Add create_sample_big_batch_dataset.py for large-scale file generation and batch ingestion
- Support configurable batch size and delay to simulate bulk upload pacing (e.g. DVUploader)
- Provide latency and throughput metrics across batches to evaluate ingestion scaling
- Update README.md and .gitignore
@pdurbin

pdurbin commented Sep 22, 2026

Copy link
Copy Markdown
Member

@tuannx thanks for the PR! I mentioned this morning during tech hours in the context of this related PR you made:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants