Restructure problem directories into problems/<id>/v1/ + resources/ by dweindl · Pull Request #326 · Benchmarking-Initiative/Benchmark-Models-PEtab

dweindl · 2026-09-25T13:33:07Z

🤖 Generated with Claude Code

Restructures each problem's directory to prepare for PEtab v2 support (each problem can later get a sibling v2/) and to separate benchmarking files from background/provenance material.

Benchmark-Models/<ProblemID>/ becomes:

problems/<ProblemID>/
  v1/
    problem.yaml
    model.xml
    conditions.tsv
    measurements.tsv
    observables.tsv
    parameters.tsv
    visualizations.tsv   # optional
    simulations.tsv      # optional
  resources/              # optional
    ...

Filenames are standardized to the short form Bertozzi_PNAS2020 already used. The canonical file set for each problem was determined by parsing its own problem.yaml (rather than hand-coded per problem), so this is mechanical and driven by what's actually referenced.

Everything not referenced by a problem's YAML moved into resources/, e.g. General_info.xlsx, petab_select/, README files, and things not explicitly called out in the issue but following the same rule: per-figure visualization tables (Lucarelli, Raimundez, Sneyd), and alternate/variant sub-problems (Fiedler's reformulated/, Fujita's test_data/, Liu's individual-based/, Oliveira's Bahia/, Smith's sim_test/, Zhao's held-out test split + orphaned second YAML, Blasi's unused _allCombinations model variant).

Also updated for the new paths/filenames: base.py, C.py, overview.py, MANIFEST.in, build.sh, .gitignore, the simulate.yml CI job, CONTRIBUTING.md, and the PR template. The README overview table was regenerated with bmp-create-overview --update.

Closes #324

Move Benchmark-Models/<ProblemID>/ to problems/<ProblemID>/v1/, standardizing filenames to the short form (problem.yaml, model.xml, conditions.tsv, measurements.tsv, observables.tsv, parameters.tsv, and the optional simulations.tsv/visualizations.tsv). This separates the PEtab files from provenance/background material, which now lives in an optional problems/<ProblemID>/resources/, and prepares each problem directory for a future sibling v2/ format. Non-PEtab and non-canonical material (anything not referenced by a problem's YAML) moves into resources/, e.g. General_info.xlsx, petab_select/, per-figure visualization tables, and other variant/ alternate problem definitions. Update base.py, C.py, overview.py, MANIFEST.in, build.sh, the simulate.yml CI job, .gitignore, CONTRIBUTING.md, and the PR template for the new paths and filenames, and regenerate the README overview table. Closes Benchmarking-Initiative#324 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The rename script rewrote each v1/problem.yaml on disk to reference the new short filenames, but never re-staged that rewrite, so the previous commit captured the old, pre-rename filenames inside these YAML files. Fix the content to match the actual files on disk. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

dweindl · 2026-09-25T14:06:55Z

Everything not referenced by a problem's YAML moved into resources/, e.g. General_info.xlsx, petab_select/, README files, and things not explicitly called out in the issue but following the same rule: per-figure visualization tables (Lucarelli, Raimundez, Sneyd), and alternate/variant sub-problems (Fiedler's reformulated/, Fujita's test_data/, Liu's individual-based/, Oliveira's Bahia/, Smith's sim_test/, Zhao's held-out test split + orphaned second YAML, Blasi's unused _allCombinations model variant).

Not sure about what the policy is or should be for adding all those additional things, but I am in favor of having only the petab-yaml-referenced files in the problems//v1/ directory (plus potentially some additional structured information file later on). For all those other files, it's not immediately clear where they should be used. For now, I would just move them to resources/. If we also want to have a model selection benchmark collection, we should probably decide now if that's supposed to be included in this repo (and if so how we would separate the basic parameter estimation problems from model selection problems).

dilpath · 2026-09-25T14:29:56Z

Everything not referenced by a problem's YAML moved into resources/, e.g. General_info.xlsx, petab_select/, README files, and things not explicitly called out in the issue but following the same rule: per-figure visualization tables (Lucarelli, Raimundez, Sneyd), and alternate/variant sub-problems (Fiedler's reformulated/, Fujita's test_data/, Liu's individual-based/, Oliveira's Bahia/, Smith's sim_test/, Zhao's held-out test split + orphaned second YAML, Blasi's unused _allCombinations model variant).

Not sure about what the policy is or should be for adding all those additional things, but I am in favor of having only the petab-yaml-referenced files in the problems//v1/ directory (plus potentially some additional structured information file later on).

Fine for me.

For all those other files, it's not immediately clear where they should be used. For now, I would just move them to resources/. If we also want to have a model selection benchmark collection, we should probably decide now if that's supposed to be included in this repo (and if so how we would separate the basic parameter estimation problems from model selection problems).

Agreed. There is no plan for a model selection benchmark collection currently. The additional curation of e.g. alternative formulations and model selection problems are useful but out-of-scope of benchmarking PEtab, so I would suggest we just keep them unstructured inside resources/ and briefly document each file in README.md or some YAML.

dweindl · 2026-09-25T14:35:28Z

so I would suggest we just keep them unstructured inside resources/ and briefly document each file in README.md or some YAML.

Works for. README.md is probably more helpful. I'd keep that for a later PR.

dilpath

Thanks!

I guess the Fujita alternative formulations are now broken. Ideally:

every PEtab problem YAML everywhere (including in resources/) can be loaded/is a valid problem when loaded
every v1/problem.yaml produces the same simulated data as in simulations.tsv, as some kind of validation that nothing broke during the restructure. Also the alternative formulations if it's easy (e.g. compare the simulation with the simulation generated by the repo commit before the restructure).

dilpath · 2026-09-25T14:32:21Z

      run: |
-        for problem in `ls -1 Benchmark-Models/`; do
+        for problem in `ls -1 problems/`; do
            python src/python/simulate.py "$problem" -j


Rename to simulate_v1.py?

I'd do that separately with other v2-related updates.

dilpath · 2026-09-25T14:34:51Z

+  - [ ] Files are placed under `problems/<ProblemID>/v1/` (`problem.yaml`, `model.xml`, `conditions.tsv`,
+    `measurements.tsv`, `observables.tsv`, `parameters.tsv`, `simulations.tsv`, `visualizations.tsv`);
+    any other material (raw data, scripts, notebooks, ...) goes in `problems/<ProblemID>/resources/`


Until now, this template looks v1/v2 agnostic, but fine to keep

Right. For now, only making space to accommodate a v2 tree later on. v2 policy, documentation, and code support to be added later.

dilpath · 2026-09-25T14:43:19Z

-    yaml_path = Path(MODELS_DIR, id_, id_ + ".yaml")
-    if not yaml_path.exists():
-        yaml_path = Path(MODELS_DIR, id_, "problem.yaml")
+    yaml_path = Path(MODELS_DIR, id_, "v1", "problem.yaml")


V1 = "v1"? PROBLEM_FILENAME = "problem.yaml"? MEASUREMENTS_FILENAME = "measurements.tsv"?

dilpath · 2026-09-25T14:46:12Z

Why is this symlink useful? Fine to keep

Just renaming it. This was added 4 years ago (#167). This is currently required for editable installations of the Python package to work.

dilpath · 2026-09-25T14:47:39Z

 *egg-info
 src/python/build
-Benchmark-Models/Smith_BMCSystBiol2013/amici_models/*
+problems/Smith_BMCSystBiol2013/v1/amici_models/*


Strange to have this in particular... gitignore any amici_models directory?

Indeed. Added 3 years ago (#180). I'll replace it by a flat amici_models.

dilpath · 2026-09-25T14:56:26Z

so I would suggest we just keep them unstructured inside resources/ and briefly document each file in README.md or some YAML.

Works for. README.md is probably more helpful. I'd keep that for a later PR.

Agreed. One argument for a YAML is that it would be easy to check via CI that every file inside resources is at least described.

dweindl · 2026-09-28T07:33:50Z

I guess the Fujita alternative formulations are now broken. Ideally:

Ideally, there wouldn't be any alternative formulations 🙈 .

* every PEtab problem YAML everywhere (including in  `resources/`) can be loaded/is a valid problem when loaded

I can add that.

* every v1/problem.yaml produces the same simulated data as in simulations.tsv, as some kind of validation that nothing broke during the restructure. Also the alternative formulations if it's easy (e.g. compare the simulation with the simulation generated by the repo commit before the restructure).

That's out of scope for me. This whole simulation table setup is a mess (see also #278, #279, #280), and I don't think this should block adding v2 problems.

dweindl · 2026-09-28T07:41:46Z

so I would suggest we just keep them unstructured inside resources/ and briefly document each file in README.md or some YAML.

Works for. README.md is probably more helpful. I'd keep that for a later PR.

Agreed. One argument for a YAML is that it would be easy to check via CI that every file inside resources is at least described.

That is true, but I think it's tedious to describe every single file if it's already part of a petab problem.

For the current state of Blasi_CellSystems2016, this would mean annotating >15 files.

├── Blasi_CellSystems2016
│   ├── resources
│   │   ├── model_Blasi_CellSystems2016_allCombinations.xml
│   │   └── petab_select
│   │       ├── 01_create_output_dirs.sh
│   │       ├── 02_create_sbml.py
│   │       ├── 03_create_petab.py
│   │       ├── 04_create_select.py
│   │       ├── input
│   │       │   ├── petab_problem.yaml
│   │       │   └── petab_select_problem.yaml
│   │       ├── model_info.py
│   │       ├── output
│   │       │   ├── model
│   │       │   │   └── model.xml
│   │       │   ├── petab
│   │       │   │   ├── conditions.tsv
│   │       │   │   ├── measurements.tsv
│   │       │   │   ├── observables.tsv
│   │       │   │   ├── parameters.tsv
│   │       │   │   └── petab_problem.yaml
│   │       │   └── select
│   │       │       ├── model_space.tsv
│   │       │       └── petab_select_problem.yaml
│   │       └── README.md

I thin, I'd prefer a human-readable readme with a high-level overview instead of a machine readable file-by-file description.

dilpath · 2026-09-28T09:40:08Z

* every PEtab problem YAML everywhere (including in  `resources/`) can be loaded/is a valid problem when loaded

I can add that.

Thanks!

* every v1/problem.yaml produces the same simulated data as in simulations.tsv, as some kind of validation that nothing broke during the restructure. Also the alternative formulations if it's easy (e.g. compare the simulation with the simulation generated by the repo commit before the restructure).
That's out of scope for me. This whole simulation table setup is a mess (see also #278, #279, #280), and I don't think this should block adding v2 problems.

I think that's fine. The point is to ensure no problem was broken somehow due to the restructure. The "validation" through AMICI ( https://github.com/AMICI-dev/AMICI/blob/main/tests/benchmark_models/benchmark_models.yaml ) is sufficient for me to ensure nothing was broken, although that doesn't include all models.

dilpath · 2026-09-28T09:47:44Z

so I would suggest we just keep them unstructured inside resources/ and briefly document each file in README.md or some YAML.

Works for. README.md is probably more helpful. I'd keep that for a later PR.

Agreed. One argument for a YAML is that it would be easy to check via CI that every file inside resources is at least described.

That is true, but I think it's tedious to describe every single file if it's already part of a petab problem.

For the current state of Blasi_CellSystems2016, this would mean annotating >15 files.

[...]

I thin, I'd prefer a human-readable readme with a high-level overview instead of a machine readable file-by-file description.

I think you're right overall, fine for me to go with a README then. These things will probably all be generated anyway.

dilpath

Thanks!

Feel free to merge once you're somewhat confident that the problems haven't changed due to the restructure.

dweindl · 2026-09-28T11:28:04Z

* every PEtab problem YAML everywhere (including in  `resources/`) can be loaded/is a valid problem when loaded

I can add that.

Thanks!

Turns out that this will require a bit more thought. There is for example Blasi_CellSystems2016/resources/petab_select/input/petab_problem.yaml which, I assume, is just a template and was never meant to be a valid problem. So we'd need some exclude list.
There are some other auxiliary petab problems with additional existing problems. I will not touch those now. I will add the check as informational-only, i.e. not failing the workflow.
All problems/*/v1/problem.yaml are passing.

dilpath · 2026-09-28T12:10:59Z

* every PEtab problem YAML everywhere (including in  `resources/`) can be loaded/is a valid problem when loaded
I can add that.
Thanks!
Turns out that this will require a bit more thought. There is for example Blasi_CellSystems2016/resources/petab_select/input/petab_problem.yaml which, I assume, is just a template and was never meant to be a valid problem. So we'd need some exclude list. There are some other auxiliary petab problems with additional existing problems. I will not touch those now. I will add the check as informational-only, i.e. not failing the workflow. All problems/*/v1/problem.yaml are passing.

Ah, I just meant as a one-off check to ensure the restructure didn't break anything. Not that we need to add something to CI. i.e. just to make sure real additional problems, like provided for Fujita, are still valid after the restructure. And agreed, fine to ignore the "intentionally invalid" things like the Blasi template.

Add bmp-check-petab-yaml: walks problems/ for any YAML file that looks like a PEtab v1 problem (has format_version and problems keys) and validates it with petablint. This catches broken supplementary problem definitions anywhere under a problem's resources/, not just the canonical MODELS. Wire it into the tests.yml CI job as an informational, non-blocking step for now, since it also surfaces a few pre-existing, unrelated data issues that are out of scope here. Fix the relative file references in resources. Flush stdout around each petablint subprocess call in both check_petablint.py and check_petab_yaml.py, since unflushed, buffered print() output otherwise gets reordered relative to the subprocess's own output when stdout isn't a tty, as in CI. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

dweindl and others added 2 commits September 25, 2026 15:29

dweindl marked this pull request as ready for review September 25, 2026 14:07

dweindl requested review from dilpath and m-philipps as code owners September 25, 2026 14:07

dilpath requested changes Sep 25, 2026

View reviewed changes

dweindl added 2 commits September 28, 2026 08:31

Extract constants

cad48f8

fix ignore

52fa52d

dilpath approved these changes Sep 28, 2026

View reviewed changes

dweindl force-pushed the restructure-petab-dirs branch from 17897b9 to 8a3b1a7 Compare September 28, 2026 16:40

dweindl merged commit fcbddf1 into Benchmarking-Initiative:master Sep 28, 2026
3 checks passed

dweindl deleted the restructure-petab-dirs branch September 28, 2026 17:29

Conversation

dweindl commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dweindl commented Sep 25, 2026

Uh oh!

dilpath commented Sep 25, 2026

Uh oh!

dweindl commented Sep 25, 2026

Uh oh!

dilpath left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dilpath commented Sep 25, 2026

Uh oh!

dweindl commented Sep 28, 2026

Uh oh!

dweindl commented Sep 28, 2026

Uh oh!

dilpath commented Sep 28, 2026

Uh oh!

dilpath commented Sep 28, 2026

Uh oh!

dilpath left a comment

Choose a reason for hiding this comment

Uh oh!

dweindl commented Sep 28, 2026

Uh oh!

dilpath commented Sep 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dweindl commented Sep 25, 2026 •

edited

Loading