Skip to main content

L2 - Validation phase

Start situation​

At the start of the validation phase, the virtual lab has the required components and the workflow has been developed and tested. Additionally the scientific (golden) user has made the data sets ready for the experiments.

During the validation phase​

The golden user will conduct experiments in the virtual lab in this phase. The following should also be done phase:

  • Data
    • Make data fair.
    • Follow the community standards of the relevant domains for reading, writing and exchanging data.
  • Metadata
    • Complete all metadata fields.
  • Scenarios
    • Make sure and describe how the virtual lab can be used on different scenarios, i.e. other datasets and with other parameters.
  • Versioning
    • Add a version number to the virtual lab so users can refer to this number when they are reporting reproducibility or bug issues.
    • Give each containerized cell and executed workflow a persistent identifier and version number. Feature is currently under development. For progress, see #280.
  • Documentation
  • Workflow
    • Check the duration of computation, memory usage, and power usage of the containers.
  • Codebase
    • Add unit tests to verify the behavior of used methods and libraries.
    • Define clear responsibilities of all notebook cells, methods and classes.
  • Infrastructure
    • Check the infrastructure requirements for the workshop:
      • The number of people taking part in a workshop.
      • The random access memory and permanent storage usage of the virtual lab.

Validation phase milestones​

  • The submission for publication of a paper in the ecosystem domain presenting the scenarios run in the virtual lab.
  • The lab becoming publicly available.
  • Potentially, besides a domain paper, the submission for publication of a technical paper focusing on the infrastructure part of the virtual research environment.

Validation phase responsibilities​

The following roles should be assigned during the validation phase:

  • Golden users: Use the virtual lab to do their research. Publish a paper in the ecosystem domain.
  • Virtual lab developers: Provide support to the golden users executing experiments in the virtual lab. Help prepare the virtual lab for workshop use.
  • Community supporter: Finds potential silver users to take part in the trainings in the next phase.
  • Virtual lab trainer: Tries out the virtual lab and makes suggestions. Helps the golden users identify issues arising when others start using the virtual lab.
  • Infrastructure supporter: Scales infrastructure if needed for the experiments. Prepares the infrastructure for the trainings.
  • VRE researcher: Potentially publishes a technical paper in addition to the domain paper.
  • Virtual lab coordinator: Ensures the community supporter is involved in the virtual lab and supports solving any issues arising.
  • VRE DevOps engineer: Support thevirtual labdevelopers and golden user by providing advise and changing the virtual research environment where necessary.

Exit condition​

The virtual lab transitions to workshop use, once validation of the lab by doing scientific experiments has been done and a scientific paper is ready for publication.

It is important to find out who would be potential silver users of the virtual lab. In case no match between the virtual lab and potential silver users can be found, the virtual lab should be archived.

To advance to workshop use, all of the following boxes should be checked, in addition to the criteria for the initial proposal, and co-development:

  • User community
    • Other potential users have been identified.
  • Data
    • All input data of the lab is FAIR.
  • Metadata
    • All the fields of the metadata standard are present.
  • Scenarios
    • The virtual lab can be used in multiple scenarios, i.e. both the parameters and datasets can be changed to suit experiments of different researchers.
  • Documentation
    • There is a user manual and at least one domain scientist who was not involved in the development of the virtual lab has reviewed the user manual. The coding experience of the reviewer of the user manual is similar to the coding experience of the intended user.
    • How to use the virtual lab on a different scenario is explained.
  • Codebase
    • Unit tests verify the behavior of used methods and libraries. There should be a testing guideline, which will be done in this issue #274.
    • The virtual lab reads, writes and exchanges data in a way that meets domain-relevant community standards. Recommendations for what standards to use are under investigation, see github issue #281.
    • The code within cells is easily human-readable and others can easily modify it. If methods have side effects, this is clear to the user.
    • The input and output of each cell is clear. It is both clear what the structure is (e.g. what data type is used) and what the data content is from a domain perspective.
  • Workflow
    • The duration of computation, memory usage, and power usage of the containers is acceptable. As there is currently no dashboard to monitor resource usage, contact the VLIC team for guidelines.
  • Deployment
    • The virtual lab is publicly available.
  • Infrastructure
    • The infrastructure requirements for the workshop are known and the necessary infrastructure has been provided:
      • The number of people taking part in a workshop.
      • The random access memory and permanent storage usage of the virtual lab are known.
      • The amount of processors the virtual lab uses is known.