digital-experiments is a lightweight tool for automatically tracking and analysing the configuration and results of coding experiments.

Compared to other such tools, digital-experiments:

• improves reproducibility by not only tracking the inputs and outputs of an experiment, but also the exact code and environment used at run-time.

• improves interpretability by automatically associating results with the experiment that generated them. No more results-v2-final.csv files.

Quickstart

1: definition: example.py

from digital_experiments import (
      experiment,
)

@experiment
def example(a: int, b: int = 3):
      # perform your experiment here:
      #   - train a model
      #   - run a simulation
      #   - etc.
      # and return the result
      # (any arbitrary object/s)
      return a ** b

if __name__ == "__main__":
      # run the experiment
      # like any other function
      print(example(2))
      print(example(b=2, a=3))

2: running

$ python example.py
8
9

3: analysis

>>> from example import example

>>> for o in example.observations():
...     print(o.config, "→", o.result)
{'a': 2, 'b': 3} →  8
{'a': 3, 'b': 2} →  9

>>> o.metadata["code"]
@experiment
def example(a, b):
    return a ** b

>>> o.metadata["timing"]
{
  "start": "2024-02-24 10:10:01",
  "end": "2024-02-24 10:10:01",
  "duration": 0.000007
}

>>> o.metadata["environment"].keys()
['system', 'pip_freeze', 'git']

>>> example.to_dataframe()
      id config.a config.b  result
0  <id1>       2         3       8
1  <id2>       3         2       9

See a Kitchen Sink example for a summary of all the functionality of digital-experiments, or The Basics for a step-by-step guide.

Installation

$ pip install digital-experiments

Extending digital-experiments

digital-experiments is designed to be easily extended and hacked to suit your needs. Three obvious ways this can be done are by implementing custom:

1. Callbacks to trigger custom logic before and after each experiment is run.

2. Backends to allow you to save your Observation to disk in custom formats.

3. Controllers to define automated workflows for your experiments.

Contributing

Contributions are welcome! Head on over to the GitHub repository to get started.