Performance Analytics for Computational Experiments (PACE)

Tool

Please download and use this tool: Python 3 version or Python 2 version to upload experiment data directly from supercomputers.
# To download using command line
wget https://pace.ornl.gov/static/tools/pace-upload (Python 2 version)
wget https://pace.ornl.gov/static/tools/pace-upload3 (Python 3 version)
chmod +x ./pace-upload


This is a Python-based tool that can be executed directly (./pace-upload) once you download and make it executable. The system-wide Python 2.x installation on Summit/Theta/Cori/Compy has the requisite 'requests' module to enable this tool to work.
It presently handles uploading user data in performance archive directories from all E3SM supported machines. The server checks for duplicates while processing uploaded data.
Please see the demo for reference. A log file is written at the end that details the upload progress and notes any errors.

---

Usage

Examples:
./pace-upload --perf-archive user-exps-set
./pace-upload --perf-archive /project/projectdirs/acme/performance_archive/sarat

Detailed usage:
$ ./pace-upload --help
usage: pace-upload [-h] [--exp-dir SOURCE] [--perf-archive SOURCE]
                   [--application {e3sm}]

PACE upload tool.

optional arguments:
  -h, --help            show this help message and exit
  --exp-dir SOURCE, -ed SOURCE
                        Root directory containing experiment(s) results.
                        Handles multiple experiment directories under root
  --perf-archive SOURCE, -pa SOURCE
                        Root directory containing performance archive. Handles
                        multiple performance archive directories under root
  --application {e3sm}, -a {e3sm}
                        Application name

---

Performance Archive Locations

• NERSC: /project/projectdirs/acme/performance_archive
  or /project/projectdirs/<project name>/performance_archive
• ALCF: /projects/ClimateEnergy_3/performance_archive
  or /projects/OceanClimate_2/performance_archive
  or /projects/<project name>/performance_archive
• OLCF: /lustre/atlas/proj-shared/cli115/performance_archive
  or /lustre/atlas/proj-shared/<project name>/performance_archive
• Compy: /compyfs/performance_archive/
• Anvil: /lcrc/group/acme/performance_archive/

Structure within performance archive directory:
• Within the performance_archive directory there is a subdirectory for each user name.
• Within each user name subdirectory is a subdirectory for each case name (as defined by the create_newcase command).
• Within each case name subdirectory is a separate directory for each job with this case name, labelled by the E3SM $LID (essentially the date and time for this particular run: %y%m%d-%H%M%S).