Application Script Examples


GROMACS Applications

GROningen MAchine for Chemical Simulations (GROMACS) is a   molecular dynamics   package mainly designed for simulations of   proteins  ,   lipids  , and   nucleic acids  . It was originally developed in the Biophysical Chemistry department of   University of Groningen  , and is now maintained by contributors in universities and research centres worldwide. GROMACS is one of the fastest and most popular software packages available, and can run on   central processing units   (CPUs) and   graphics processing units   (GPUs).

 

Step 1:  Log in to ParamShakti with your login credentials.                                             

                        ssh   your_name@paramshakti.iitkgp.ac.in

Step 2: Load Gromacs software.

       Check for availability of the software using “module avail gromacs”. It shows the list of available   

        gromacs software versions on ParamShakti. Then, load the gromacs software using “module load

        gromacs_version”   (For example, “module load apps/gromacs/2020/cpu”).

 Step 3: Gromacs Input File

            

 

        a)create a directory in your scratch folder and navigate to that directory.

             “mkdir /scratch/$USER/gromacs; cd /scratch/$USER/gromacs”

       b)Input file can be download to /scratch/$USER/ from 

             “wget https://ftp.gromacs.org/pub/benchmarks/water_GMX50_bare.tar.gz”

             “tar xzf water_GMX50_bare.tar.gz”

        c) Navigate to directory 3072 ”cd water-cut1.0_GMX50_bare/3072”

             you will see 4 input files “conf.gro  pme.mdp  rf.mdp   topol.top”

        d) Generate the "topol.tpr" binary file using the command "module load apps/gromacs/2020/cpu;gmx_mpi grompp -f pme.mdp -c conf.gro -p topol.top"

             you will see 4 input files “conf.gro  pme.mdp  rf.mdp   topol.top”

          

       The mdp option can be used is pme with 50000 steps

pme.mdp

title                    = dppc

cpp                      = /lib/cpp

integrator               = md

nsteps                   = 50000           ; !autoset

nstlist                  = 10

nstfout                  = 0

nstxout                  = 0

nstvout                  = 0

nstxtcout                = 0

nstlog                   = 0

dt                       = 0.002

constraints = all-bonds

nstenergy                = 0

ns_type                  = grid

coulombtype              = PME          ; !autoset

rlist                    = 1.0          ; !autoset

rvdw                     = 1.0          ; !autoset

rcoulomb                 = 1.0          ; !autoset

tcoupl                   = v-rescale

tc_grps                  = system

tau_t                    = 0.1

ref_t                    = 300

 

freezegrps                =              ; !autoset

freezedim                 =              ; !autoset

 

fourier_spacing = 0.125                 ; !autoset

nstcalcenergy            = 100                  ; !autogen

cutoff-scheme            = verlet               ; !autogen

 

topol.top

 

#include "oplsaa.ff/forcefield.itp"

[ moleculetype ]

; molname       nrexcl

SOL             2

 

[ atoms ]

;   nr   type   resnr residue  atom   cgnr      charge       mass

     1  opls_116    1    SOL     OW       1      -0.8476

     2  opls_117    1    SOL    HW1       2       0.4238

     3  opls_117    1    SOL    HW2       3       0.4238

 

#ifndef FLEXIBLE

[ settles ]

; OW    funct   doh      dhh

1       1       0.1      0.16330

 

[ exclusions ]

1       2       3

2       1       3

3       1       2

#else

[ bonds ]

; i     j       funct    length  force.c.

1       2       1        0.1     345000  0.1      345000

1       3       1        0.1     345000  0.1      345000

 

[ angles ]

; i     j       k        funct   angle   force.c.

2       1       3        1       109.47  383      109.47  383

#endif

 

[ system ]

Water

 

[ molecules ]

SOL    256000 

Step 4: Copy the slurm script given below and save it as “submit.sh”. Now, submit the job using the command “sbatch submit.sh”. On successful submission, you can see the job status using “squeue -u $USER”.

MPI-CPU Script

 This will initiate 4 MPI Process with 4 threads each .

#!/bin/bash

#SBATCH -J job_name                # name of the job

#SBATCH -p medium            # name of the partition: available options "large, medium or shared"

#SBATCH -n 4                      # no of processes or tasks

#SBATCH --cpus-per-task=4          # no of threads per process or task

#SBATCH -t 01:00:00                # walltime in HH:MM:SS, Max value 72:00:00

#list of modules you want to use, for example

module load apps/gromacs/2020/cpu

#name of the executable

exe="gmx_mpi"

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

#run the application

mpirun -bootstrap slurm -n $SLURM_NTASKS $exe mdrun -s topol.tpr    # specify the application command-line options, if any, after $exe

GPU Script

 This will initiate 4 MPI Process with 6 threads(4 Worker threads and 2 communication threads) in a single GPU device .

 

#!/bin/bash

#SBATCH -J job_name                # name of the job

#SBATCH -p gpu            # name of the partition: available options "gpu"

#SBATCH -n 4                       # no of processes or tasks

#SBATCH --gres=gpu:1               # request gpu card: it should be either 1 or 2

#SBATCH --cpus-per-task=4          # no of threads per process or task

#SBATCH -t 01:00:00                # walltime in HH:MM:SS, max value 72:00:00

#list of modules you want to use, for example

module load apps/gromacs/2020/gpu_cuda_10.1

#name of the executable

exe="gmx_mpi"

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

#run the application

mpirun -bootstrap slurm -n $SLURM_NTASKS $exe mdrun -s topol.tpr   # specify the application command-line options, if any, after $exe

 

Step 5: Gromacs output files 

        After the job starts running, you can see the output files related to Gromacs in the same directory. For example, 

[xxxxxxx@login01 3072]$ cat slurm-112741.out

==========================================

SLURM_CLUSTER_NAME = param-shakti

SLURM_JOB_ACCOUNT = xxxxxxx

SLURM_JOB_ID = 112741

SLURM_JOB_NAME = job_name

SLURM_JOB_NODELIST = cn098

SLURM_JOB_USER = xxxxxxx

SLURM_JOB_UID = 6034

SLURM_JOB_PARTITION = medium

SLURM_TASK_PID = 40413

SLURM_SUBMIT_DIR = /scratch/xxxxxxx/gromacs/water-cut1.0_GMX50_bare/3072

SLURM_CPUS_ON_NODE = 16

SLURM_NTASKS = 4

SLURM_TASK_PID = 40413

==========================================

                  :-) GROMACS - gmx mdrun, 2020-UNCHECKED (-:

 

                             GROMACS is written by:

     Emile Apol      Rossen Apostolov      Paul Bauer     Herman J.C. Berendsen

    Par Bjelkmar      Christian Blau   Viacheslav Bolnykh     Kevin Boyd

 Aldert van Buuren   Rudi van Drunen     Anton Feenstra       Alan Gray

  Gerrit Groenhof     Anca Hamuraru    Vincent Hindriksen  M. Eric Irrgang

  Aleksei Iupinov   Christoph Junghans     Joe Jordan     Dimitrios Karkoulis

    Peter Kasson        Jiri Kraus      Carsten Kutzner      Per Larsson

  Justin A. Lemkul    Viveca Lindahl    Magnus Lundborg     Erik Marklund

    Pascal Merz     Pieter Meulenhoff    Teemu Murtola       Szilard Pall

    Sander Pronk      Roland Schulz      Michael Shirts    Alexey Shvetsov

   Alfons Sijbers     Peter Tieleman      Jon Vincent      Teemu Virolainen

 Christian Wennberg    Maarten Wolf      Artem Zhmurov

                            and the project leaders:

        Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel

 

Copyright (c) 1991-2000, University of Groningen, The Netherlands.

Copyright (c) 2001-2019, The GROMACS development team at

Uppsala University, Stockholm University and

the Royal Institute of Technology, Sweden.

check out http://www.gromacs.org for more information.

 

GROMACS is free software; you can redistribute it and/or modify it

under the terms of the GNU Lesser General Public License

as published by the Free Software Foundation; either version 2.1

of the License, or (at your option) any later version.

 

GROMACS:      gmx mdrun, version 2020-UNCHECKED

Executable:    /home/apps/gromacs_2020/bin/gmx_mpi

Data prefix:   /home/apps/gromacs_2020

Working dir:   /scratch/xxxxxx/gromacs/water-cut1.0_GMX50_bare/3072

Command line:

  gmx_mpi mdrun -s topol.tpr

 

Compiled SIMD: AVX2_256, but for this host/run AVX_512 might be better (see

log).

Reading file topol.tpr, VERSION 2020-UNCHECKED (single precision)

Changing nstlist from 10 to 40, rlist from 1 to 1.099

 

Using 4 MPI processes

 

Non-default thread affinity set, disabling internal thread affinity

 

Using 4 OpenMP threads per MPI process

 

starting mdrun 'Water'

5000 steps,     10.0 ps.

 

Writing final coordinates.

 

 

Dynamic load balancing report:

 DLB was off during the run due to low measured imbalance.

 Average load imbalance: 0.5%.

 The balanceable part of the MD step is 60%, load imbalance is computed from this.

 Part of the total run time spent waiting due to load imbalance: 0.3%.

 

 

               Core t (s)   Wall t (s)        (%)

       Time:    19057.053     1191.067     1600.0

                  (ns/day)    (hour/ns)

Performance:         0.726       33.079

 

GROMACS reminds you: "The Microsecond is Within Reach" (P.J. Van Maaren)

 

[xxxxxx@login01 3072]$

LAMMPS applications

LAMMPS is an acronym for Large-scale Atomic/ Molecular Massively Parallel Simulator. This is extensively used in the fields of Material Science, Physics, Chemistry and may others. More information about LAMMPS may please be found at  https://lammps.sandia.gov.

Step 1:  Log in to ParamShakti with your login credentials.                                             

                        ssh   your_name@paramshakti.iitkgp.ac.in

Step 2: Load LAMMPS software.

       Check for availability of the software using “module avail lammps”. It shows the list of available   

        lammps software versions on ParamShakti. Then, load the lammps software using “module load

        lammps_version”   (For example, “module load apps/lammps/12.12.2018/intel”).

 Step 3: LAMMPS Input file

                Create a new directory and copy the LAMMPS input file given below and save the file as “in.lj”.

# 3d Lennard-Jones melt

 

variable        x index 1

variable        y index 1

variable        z index 1

 

variable        xx equal 64*$x

variable        yy equal 64*$y

variable        zz equal 64*$z

 

units           lj

atom_style      atomic

 

lattice         fcc 0.8442

region          box block 0 ${xx} 0 ${yy} 0 ${zz}

create_box      1 box

create_atoms    1 box

mass            1 1.0

 

velocity        all create 1.44 87287 loop geom

 

pair_style      lj/cut 2.5

pair_coeff      1 1 1.0 1.0 2.5

 

neighbor        0.3 bin

neigh_modify    delay 0 every 20 check no

 

fix             1 all nve

 

run             1000000

 

Step 4: Copy the slurm script given below and save it as “submit.sh”. Now, submit the job using the command “sbatch submit.sh”. On successful submission, you can see the job status using “squeue -u $USER”.

 MPI-CPU Script

 This will initiate 4 MPI Process with 1 threads(1 worker thread) each .

#!/bin/bash

#SBATCH -J job_name                # name of the job

#SBATCH -p medium       # name of the partition: available options "large, medium or shared "

#SBATCH -n 4                       # no of processes or tasks

#SBATCH --cpus-per-task=1          # no of threads per process or task

#SBATCH -t 01:00:00                # walltime in HH:MM:SS, max value 72:00:00

 

#list of modules you want to use, for example

module load apps/lammps/7Aug19/intel

#name of the executable

exe="lmp_mpi"

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

#run the application

mpirun -bootstrap slurm -n $SLURM_NTASKS $exe -in in.lj   # specify the application command-line options, if any, after $exe

 GPU Script

 This will initiate 4 MPI Process in a single GPU device .

#!/bin/bash

#SBATCH -J job_name                # name of the job

#SBATCH -p gpu            # name of the partition: available options "gpu"

#SBATCH -n 4                       # no of processes or tasks

#SBATCH --gres=gpu:1               # request gpu card: it should be either 1 or 2

#SBATCH --cpus-per-task=1          # no of threads per process or task

#SBATCH -t 01:00:00                # walltime in HH:MM:SS, max value 72:00:00

 

#list of modules you want to use, for example

module load apps/lammps/29Oct2020/cuda10.2/gpu

#name of the executable

exe="lmp_gpu"

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

#run the application

 mpirun -bootstrap slurm -n $SLURM_NTASKS $exe -sf gpu -pk gpu 1 -in in.lj   # specify the application command-line options, if any, after $exe

Step 5: LAMMPS output files 

        After the job starts running, you can see the output files related to LAMMPS in the same directory. For example, see log.lammps file  

LAMMPS (22 Aug 2018)

   using 1 OpenMP thread(s) per MPI task

Lattice spacing in x,y,z = 1.6796 1.6796 1.6796

Created orthogonal box = (0 0 0) to (107.494 107.494 107.494)

   8 by 10 by 16 MPI processor grid

Created 1048576 atoms

   Time spent = 0.048476 secs

Neighbor list info ...

   update every 20 steps, delay 0 steps, check no

   max neighbors/atom: 2000, page size: 100000

   master list distance cutoff = 2.8

   ghost atom cutoff = 2.8

   binsize = 1.4, bins = 77 77 77

   1 neighbor lists, perpetual/occasional/extra = 1 0 0

   (1) pair lj/cut, perpetual

       attributes: half, newton on

      pair build: half/bin/atomonly/newton

      stencil: half/bin/3d/newton

      bin: standard

Setting up Verlet run ...

   Unit style    : lj

   Current step  : 0

   Time step     : 0.005

Per MPI rank memory allocation (min/avg/max) = 2.699 | 2.703 | 2.708 Mbytes

Step Temp E_pair E_mol TotEng Press

       0          1.44   -6.7733681            0    -4.6133701   -5.0196704

 1000000    0.65695755   -5.7125359            0    -4.7271005   0.48799127

Loop time of 723.716 on 1280 procs for 1000000 steps with 1048576 atoms

 

Performance: 596918.946 tau/day, 1381.757 timesteps/s

99.5% CPU use with 1280 MPI tasks x 1 OpenMP threads

 

MPI task timing breakdown:

Section |  min time   |  avg time  |  max time  |%varavg| %total

---------------------------------------------------------------

Pair    | 424.38      | 435.47     | 461.05     |   26.2 | 60.17

Neigh   | 59.782      | 60.365     | 62.991     |    3.9 |  8.34

Comm    | 193.24      | 219.39     | 231.11     |   38.5 | 30.31

Output  | 0.00013494 | 0.00085223 | 0.0088639  |   0.0 |  0.00

Modify  | 6.4813      | 6.6462     | 7.541      |    5.6 |  0.92

Other   |             | 1.841      |            |       |   0.25

 

Nlocal:    819.2 ave 845 max 786 min

Histogram: 3 2 34 115 256 372 315 137 33 13

Nghost:    2417.97 ave 2468 max 2369 min

Histogram: 8 31 81 216 314 327 202 76 22 3

Neighs:    30698 ave 32432 max 28796 min

Histogram: 4 16 47 194 306 325 245 103 34 6

 

Total # of neighbors = 39293494

Ave neighs/atom = 37.4732

Neighbor list builds = 50000

Dangerous builds not checked

Total wall time: 0:12:03

NAMD Application

Nano Scale Molecular Dynamics (NAMD) software for molecular dynamics simulation is designed for high-performance simulations of Large Macro Molecular system on parallel computers. This software also makes use of GPGPUs. NAMD was developed by the Theoretical and Computational Biophysics Group in the Beckman Institute for Advanced Science and Technology at the University of Illinois at Urbana-Champaign. More information about NAMD may please be found at   http://www.ks.uiuc.edu/Research/namd/  

Step 1:  Log in to ParamShakti with your login credentials.                                             

                        ssh   your_name@paramshakti.iitkgp.ac.in

Step 2: Load namd software.

       Check for availability of the software using “module avail namd”. It shows the list of available   

        namd software versions on ParamShakti. Then, load the namd software using “module load

        namd_version”   (For example, “module load apps/namd/2.13/impi2019v5/cpu”).

 Step 3: Namd Input file 

            

 

 

 a)create a directory in your scratch folder and navigate to that         

      directory.

      “mkdir /scratch/$USER/namd; cd /scratch/$USER/namd”

 

 b)Input file can be download to /scratch/$USER/ from

  “wget http://www.ks.uiuc.edu/Research/namd/utilities/stmv.tar.gz”

  “tar xzf stmv.tar.gz”

   mv stmv/* .

 

 c)Edit  stmv.namd and change value of numsteps e.g. to 10000

 

stmv.namd 

        

#############################################################

## ADJUSTABLE PARAMETERS                                   ##

#############################################################

structure          stmv.psf

coordinates        stmv.pdb

 

 

#############################################################

## SIMULATION PARAMETERS                                   ##

#############################################################

 

# Input

paraTypeCharmm      on

parameters          par_all27_prot_na.inp

temperature         298

 

 

# Force-Field Parameters

exclude             scaled1-4

1-4scaling          1.0

cutoff              12.

switching           on

switchdist          10.

pairlistdist        13.5

 

 

# Integrator Parameters

timestep            1.0

nonbondedFreq       1

fullElectFrequency  4

stepspercycle       20

 

 

# Constant Temperature Control

langevin            on    ;# do langevin dynamics

langevinDamping     5      ;# damping coefficient (gamma) of 5/ps

langevinTemp        298

langevinHydrogen    off     ;# don't couple langevin bath to hydrogens

 

 

# Constant Pressure Control (variable volume)

useGroupPressure      yes ;# needed for rigidBonds

useFlexibleCell       no

useConstantArea       no

 

langevinPiston        on

langevinPistonTarget  1.01325 ;#   in bar -> 1 atm

langevinPistonPeriod  100.

langevinPistonDecay   50.

langevinPistonTemp    298

 

cellBasisVector1     216.832     0.   0.

cellBasisVector2     0.    216.832    0.

cellBasisVector3     0.     0   216.832

cellOrigin           0.     0.   0.

 

PME                  on

PMEGridSizeX         216

PMEGridSizeY         216

PMEGridSizeZ         216

 

# Output

outputName          /usr/tmp/stmv-output

 

outputEnergies      20

outputTiming        20

 

numsteps            10000

Step 4: Copy the slurm script given below and save it as “submit.sh”. Now, submit the job using the command “sbatch submit.sh”. On successful submission, you can see the job status using “squeue -u $USER”. It will initiate 2 MPI Process, each with 5 threads (4 worker threads and 1 communication thread).

MPI-CPU Script

 

#!/bin/bash

#SBATCH -J job_name                # name of the job

#SBATCH -p medium            # name of the partition: available options "large, medium or shared "

#SBATCH -n 2                      # no of processes or tasks

#SBATCH --cpus-per-task=4          # no of threads per process or task

#SBATCH -t 01:00:00                # walltime in HH:MM:SS, Max value 72:00:00

 

#list of modules you want to use, for example

module load apps/namd/2.13/impi2019v5/cpu

 

#name of the executable

exe="namd2"

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

#run the application

mpirun -bootstrap slurm -n $SLURM_NTASKS $exe +ppn $OMP_NUM_THREADS +idlepoll ./stmv.namd     # specify the application command-line options, if any, after $exe

GPU Script

This will initiate 2 MPI Process with 9 threads each (6 worker threads per process + addition 3 communication threads)

#!/bin/bash

#SBATCH -J job_name                # name of the job

#SBATCH -p gpu             # name of the partition: available options "gpu"

#SBATCH -n 2                       # no of processes or tasks (Max 2 for this script)

#SBATCH --gres=gpu:2               # request gpu card: it should be either 1 or 2

#SBATCH --cpus-per-task=6          # no of threads per process or task

#SBATCH -t 01:00:00                # walltime in HH:MM:SS, max value 72:00:00

 

#list of modules you want to use, for example

module load apps/namd/2.13/impi2019v5/cuda

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

 

#name of the executable

exe="namd2"

 

#run the application

mpirun -bootstrap slurm -n $SLURM_NTASKS $exe +ppn $OMP_NUM_THREADS +idlepoll ./stmv.namd  +devices 0,1    # specify the application command-line options, if any, after $exe

 

 

Step 5: NAMD output files 

        After the job starts running, you can see the output files related to Namd in the same directory. For example, 

 

TIMING: 10000  CPU: 140.503, 0.0137648/step  Wall: 142.867, 0.0139643/step, 0 hours remaining, 2153.882812 MB of memory in use.

ETITLE:      TS           BOND          ANGLE          DIHED          IMPRP               ELECT            VDW       BOUNDARY           MISC        KINETIC               TOTAL           TEMP       POTENTIAL         TOTAL3        TEMPAVG            PRESSURE      GPRESSURE         VOLUME       PRESSAVG      GPRESSAVG

 

ENERGY:   10000     367211.7587    279345.1935     82007.0777      5091.8001       -4523649.5745    385248.5094         0.0000         0.0000    945987.1898       -2458758.0453       297.5373   -3404745.2351  -2450238.2196       297.4597            -56.1661       -17.8061   10187151.2683        17.4129        18.8925

 

WRITING EXTENDED SYSTEM TO OUTPUT FILE AT STEP 10000

WRITING COORDINATES TO OUTPUT FILE AT STEP 10000

The last position output (seq=-2) takes 0.012 seconds, 2232.031 MB of memory in use

WRITING VELOCITIES TO OUTPUT FILE AT STEP 10000

The last velocity output (seq=-2) takes 0.012 seconds, 2232.105 MB of memory in use

====================================================

 

WallClock: 168.633499  CPUTime: 165.723175  Memory: 2232.113281 MB

[Partition 0][Node 0] End of program

 

real    2m55.139s

user    96m16.215s

sys     9m7.399s

 

o/html>