FEM Software¶
Its all in the modules
All packages described in this section, are organized in so-called modules. To list the available versions of a package and load a particular, e.g., ANSYS, version, invoke the commands
marie@login$ module avail ANSYS
[...]
marie@login$ # module load ANSYS/<version>
marie@login$ # e.g.
marie@login$ module load ANSYS/2022R2
The section runtime environment provides a comprehensive overview on the module system and relevant commands.
Abaqus¶
Abaqus is a general-purpose finite element method program designed for advanced linear and nonlinear engineering analysis applications with facilities for linking-in user developed material models, elements, friction laws, etc.
Guide by User¶
Eike Dohmen (from Inst. f. Leichtbau und Kunststofftechnik) sent us the description of his Abaqus calculations. Please try to adapt your calculations in that way. Eike is normally a Windows user and his description contains also some hints for basic Unix commands: Abaqus-Slurm.pdf (only in German).
General¶
Abaqus calculations should be started using a job file (aka. batch script). Please refer to the page covering the batch system Slurm if you are not familiar with Slurm or writing job files.
Usage of Abaqus
(Thanks to Benjamin Groeger, Inst. f. Leichtbau und Kunststofftechnik)).
- Prepare an Abaqus input-file. You can start with the input example from Benjamin: Rot-modell-BenjaminGroeger.inp
- Prepare a job file on ZIH systems like this
#!/bin/bash ### needs ca 20 sec with 4cpu ### generates files: ### yyyy.com ### yyyy.dat ### yyyy.msg ### yyyy.odb ### yyyy.prt ### yyyy.sim ### yyyy.sta #SBATCH --nodes=1 # with >1 node Abaqus needs a nodeliste #SBATCH --ntasks-per-node=4 #SBATCH --mem=2048 # total memory #SBATCH --time=00:04:00 #SBATCH --job-name=yyyy # give a name, what ever you want #SBATCH --mail-type=END,FAIL # send email when the job finished or failed #SBATCH --mail-user=marie@tu-dresden.de # set your email #SBATCH --account=p_number_crunch # charge compute time to project p_number_crunch # Abaqus has its own MPI unset SLURM_GTIDS # load module and start Abaqus module load ABAQUS/2022 abaqus interactive input=Rot-modell-BenjaminGroeger.inp job=yyyy cpus=4 mp_mode=mpi - Start the job file (e.g., name
batch-Rot-modell-BenjaminGroeger.sh)marie@login$ sbatch batch-Rot-modell-BenjaminGroeger.sh # Slurm will provide the Job Id (e.g., 3130522) - Control the status of the job
marie@login$ squeue --me # in column "ST" (Status) you will find a R=Running or P=Pending (waiting for resources)
Ansys¶
Ansys is a general-purpose finite element method program for engineering analysis, and includes preprocessing, solution, and post-processing functions. It is used in a wide range of disciplines for solutions to mechanical, thermal, and electronic problems. Ansys and Ansys CFX used to be separate packages in the past and are now combined.
In general, HPC systems are not designed for interactive working with GUIs. Even so, it is possible to start a Ansys workbench on the login nodes interactively for short tasks. The second and recommended way is to use job files. Both modes are documented in the following.
Since the MPI library that Ansys uses internally (Platform MPI) has some problems integrating
seamlessly with Slurm, you have to unset the enviroment variable SLURM_GTIDS in your
environment befor running Ansysy workbench in interactive and batch mode.
Using Workbench Interactively¶
Ansys workbench (runwb2) can be invoked interactively on the login nodes of ZIH systems for short
tasks.
Note
X11 forwarding needs to enabled when establishing the
SSH connection. For OpenSSH the corresponding option is -X and it is valuable to use
compression of all data via -C.
[marie@login ~]$ start_ansysworkbench.sh
# SSH connection established using -CX
[marie@login$ ~]$ # module load ANSYS/<version>
[marie@login$ ~]$ # e.g.
[marie@login$ ~]$ module load ANSYS/2022R2
[marie@login$ ~]$ runwb2
If more time is needed, a CPU has to be allocated like this (see batch systems Slurm for further information):
[marie@login$ ~]$ # module load ANSYS/<version>
[marie@login$ ~]$ # e.g.
[marie@login$ ~]$ module load ANSYS/2023R1
[marie@login$ ~]$ srun --time=00:30:00 --x11=first [SLURM_OPTIONS] --pty bash
[...]
[marie@compute$ ~]$ runwb2
Better use DCV
The software NICE Desktop Cloud Visualization (DCV) enables to remotly access OpenGL-3D-applications running on ZIH systems using its GPUs (cf. virtual desktops).
Ansys can be used under DCV to make use of GPU acceleration. Follow the instructions within
virtual desktops to set up a DCV session. Then, load a Ansys module, unset
the environment variable SLURM_GTIDS, and finally start the workbench:
marie@gpu$ module load ANSYS
marie@gpu$ unset SLURM_GTIDS
marie@gpu$ runwb2
Using Workbench in Batch Mode¶
The Ansys workbench (runwb2) can also be used in a job file to start calculations (the solver,
not GUI) from a workbench project into the background. To do so, you have to specify the -B
parameter (for batch mode), -F for your project file, and can then either add different commands via
-E parameters directly, or specify a workbench script file containing commands via -R.
Ansys Job File
#!/bin/bash
#SBATCH --time=0:30:00
#SBATCH --nodes=1
#SBATCH --ntasks=2
#SBATCH --mem-per-cpu=1000M
unset SLURM_GTIDS # Odd, but necessary!
# module load ANSYS/<version>
# e.g.
module load ANSYS/2020R2
runwb2 -B -F Workbench_Taurus.wbpj -E 'Project.Update' -E 'Save(Overwrite=True)'
#or, if you wish to use a workbench replay file, replace the -E parameters with: -R mysteps.wbjn
Running Workbench in Parallel¶
Unfortunately, the number of CPU cores you wish to use cannot simply be given as a command line
parameter to your runwb2 call. Instead, you have to enter it into an XML file in your home
directory. This setting will then be used for all your runwb2 jobs. While it is also possible
to edit this setting via the Mechanical GUI, experience shows that this can be problematic via
X11-forwarding and we only managed to use the GUI properly via DCV, so we
recommend you simply edit the XML file directly with a text editor of your choice. It is located
under:
$HOME/.mw/Application Data/Ansys/v181/SolveHandlers.xml
(mind the space in there.) You might have to adjust the Ansys version
(here v181) in the path to your preferred version. In this file, you can find the parameter
<MaxNumberProcessors>2</MaxNumberProcessors>
that you can simply change to something like 16 or 24. For now, you should stay within single-node
boundaries, because multi-node calculations require additional parameters. The number you choose
should match your used --cpus-per-task parameter in your job file.
Running MAPDL¶
Ansys Parametric Design Language (APDL) is a powerful structured scripting language used to interact with the Ansys Mechanical solver. Mechanical APDL (MAPDL), a finite element analysis program, is driven by APDL. APDL and MAPDL can be used for many tasks, ranging from creating geometries for analysis to setting up sophisticated solver settings for highly complex analyses
Shared-Memory Mode¶
MAPDL can be invoked in so-called shared-memory mode to make use of threads in order to speedup computation. The multi-threading approach is restricted to one node. In contrast, MAPDL offers a MPI-parallel mode to distribute the computation across multiple nodes. This mode is described below.
Interactive mode¶
marie@login$ srun --nodes 1 --ntasks-per-node=4 --time=0:20:00 --mem-per-cpu=1700 --pty bash -l
marie@node$ module load ANSYS/2021R2
marie@node$ mapdl -smp -np ${SLURM_NTASKS}
Batch mode¶
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4
#SBATCH --mem=2000
#SBATCH --job-name=ansys_mapdl
#SBATCH --output=output_ansys_mapdl
#SBATCH --time=01:00:00
module load ANSYS/2021R2
# -smp use shared memory parallel version
# -b (batch mode)
# -np specify number of cpu's to use
# -j jobname
mapdl -smp -b -np ${SLURM_NTASKS} -j solution -i <input-file>
marie@login$ sbatch mapdl_job.sh
Distributed-Memory Mode¶
MAPDL can be run in distributed-memory mode using multiple compute nodes in either interactive as well as batch mode as shown in the following.
In both cases, it is necessary to create a nodelist and provide it to MAPDL via -machines command
line option.
Interactive Mode¶
marie@login$ srun --nodes 4 --ntasks-per-node=4 --time=0:20:00 --mem-per-cpu=1700 --pty bash -l
# generate node list
marie@node$ NODELIST=$(for node in $( scontrol show hostnames ${SLURM_JOB_NODELIST} | uniq ); do echo -n "${node}:${SLURM_NTASKS_PER_NODE}:"; done | sed 's/:$//')
marie@node$ KMP_AFFINITY=none mapdl -machines ${NODELIST}
Batch Mode¶
#!/bin/bash
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=4
#SBATCH --mem=2000
#SBATCH --job-name=ansys_mapdl
#SBATCH --output=output_ansys_mapdl
#SBATCH --time=01:00:00
module purge
module load ANSYS/2021R2
# generate node list
NODELIST=$(for node in $( scontrol show hostnames ${SLURM_JOB_NODELIST} | uniq ); do echo -n "${node}:${SLURM_NTASKS_PER_NODE}:"; done | sed 's/:$//')
# -b (batch mode)
# -machines xxx specify machines list for distributed Ansys
# -j jobname
setenv KMP_AFFINITY none
mapdl -b -machines ${NODELIST} -j solution -i <input-file>
marie@login$ sbatch mapdl_job.sh
COMSOL Multiphysics¶
COMSOL Multiphysics (formerly FEMLAB) is a finite element analysis, solver and Simulation software package for various physics and engineering applications, especially coupled phenomena, or multiphysics.
COMSOL may be used remotely on ZIH systems or locally on the desktop, using ZIH license server.
For using COMSOL on ZIH systems, we recommend the interactive client-server mode (see COMSOL manual).
Client-Server Mode¶
In this mode, COMSOL runs as server process on the ZIH system and as client process on your local workstation. The client process needs a dummy license for installation, but no license for normal work. Using this mode is almost undistinguishable from working with a local installation. It also works well with Windows clients. For this operation mode to work, you must build an SSH tunnel through the firewall of ZIH. For further information, please refer to the COMSOL manual.
Usage¶
Server Process
Start the server process with 2 cores, 4 GB RAM and max. 2 hours running time using an interactive Slurm job like this:
marie@login$ module load COMSOL
marie@login$ srun --ntasks=1 --cpus-per-task=2 --mem-per-cpu=4096 --time=02:00:00 --pty --x11=first server -np 2
Background Job
Interactive working is great for debugging and setting experiments up. But, if you have a huge workload, you should definitively rely on job files. I.e., you put the necessary steps to get the work done into scripts and submit these scripts to the batch system. These two steps are outlined:
- Create a job file, e.g.
#!/bin/bash #SBATCH --time=24:00:00 #SBATCH --nodes=2 #SBATCH --ntasks-per-node=2 #SBATCH --cpus-per-task=12 #SBATCH --mem-per-cpu=2500 module load COMSOL srun comsol -mpi=intel batch -inputfile ./MyInputFile.mph
Interactive Usage with X11 Forwarding¶
marie@login$ start_comsol.sh
If you'd like to work interactively using COMSOL, you can request for an interactive job with, e.g., 4 cores and 2500 MB RAM for 8 hours and X11 forwarding to open the COMSOL GUI:
marie@login$ module load COMSOL
marie@login$ srun --ntasks=1 --cpus-per-task=2 --mem-per-cpu=2500 --time=01:00:00 --pty --x11=first comsol -np 2
Note
Please make sure, that the option Preferences --> Graphics --> Renedering is set to software rendering. Than, you can work from within the campus network.
LS-DYNA¶
LS-DYNA is a general-purpose, implicit and explicit FEM software for
nonlinear structural analysis. Both, the shared memory version and the distributed memory version
(mpp) are installed on ZIH systems.
You need a job file (aka. batch script) to run the MPI version.
Minimal Job File
#!/bin/bash
#SBATCH --time=01:00:00 # walltime
#SBATCH --ntasks=16 # number of processor cores (i.e. tasks)
#SBATCH --mem-per-cpu=1900M # memory per CPU core
module load LS-DYNA
srun mpp-dyna i=neon_refined01_30ms.k memory=120000000
Submit the job file named job.sh to the batch system via
marie@login$ sbatch job.sh
Please refer to the section Slurm for further details and options on the batch system as well as monitoring commands.