Compile on Cray XC-40

Submitted by andrew.downs on Tue, 09/27/2016 - 08:38

Forums 

Installation

Hello,

Here's an arch.mk file I've cobbled together for our machine Excalibur at ARL. It's a Cray XC-40 and hopefully this will help other users, it's still in progress and I need to clean up some directories so it'll work more universally. So far I have:

# arch.mk for BerkeleyGW codes
#
# suitable for Excalibur at ARL
#
# Andrew Downs
#
# 2016, ARL
#
# Run the following command before compiling:
# module swap intel/16.0.0.109 intel/15.0.1.133
# module load cray-hdf5-parallel
# module load fftw/3.3.4.4
# module load cray-libsci

# Precompiler options

COMPFLAG = -DINTEL
PARAFLAG = -DMPI -DOMP
MATHFLAG = -DUSESCALAPACK -DUNPACKED -DUSEFFTW3 -DHDF5
# Only uncomment DEBUGFLAG if you need to develop/debug BerkeleyGW.
# The output will be much more verbose, and the code will slow down by ~20%.
#DEBUGFLAG = -DDEBUG

FCPP = /usr/bin/cpp -ansi
F90free = ftn -free -qopenmp
LINK = ftn -qopenmp
FOPTS = -O3 -xAVX -g
FNOOPTS = $(FOPTS)
MOD_OPT = -module
INCFLAG = -I

C_PARAFLAG = -DPARA -DMPICH_IGNORE_CXX_SEEK
CC_COMP = CC
C_COMP = cc
C_LINK = CC
C_OPTS = -O3 -xAVX -qopenmp
C_DEBUGFLAG =

REMOVE = /bin/rm -f

# Math Libraries
#
FFTWPATH = /opt/cray/fftw/3.3.4.4/x86_64
# link line for FFTW2 library. Sometimes needs to be -ldfftw
# Note: To use FFTW3 from MKL, do not link the fftw3xf interfaces.
FFTWLIB = -L$(FFTWPATH)/lib {-lfftw3_omp} -lfftw3
# FFTW3 requires include file fftw3.f03 instead.
FFTWINCLUDE = $(FFTWPATH)/include

HDF5_DIR = /opt/cray/hdf5-parallel/1.8.14/INTEL/14.0
HDF5_LDIR = $(HDF5_DIR)/lib
HDF5LIB = $(HDF5_LDIR)/libhdf5hl_fortran.a \
$(HDF5_LDIR)/libhdf5_hl.a \
$(HDF5_LDIR)/libhdf5_fortran.a \
$(HDF5_LDIR)/libhdf5.a -lz -ldl
HDF5INCLUDE = $(HDF5_DIR)/include

PERFORMANCE =

#LAPACKLIB = ${FFTWLIB}

TESTSCRIPT = qsub edison.scr

jornada's picture

Submitted by jornada on Wed, 09/28/2016 - 18:02

Many thanks, Andrew!

If you have some time in the future, can you also tell us if you managed to get the testsuite running? It was a bit tricky to run it on other cray systems such as Mira, because the login and compute nodes had different endianness. Hopefully this is not the issue for you..

Submitted by andrew.downs on Thu, 09/29/2016 - 08:07

You're welcome.

Different endianness? Wow...I hope that's not the case here, I don't think it is, but I hadn't even thought about that.

I'm running tests all this week and I'll let you know as soon as I have something to report, so far so good though.

I should clarify that I'm not running "the" testsuite yet, I don't really know how it works or how to pick it apart on the XC-40. What I am doing is working closely with a GW user and we're just hammering the system with his real-world input data and seeing how my build holds up.