Index of /examples/idl

	Name	Last modified	Size	Description
	Parent Directory		-
	qsub_idl.qsub	2017-09-13 11:18	1.2K
	my_idl_test_args.pro	2017-09-13 11:04	421
	my_idl_test.pro	2017-09-13 10:57	351
	idl_batch_args.pro	2017-09-13 11:02	639
	idl_batch.pro	2017-09-13 10:05	564

Running IDL (and ENVI) on the SCC

Introduction

IDL, short for Interactive Data Language, is a programming and development environment used for data analysis, especially in astronomy, remote sensing, earth sciences, and medical imaging. IDL and its companion image processing system ENVI are both available on the SCC via the module system.

IDL and ENVI are installed as part of the same release. The versions available on the SCC as this is written (April 2021) are ENVI 5.5.2 and IDL 8.7.2. They can be loaded using the following commands. The ENVI module also provides access to the IDL interpreter so only one of the modules is required for using either IDL or ENVI with IDL. The module avail command can also be used to see what versions are available.

module load idl/8.7.2
module load envi/5.5.2
module avail idl
module avail envi

Running IDL and ENVI

The IDL interpreter can be run as shown with the idl command:

[user@scc2 idl]$ module load idl/8.7.2 IDL 8.7.2 (linux x86_64 m64). (c) 2019, Harris Geospatial Solutions, Inc. Licensed for use by: Boston University (Main) License: MNT-5522602 https://harrisgeospatial.flexnetoperations.com IDL>

[user@scc2 idl]$ module load envi/5.5.2   # in a terminal window within the OnDemand desktop
IDL 8.7.2 (linux x86_64 m64).
(c) 2019, Harris Geospatial Solutions, Inc.

Licensed for use by: Boston University (Main)
License: MNT-5522602
https://harrisgeospatial.flexnetoperations.com
 
% Restored file: ENVI.
% Loaded DLM: PNG.
% Loaded DLM: HPGRAPHICS.
% Loaded DLM: IDLJSON.
% Loaded DLM: XML.
ENVI> 

[user@scc2 idl]$ module load envi/5.5.2
IDL 8.7.2 (linux x86_64 m64).
(c) 2019, Harris Geospatial Solutions, Inc.

Licensed for use by: Boston University (Main)
License: MNT-5522602
A new version is available: IDL 8.8
https://harrisgeospatial.flexnetoperations.com
 
% Restored file: ENVI.
% Loaded DLM: PNG.
% Restored file: ENVI_M01.
% Restored file: ENVI_M02.
% Restored file: ENVI_M03.
% Restored file: ENVI_M04.
% Restored file: ENVI_M05.
% Restored file: ENVI_M06.
% Restored file: ENVI_M07.
% Restored file: ENVI_M08.
% Restored file: ENVI_D01.
% Restored file: ENVI_D02.
% Restored file: ENVI_D03.
% Restored file: ENVI_CW.
% Restored file: ENVI_IDL.
% Restored file: ENVI_IOU.
% Restored file: ENVI_RV.
% Loaded DLM: IDLJSON.
ENVI> 

IDL Batch Jobs on the SCC

IDL programs can be run on the SCC using the qsub batch system. In order to run an IDL program on a compute node an additional IDL batch program (not to be confused with the SCC batch) needs to be written. The IDL batch program is fed to the IDL interpreter and instructs the interpreter to run the actual desired program.

A description of the IDL batch syntax is provided by Harris Geospatial Solutions, the publisher of the IDL system. Briefly, an IDL batch is a .pro file that does not have the starting PRO or ending END commands, cannot contain any loops, and cannot make use of multi-line commands using the BEGIN and END keywords.

There are two examples provided here. The first consists of the IDL batch program idl_batch.pro which runs the simple example my_idl_test.pro. The second example consists of the IDL batch program idl_batch_args.pro which runs the example my_idl_test_args.pro. This example shows how to use runtime arguments by setting environment variables with the IDL batch program. A matching script for use with the SCC qsub command, qsub_idl.sh, is provided that runs both examples. To try them, download all 5 files to directory and submit them to the SCC with the command:

qsub qsub_idl.sh

Multiple Cores and IDL

Multiple cores will be automatically used by IDL if they are available and the size of the problem meets some heuristics.The heuristics are set so that the speedup by applying multiple cores is worth the overhead of splitting up work and collecting results. The official documentation on the IDL CPU structure contains information on how to manipulate these heuristics and how to set the number of cores used by IDL. The use of multiple cores is typically transparent to the user and does not require any changes to the program code. This is separate from the IDL_IDLBridge system that can be used to link multiple computers to share the computations for a single IDL program. If you are interested in using the IDL_IDLBridge capability please contact RCS for assistance.

By default IDL will attempt to use all of the available cores on a compute node. This must be limited to the number of cores requested for an SCC batch using the -pe omp N argument to qsub. Please refer the the RCS documentation on submitting jobs for more information on requesting multiple cores. When a job runs an environment variable, $NSLOTS, is automatically set that indicates the number of requested cores. The following IDL code will check for the existence of the $NSLOTS variable. If it's found it will set the IDL core limit (the value !CPU.TPOOL_NTHREADS) to match the value of $NSLOTS. If $NSLOTS is not set (which is the case when running IDL from a login node) the IDL core limit is set to 1. This code is used in the example IDL batch programs provided in this directory.

; Convert the NSLOTS environment variable to a number ; and assign it to limit the number of IDL cores. ; If NSLOTS is not defined then set the number ; of allowed cores to 1 so that IDL behaves on ; login nodes. IF STRLEN(GETENV('NSLOTS')) GT 0 THEN CPU, TPOOL_NTHREADS = LONG(GETENV('NSLOTS')) ELSE CPU, TPOOL_NTHREADS = 1