Tips for Macintosh Users

This page has some useful tips and tricks for running IDL on Mac OS X systems. If you have tips to share, please send them to me, k-bowman@tamu.edu, and I'll try to get them up on this page. You can also visit the Resources page for links to other web sites with IDL and OS X information.

Running IDL on Mac OS X

On Macs you can use the IDL Development Environment (idlde) or you can run IDL in a standard terminal window and use your favorite text editor.  I prefer the latter method, so here are a few tips for configuring your IDL environment.

There are two main terminal options: use an xterm window under X11 (X11 must be running anyway to display IDL graphics) or use Apple's Terminal.app (the Terminal program in the /Applications/Utilities folder). I use Terminal.app because cut and paste work much better than is the case for xterm windows.

The version of X-Windows supplied by Apple is usually out of date, so be sure to download and install the latest version of XQuartz.

Opening invisible files and directories

The OS X Finder, which displays your directories (folders) and files, is normally configured to hide invisible files (. or dot files) and many Unix directories (such as/usr) from the user. This greatly simplifies things for typical home or office users, but can be an annoyance for technical users.

It is easy to open an invisible text file using BBEdit or TextWrangler, however. Simply choose the Open Hidden... command from the File menu. This opens the standard open file dialog box with the normally hidden files and directories made visible. Set the Enable menu to All Files. You can then see and open your.login file, for example, or navigate to /usr/local, which is normally invisible in the Finder.

A second approach is to make the files and directories visible in the Finder. To do this, enter the following command in a Terminal window

> defaults write com.apple.finder AppleShowAllFiles YES

This command sets a preference in the Finder preferences file, which is located in~/Library/Preferences/com.apple.finder.plist. You may have to wait a minute or two for the Finder to notice the preference change and update the Finder windows. To make dot files invisible once more, enter

> defaults write com.apple.finder AppleShowAllFiles NO

in a Terminal window. By the way, com.apple.finder.plist is an XML file. Double clicking on the file opens it in the Apple Property List Editorapplication.

The disadvantage of making all the invisible files visible in the Finder is, well, that all of the invisible files are visible. This is like being forced to use ls -a all the time. Therefore, I do not make invisible files visible in the Finder. I use the Open Hidden command in BBEdit instead.

To navigate to an invisible directory in the Finder without making all the invisible files visible, use the Go To Folder... command (cmd-shift-G) in the Finder's Gomenu. You can enter any path in the dialog box, including normally invisible paths such as /usr/local.

Getting a full file path using the Finder and Terminal

The OS X filesystem uses standard Unix syntax. To get the full path to a file, you can use a Terminal or xterm window and standard Unix methods to navigate to the file by using the cd command. Then use pwd to get the full path to that directory.  Here is another simple way to get file paths that uses the Finder with Terminal.

Have a Terminal window open.  (That's Terminal.app, not an xterm.)

  1. In the Finder, navigate to the file or directory that you want to use.
  2. Drag the file or directory into the Terminal window. Presto, the full path appears.

You can copy and paste that path if you need to use it elsewhere.  You can enter a Unix command like ls before dragging and dropping a folder to get a listing of what is in the directory.

bowman> ls /Volumes/csrpd1/1B01/V6/2004/200409/20040905/ 
1B01.040905.38795.6.HDF     1B01.040905.38796.6.HDF
1B01.040905.38797.6.HDF     1B01.040905.38798.6.HDF
1B01.040905.38799.6.HDF     1B01.040905.38800.6.HDF

When you want to open a file in IDL interactively, you can do the following. First type:

IDL> file = '

at the IDL prompt.  Drag and drop the file of interest into the terminal window. Delete the trailing blank that is automatically added. Add the final quote.  Hit return. The result will look something like this:

IDL> file = '/Volumes/csrpd1/PROGSCI/v4/502rf05/502rf05.nc'

Then open the file with the appropriate command (depending on file type), such as:

IDL> id = ncdf_open(file)

File names and case sensitivity

Mac OS X supports several different filesystems, including several flavors of the Mac "native" filesystem, HFS.  Because they support some handy OS X features, HFS volumes are preferred for most uses.  HFS volumes can be created to be either case sensitive or case insensitive.  To avoid some subtle problems, boot volumes should be case insensitive. Note that this is case insensitivity is not standard Unix behavior, so some caution is called for when you create new files.  I create non-bootable volumes, such as volumes for data storage, as case-sensitive for better consistency with our Unix/Linux file servers.

Disk volumes are initialized with the Disk Utility program, which, not surprisingly, is found in /Applications/Utilities. 

By the way, even case insensitive HFS volumes are case preserving, so you can have confusion when you have file names that differ only in case.

Syntax-coloring for BBEdit and TextWrangler

BBEdit and TextWrangler are full-featured, point-and-click, text editors for Macintosh computer systems from Barebones Software. Either program is an excellent IDL programming editor for Mac users who prefer the command line to the IDL Development Environment. The Barebones web site explains the differences between the free TextWrangler and the commercial BBEdit.  I use TextWrangler in our Mac lab to teach introductory IDL programming.

Click on the appropriate link below to download a zip file containing the IDL syntax-coloring plug-in for BBEdit and TextWrangler. This is an Apple preferences list (.plist) XML file. Many thanks to Will Powell of Macquarie University, Australia for putting this together.

Since TextWrangler is free, we recommend that TextWrangler users download the latest version of TextWrangler and use Version 2.0 of the plug-in.

The syntax-coloring plug-in adds basic syntax coloring to either BBEdit or TextWrangler. To use it, unzip the downloaded file and and put the file IDL Configuration.plist into ~/Library/Application Support/BBEdit/Language Modules/ or ~/Library/Application Support/TextWrangler/Language Modules/, depending on which program you have. You may also need to go into the Preferences panel for BBEdit or TextWrangler and do two things: turn on syntax coloring in the Editor Defaults pane and make sure that IDL is linked to .pro files in the Languages pane.

The plist file includes a list of built-in IDL routines and functions that is used to determine which words should be colored. Will Powell wrote a short IDL program to allow this list to be updated, get_routine_names.pro. The program generates a text file that can be copied and pasted into the plist file. This should only be necessary when IDL is updated and new system routines are added.

Version 2.0 of the plug-in will identify the start and end point of functions and procedures, so TextWrangler can display them in its popup menu and allow you to use the 'Go to Function Start' and 'Go to Function End' menu commands. This allows you to navigate long source files which contain multiple routines much more easily. When used with BBEdit, you can also do Code Folding - which hides routines from view while you're not interested in them.

The plug-in does not provide full, context-sensitive, syntax coloring, but it does highlight strings, comments, and built-in procedure and function names.

Creating PostScript and PDF files

The OS X user interface is based on Postscript and PDF, so OS X applications generally handle PS and PDF files well.  PostScript graphics files generated by IDL can be viewed in OS X simply by double clicking on the file.  OS X uses the pstopdf program to convert the files to PDF, which are then automatically opened in Apple's Preview utility. You can get more information about pstopdf  from the man pages.

> man pstopdf

There are several IDL procedures available on the web that handle many of the details of switching to the PS device and then switching back to the default OS X graphics system (X11).  Click here to download my versions of these files: PS_ON and PS_OFF, as well as PSTOPDF.  PSTOPDF is an IDL procedure that SPAWNs a pstopdf process to convert a PS file to PDF. These programs are based on original programs by Liam Gumley and Ben Tupper.

The following commands create a PS graphics file test.ps and convert it to test.pdf.

IDL> ps_on, filename = 'test.ps'
IDL> surface, dist(50)
IDL> ps_off
IDL> pstopdf, 'test.ps'

For the highest quality PostScript graphics, I can find no substitute for Adobe Illustrator. Illustrator can open and save files in Illustrator (.ai), PS (.ps), EPS (.eps), and PDF (.pdf) format. Simply drag an IDL PS file onto the Illustrator application icon to open it. Files opened in Illustrator must be one page only, although pstopdf will convert multipage PS files to multipage PDF files.

The built-in Preview app is also handy for extracting a single page from a multi-page PDF file, or for combining multiple PDF files into one file.  To do these operations, display the thumbnail views.  Click a page; then select Copy.  Creating a new document will automatically insert the copy page into the document.  To create a multi-page document, simply drag the PDF files into the thumbnail views on top of the existing document.

Making movies in OS X

IDL has built-in functions to create animation sequences using MPEG compression. Because MPEG compression was designed for movies (that is, photographs of the real world), it often does not work well with scientific graphics. Line graphics, for example, usually display compression artifacts. Also, because MPEG uses proprietary technology, the MPEG functions require an additional license. For information about obtaining a free MPEG license, see this Tech Tip.

There many external applications that can inexpensively make high-quality (lossless) QuickTime animation sequences under OS X.  This tip discusses four programs:  QuickTime Pro, GraphicConverter, ImageJ, and ImageMagick.

WARNING: QuickTime movies are usually portable between OS X and Windows (if QuickTime is installed on the Windows system). But, if you need to show your movie on a Windows systems (at a conference, for example), make sure you test things ahead of time. PowerPoint often does not play well with file formats not created by Microsoft. Apple's Keynote, on the other hand, handles QuickTime (and PDF) files seamlessly.

In all cases, the IDL part of the process is similar: create the individual images that will comprise the animation by using whatever IDL graphics you need. Save the images in a directory in sequence, using something like this:

WRITE_PNG, 'frame_' + STRING(i, FORMAT="(I4.4)") + '.png', TVRD(TRUE=1)

where i is the frame number.

IDL will write a variety of standard graphics image formats, but I prefer PNGs. PNG is a lossless image compression format that handles both 8- and 24-bit color images. The compression ratio is typically as good as GIFs or better, and the format is public domain. PNGs are interleaved across the first dimension, so TRUE=1 is necessary in the TVRD call.

Using QuickTime Pro

You can convert the frames into a movie by using Apple's QuickTime, but you have to pay Apple $30 to upgrade QuickTime to QuickTime Pro. This is probably the easiest interactive method. To use QuickTime Pro:

  1. Choose Open Image Sequence from the File menu.
  2. Navigate to the directory containing the frames and select the first frame.
  3. Click Open
  4. Set the frame rate in the dialog box and click OK.
  5. Save the resulting movie. Generally you will want to save the movie as "self-contained".

Using GraphicConverter

GraphicConverter is an outstanding shareware program ($30) from Lemke Software that will convert between a huge variety of graphic formats. GraphicConverter will give you the most options for input and output. To convert your images into a QuickTime movie:

  1. Choose Convert & Modify from the File menu.
  2. On the left side, navigate to the directory containing the frames.
  3. Select all the frames that you want in the movie. You can click on one frame and then use cmd-A to select all of the files in the directory.
  4. On the right side, navigate to the directory where you want the movie to go. Set the Dest. Format to QuickTime (.mov).
  5. Click the Options button and select the PNG compressor.
  6. Set the frame rate desired. You may need to experiment with this to get the playback rate that works for your animation.
  7. Turn off Key frames, if set; and set the Depth to 256 colors (8-bit) or Millions of Colors (24-bit), depending on how you created the frames.
  8. Click OK to get back to the Convert & Modify dialog.
  9. Click Go.
  10. At the top of the dialog box select "Make one movie from all source pictures".
  11. Give your movie a name. The default file suffix is .mov.
  12. Check the "Use dimension of first image" box to set the size of the output movie.
  13. Check the "Flatten Movie" box near the bottom.
  14. Click Go. You may need to wait after if finishes reading the files for the QuickTime movie to be created depending on the number and size of the frames.
  15. Double click the movie file to play in in QuickTime.

There are two movies on this page that were created using IDL and GraphicConverter.

Last, don't forget to pay the shareware fee for GraphicConverter. It is excellent software and continued development depends on user support.

Using ImageJ

Ben Tupper sent the following macro to use the free NIH ImageJ program to make an animation:

"ImageJ has functionality to import a sequence of images and then save to QT format. The quality looks pretty good to my eye. Here is a simple ImageJ macro for doing so..."

// ImageJ macro
//
// Use this macro to import a series of images and convert them to a
// QuickTime movie format
// must have an up to date version of ImageJ requires("1.37")
// this first step invokes two dialogs...
// the first asks the user to locate the first image in the sequence
// to open the second asks the user for increment, filename filters, etc.
run("Image Sequence...");
// this second step also invokes two dialogs
// one to locate the destination filename (for the newly created movie)
// one to ask the user to specify the quality and frame rate of the output 
run("QuickTime Movie...");
// it is that easy!

Using ImageMagick

Carsten Lechte suggests using the free software ImageMagick to make GIF animations:

"The point I would like to make is that it may be worthwhile to use EPS files as the basis for the movie. I needed nice-looking EPS versions of some of the plots for my paper anyway, and I did not want to have to tweak my plotting routines for two kinds of output format. EPS seemed the most versatile. Of course, this costs another conversion step in the movie generation."

convert -adjoin -delay 10 lots_of_plots*.eps result_anim.gif

Creating IDL graphics without a display

Using the Z-Buffer

Beginning with version 6.4, IDL supports a full 24-bit Z-buffer device.  (Previous versions had been 8-bit only.)  The Z-buffer is a ‘virtual’ device that is handled internally by IDL, so no graphics hardware is required.  This makes the Z-buffer very useful for creating graphics in the background, via automated scripts on a ‘headless’ server, for example.  Using the Z-buffer is generally easier than using the X-Windows virtual frame buffer (Xvfb, discussed below).

To configure the Z-buffer device , you do something like this

SET_PLOT, 'Z'
DEVICE, SET_PIXEL_DEPTH    = 24, $
        SET_RESOLUTION     = [1024, 768], $
        SET_CHARACTER_SIZE = [6, 10], $
        Z_BUFFERING        = 0

These commands switch the graphics device to the Z-buffer, set the pixel depth to 24-bit, create a 1024 x 768 graphics display, set the character size to match the X-Windows device (so that graphics are scaled the same), and turn off the three-dimensional Z-buffering (this is useful for 2-D plotting, as it avoids some possible 3-D rendering problems).  If you want to use the Z-buffer for 3-D rendering, omit the Z_BUFFERING keyword.  To make the graphics scaling match the WIN device, set the character size to [7, 10].

Note:  the Z-buffer is a direct graphics device, so you cannot use it for object graphics, such as the iTools.

After you have drawn your graphic, you can save the bitmap image to a file (PNG, for example) like this

WRITE_PNG, ‘myfile.png’, TVRD(TRUE = 1)

The Z-buffer device is not a windowing device, so you cannot use the WINDOW command.  (The Z-buffer is the window.)  Also, it does not support hardware fonts, but you can use TrueType fonts.

If you are plotting maps with MAP_SET and using the SCALE keyword, it is important to know that IDL assumes different pixel sizes for the X, Z, and WIN devices.  You will have to adjust your SCALE value for each device (by the ratio of the pixel sizes) if you want the maps to appear the same relative size.  The pixels per cm for the X, Z, and WIN devices are 40, 26, and 32, respectively.

Using X-Windows

Because Mac OS X is a version of Unix, it uses X-Windows to display graphics on the terminal screen. (The late, lamented Mac native version of IDL did not make the transition from Mac OS 9.) Sometimes, however, it is necessary to run IDL and produce graphics without having an X-Windows display attached. For example, you may want to automatically run IDL on your server to create graphics for a web page. Servers often have no display attached and are managed via various kinds of remote access software, such as Apple's Remote Desktop or the public domain VNC.

Fortunately, recent versions of X-Windows include software (Xvfb) to emulate a computer display, allowing graphics to be drawn completely in memory. Xfvb stands for "X virtual frame buffer", and it functions just like a hardware graphics display, although it lacks the hardware graphics acceleration provided by a real graphics card. The technique shown below should work with IDL on any flavor of Unix that includes Xvfb.

Here is a t-shell script that demonstrates how to start Xvfb, run an IDL program, and exit. In the process it demonstrates some basic shell scripting techniques, including how to pass arguments from the shell into an IDL procedure.  This approach can be used to generate IDL graphics completely unattended on a system with or without a display.

#!/bin/tcsh
# Run IDL program to create GFS forecast maps.
#
#      Usage:  gfs_make_maps hour
#
if (${#argv} == 0) then            #If no analysis hour is provided
   set HOUR=00                     #Default forecast start time
else
   set HOUR=$1                     #Use analysis hour provided
endif
set YEAR=`date "+%Y"`              #Default year is current year
set MONTH=`date "+%m"`             #Default month is current year
set DAY=`date "+%d"`               #Default day is current year
echo"*************************************************************"
echo "Start time    : " `date "+%Y-%m-%d %H:%M:%S"`
echo ""
echo Report of GFS forecast map creation.
echo Script is ~/bin/gfs_make_maps
echo Executing GFS_MAKE_MAPS
source /Applications/rsi/idl/bin/idl_setup   #Set up IDL paths
#Start X-windows server in background
/usr/X11R6/bin/Xvfb :1 -screen 0 1280x1024x24 -ac -terminate &
setenv DISPLAY :1.0                          #Set X-windows display
#Start IDL and run program
(echo GFS_MAKE_MAPS, MAKE_DATE\(${YEAR},${MONTH},${DAY},${HOUR}\) ; \
echo exit)                    | \
idl >>& ~/logs/gfs_make_maps.log
killall -9 Xvfb                              #Kill X-windows server
echo ""
echo "End time        : " `date "+%Y-%m-%d %H:%M:%S"`

I named the script gfs_make_maps to match the name of the IDL procedure. I keep my personal shell scripts in ~/bin. To run the script from the command line you would simply type

> ~/bin/gfs_make_maps >~/logs/gfs_make_maps.log

The IDL program GFS_MAKE_MAPS reads some meteorological data and plots maps. To run, it requires the year, month, day, and hour. The first part of the script gets the current value of year, month, and day using the Unix date command. The hour is read from the command line. If omitted, as in this case, the default value is 00. The output from the command is written to the file ~/logs/gfs_make_maps.log, although this is optional.

Next, the script prints some information about its execution with the echocommand.

The next few statements set up the IDL environment by running the idl_setupscript. This is necessary if the main script is run as a cron job, because cronjobs do not automatically run the user's shell startup scripts. (See below for an example of how to use cron to schedule jobs to run at specific times.)

Next, the script uses the echo command to prepare the IDL commands to be executed. The extra braces, dollar signs, and backslashes are part of the shell script syntax. In this case the year, month, day, and hour are used to create a date structure with an IDL function called MAKE_DATE, which is then passed into theGFS_MAKE_MAPS procedure. The output of the echo commands are piped into IDL using a Unix pipe |. This functions exactly as if the commands had been typed at the command line prompt on an interactive terminal. Any IDL output is appended to the log file ~/logs/gfs_make_maps.log, which can be very helpful for debugging.

When the IDL job finishes and exits, the script kills the Xvfb process, prints a completion message, and exits.

To run this script automatically at 14:15 local time every day to process the 12Z files, you would add a line like this to your crontab file.

15 14 * * * ~/bin/gfs_make_maps 12 > ~/logs/gfs_make_maps.log  2>&1

The log file will be erased and created anew each day.

Current weather screensaver or how to automatically download files

This tip is not directly related to IDL, but demonstrates how to use some standard Unix commands to download image files automatically from a web site.  The built-in Apple screensaver can be set to display the images automatically when the screensaver switches on. 

There are many Mac programs that will display current weather data, but you may wish to set up your own system for downloading weather maps and images.  These methods will be familiar to many Unix users, but Mac users may not be aware of some of the Unix tricks possible with Mac OS X.

The shell script below uses the command line utility curl to download NWS radar images, which are created in GIF format. The script prefixes the file names with numbers so that they will show in a regular order: first the national image (latest.gif), then regional maps from west to east across the northern tier of states, then regional maps across the southern tier. Here is a listing of the script:

#!/bin/sh
# Get selected radar images from the NWS web site using curl
#
#get_radar
#
#Base NWS URL for NEXRAD imagery
BASE_URL=http://radar.weather.gov/ridge/Conus/RadarImg
#Output directory
BASE_OUT=~/wx_images
#Create output directory, if needed
mkdir -p ${BASE_OUT}
curl -s -o ${BASE_OUT}/01_latest.gif       ${BASE_URL}/latest.gif
curl -s -o ${BASE_OUT}/02_pacnorthwest.gif ${BASE_URL}/pacnorthwest.gif
curl -s -o ${BASE_OUT}/03_northrockies.gif ${BASE_URL}/northrockies.gif
curl -s -o ${BASE_OUT}/04_uppermissvly.gif ${BASE_URL}/uppermissvly.gif
curl -s -o ${BASE_OUT}/05_centgrtlakes.gif ${BASE_URL}/centgrtlakes.gif
curl -s -o ${BASE_OUT}/06_northeast.gif    ${BASE_URL}/northeast.gif
curl -s -o ${BASE_OUT}/07_pacsouthwest.gif ${BASE_URL}/pacsouthwest.gif
curl -s -o ${BASE_OUT}/08_southrockies.gif ${BASE_URL}/southrockies.gif
curl -s -o ${BASE_OUT}/09_southplains.gif  ${BASE_URL}/southplains.gif
curl -s -o ${BASE_OUT}/10_southmissvly.gif ${BASE_URL}/southmissvly.gif
curl -s -o ${BASE_OUT}/11_southeast.gif    ${BASE_URL}/southeast.gif

I recommend that you create a directory (folder) called bin in your home directory. Copy the lines above and save them in a file named get_radar.sh in the bin folder. The GIF files are saved to the directory wx_images in the user's home directory.  If this directory does not already exist, it is created by the script.  You can use a text editor to change the output directory, specified by the BASE_OUT variable, if you prefer.  Finally, use the Terminal application to make get_radar anexecutable file, as follows

> cd ~/bin
> chmod u+x get_radar.sh
> ls -l get_radar
-rwxr--r--   1 bowman  bowman  1094 Nov  4 22:05 get_radar.sh*

You can test the program by running it as follows

> ./get_radar

This should download the GIF files listed in the script to the directory you specified.

To run this script automatically at regular intervals, you can use the cron command.  Copy the following lines and save them in a file called crontab.txt.

#       Download NWS radar images every 30 minutes
        00 * * * * ~/bin/get_radar
        30 * * * * ~/bin/get_radar

The first line is a comment. The numbers and asterisks at the beginning of the other two lines indicate that the get_radar command should be run at 00 and 30 minutes past the hour, every hour of every day. You can put this file in the bin directory also. The crontab command is used to read the crontab file.

> cd
> crontab bin/crontab.txt

You can check whether the crontab is loaded like this (that is a lowercase L, not a one)

> crontab -l
#       Download NWS radar images every 30 minutes
         00 * * * * ~/bin/get_radar
         30 * * * * ~/bin/get_radar

To remove the crontab and shut off the automatic download, enter

> crontab -r

Finally, to have the images displayed by the screensaver, choose System Preferences from the Apple menu. Then select the Desktop and Screensaver panel. From the scrolling list, pick Choose folder..., and then select the folder in which you are saving the GIF files. You can test your setup with the Test button.

Additional curl commands can be added to the get_radar script to download other files, such as GOES satellite images.