Recent Changes - Search:


SPIRE Interactive Analysis (SPIA)

A quick fix for HIPE 12.1 is available below.


The software package provided here ia a means to provide a structured GUI based access to the more intricate parts of the scan map photometer pipeline for SPIRE without the immediate need to resort to scripts. It consists of one Jython file containing a number of HIPE tasks that, once run, are registered with the system. These tasks cast the data analysis steps for SPIRE scan maps into logical modules allowing interactive access to many options. The package consists of basic custom taylored functions for IO, Level 1 and Level 2 processing of SPIRE photometer scan map datasets. In HIPE 12 the spectrometer task spiaFtsBackground was added. SPIA stands now rather for SPIRE Interactive Analysis (SPIA).

Video Tutorials

Using SPIA to retrieve an observation from the Herschel Science Archive (B. Schulz, Feb 2011, 8min)

This video tutorial shows how to use the spiaCopyHsa task of SPIA to download observations directly from the Herschel Science Archive into a local pool. It also shows how to set credentials to obtain access to proprietary data.

A demo of running a the standard data reduction task of SPIA with some different parameters (B. Schulz, Feb 2011, 26min)

This video tutorial shows how to reprocess a SPIRE scan-map observation with a new calibration context, re-derive Level 1 and Level 2 data, look at the results, and save them to a local pool.

Task List

Task NameCategoryDescription
spiaCalCopyHsaSPIREDeprecated! Use something like cal=spireCal(‘spire_cal_11_0′, saveTree=True) instead. Loads specified calibration context from HSA into session and saves it to local store as default calibration context.
spiaLoadCalSPIRELoads calibration context into session. Option to get newest calibration from own local store or potentially older calibration that came originally with the data.
spiaLoadUrnSPIRELoads observation context from local store identified by URN.
spiaLoadObsSPIRELoads SPIRE observation context from local store identified by obsid. Optional path to different local store directory can be given (somewhat outdated, use product browser instead).
spiaCopyHsaSPIRECopy observation from HSA to local store.
spiaLevel0_5SPIREReduce SPIRE photometer scan map data from Level 0 to Level 0.5 and run jump detectors and deglitchers without the “repair” option. Most parameters are GUI accessible. The masks can be inspected and corrected afterwards using the spireMaskEditorTool.
spiaLevel1RepairSPIREReduce SPIRE photometer data from Level 0.5 to Level 1, assuming that spiaLevel0_5 was used before to mask glitches and jumps in the Level 0.5 data. This module only “repairs” previously flagged glitches and completes processing to Level 1.
spiaLevel1SPIREReduce SPIRE photometer scan map data from Level 0 to Level 1. Many parameters especially of the deglitchers are GUI accessible. This reflects the standard pipeline processing if parameters are kept at their default settings. This module detects and repairs glitches in one go on Level 1 but doesn’t set the masks at Level 0.5 nor allows for intermediate inspection at Level 0.5. To inspect and potentially correct glitch and jump detections you can run the tasks spiaLevel0_5 and spia_Level1Repair instead.
spiaLevel2SPIREReduce SPIRE photometer scan map data from Level 1 to Level 2 maps. Most parameters are GUI accessible. Allows combination of up to 5 Level1 contexts into one map and does allow for different baseline removal algorithms.
spiaSaveObsSPIRESave observation to pool. Options for different path to local store directory and saving of Level 1 or Level 2 only stores.
spiaSaveMaps2FitsSPIRESave Level 2 maps to FITS files.
spiaConcatL1SPIREConcatenate selectable Level1 signal timelines from all building blocks into one table dataset that can be easily inspected with TableOverPlotter.
spiaPlotPositionSPIREPlots the Level 1 track of a detector over one of the Level 2 maps of an observation context. The context must contain both Level 1 and Level2 products.
spiaFtsBackgroundSPIREAllows different choices of background subtraction to improve the background removal for observations of faint sources.

The tasks appear in the Task View of HIPE and are typically invoked by double click so that they present their parameters in a GUI.


The SPIA User’s Manual describes how to use the package in more detail. For the impatient we provide here the sequence of tasks as they would be used in a typical scientific data reduction session. It is assumed that the user is familiar with the SPIRE Observer’s Manual and in particular the elements of the SPIRE photometer pipeline. The SPIA tasks provide help via tooltips through the GUIs, that are opened by hovering the mouse pointer over the respective parameter name. A diagram illustrating the relations between the tasks, the data stores and the observation context is shown below.

SPIA Workflow
Illustration of the workflow of the SPIA tasks when reducing SPIRE photometer scan map data.

A typical data reduction session using SPIA could look like this:

  1. Download an observation from the Herschel Science Archive (HSA) into your local pool using the task spiaCopyHsa.
  2. Load the observation context into your session using either the task spiaLoadObs, spiaLoadUrn, or the PAL Browser.
  3. Inspect the results of the HSA pipeline processing starting at the highest product level, i.e. use the map viewer in HIPE with the Level 2 products.
  4. Load the calibration product tree to be used for processing using the task spiaLoadCal.
  5. Open the task spiaLevel1 GUI, modify the input parameters as needed, and execute the task.
  6. Inspect the Level 1 products of the newly produced observation context using HIPE viewers like “Detector Timeline Viewer”, “Table Viewer”, or “SPIRE Mask Editor”.
  7. Repeat the previous step with other parameters if necessary, to replace the Level 1 products in the observation context just created or yet to create another copy of the observation context as container for this version of Level 1 products.
  8. Open the task spiaLevel2 GUI, modify the input parameters as needed, and execute the task.
  9. Inspect the maps in the newly created Level 2 context and re-run the last step with updated parameters as necessary.
  10. Save the entire observation or optionally only specific levels using the task spiaSaveObs.
  11. Save the Level 2 maps as FITS files in the directory structure of your computer using task “spiaSaveMaps2Fits”.

The package was demonstrated at the ADASS 2010 in Boston (Poster). A preprint of the article can be found at

A more in-depth description of the destriper module, which is supported by the “spiaLevel2″ task of SPIA, can be found on the destriper homepage, which also includes a step-by-step test procedure.


Note: For HIPE versions 12 or greater no separate installation of SPIA is required anymore!

For HIPE versions below 12 it is suggested to download the latest version of the list below (top of the list). There are two options to run SPIA, 1) using a Jython script or 2) installing a HIPE plugin.

About the Jython script:

The Jython file just needs to be run in HIPE once, pressing the green double arrow button. This will translate the code and register all tasks with the HIPE task list. The tasks are usually started from the task view of HIPE by double mouse click, and present their input parameters in a GUI for inspection and modification. However, the tasks can also be included in Jython processing scripts. When executed from a GUI the equivalent Jython command line appears on the HIPE log view for guidance.

For installation of the plugin:

  1. Open HIPE and under menu point “Tools” the entry “Plugins”.
  2. Click “Install new plug-in” in the new window.
  3. Either supply the URL of the plugin (for version 1.11.1 it would be or download the plugin and use the file selector dialog to provide its location on disk.
  4. Restart HIPE and check the Task view for tasks where the name starts with “spia_”
  5. From now on all SPIA tasks will be permanently available in your HIPE installation without running any script.
HIPE 12.113-Jul-2014spire_ia_tools_13.23.17.jarA bug that renders two deglitchers in the task spiaLevel1 unusable was found only recently. We have prepared a quick fix that will only work on a standard HIPE 12.1 installation. Please download the file spire_ia_tools_13.23.17.jar and follow the instructions in the README.txt file.
HIPE 1221-Mar-2014N/AFrom HIPE 12 onwards the SPIA tasks are an integral part of the system and no installation of a plug-in is required anymore.
1.11.1 plugin11-Sep-2013spia_1.11.1.jarVersion 1.11.1 will work with HIPE 10, and 11. SPIA will be part of HIPE starting with version 12. This intermediate version fixes a bug that prevented the cooler burp correction to work although the detection worked fine. A few outdated tooltips were updated as well.
1.11 plugin22-Aug-2013spia_1.11.jarVersion 1.11 will work with HIPE 10, 11 and 12. This version was made to cover the changes between HIPE 10 an HIPE 11. The BaselineRemoval was adapted to use the new parameter name outlierThreshold and the code was changed internally to rather use an argument dictionary. From now on the version of the calibration tree is recorded in Level 1 products and the observation context. The new Cooler Burp detection was added to Level 1 processing. The task spiaLoadObs() was improved to allow a search for substrings of tags and one can now select by index if several observations matched the criteria. The GUI layout was also changed a little and obsids are now printed in hex and decimal. For the zeropoint correction the default HFI gain correction factors were updated to fit the new ESA4 Neptune model flux calibration together with the HFI DX9 release. The factors are 0.965 and 1.000 for PMW and PLW respectively. Three new jump detector parameters, DestJumpDet, DestJumpThresh, DestJumpIter were included into the destriper. The task spiaCalCopyHsa() was removed because the functionality is now fully covered by the spireCal() task. In case the observation context is not copied, it is now ensured that all to-be-produced levels are removed to avoid feedback effects. In particular the double application of SSO proper motion correction is prevented that way. A bug was fixed that occurred when combining several observations with destriping, using the diagnostic products as start parameters. In this case a crash occurred before the fix, because the different diagnostic tables had not been combined. All calls to JTask were replaced by Task for HIPE V11 an higher to ensure HIPE 12 compatibility. An unregister task statement was added at the beginning of and to support the recompilation of tasks during the session for debugging purposes.
1.10.2 plugin20-Feb-2013spia_1.10.2.jarVersion 1.10.2 will work with HIPE 7,8,9,10, and 11. This version updates a deprecated parameter in the baselineRemovalMedian and baselineRemovalPolynomial tasks that is needed from HIPE 11.0.1674 on.
1.10.1 plugin12-Dec-2012spia_1.10.1.jarVersion 1.10.1 will work with HIPE 7,8,9,and 10. This version contains two bug fixes: 1) The version number has been corrected. 2) Some left-over variables from the plug-in initialization were removed.
1.10 plugin07-Dec-2012spia_1.10.jarVersion 1.10 will work with HIPE 7,8,9,and 10. This version was made to cover the substantial changes between HIPE 9 an HIPE 10, in particular the change of names of the Level 2 products and the re-definition of the Level 1 products. In Level 1 processing with HIPE 10 the cutting of timelines was moved before the association of sky pointing. This speeds up processing. The version of the calibration tree used is now recorded in the metadata. A bug was fixed that affected Level 0.5 processing. If Level 0.5 reprocessing is activated and the calibration tree is incomplete, i.e. it doesn’t have the static part saved, it is automatically updated. The internal code was cleaned up using parameter dictionaries in calls to more complex pipeline tasks, especially when there were API changes between HIPE versions. In Level 2 processing product names were updated while keeping SPIA compatible with the old names as well. The browse image generation code was simplified as well. In HIPE 10 the SSO proper motion correction has been implemented as a single HIPE task. This is now reflected in the SPIA code as well. This correction remains available in SPIA also with HIPE 9, although in this case it is implemented as a SPIA script. At Level 2 the ssoMode parameter was removed for HIPE 10 as now SSO proper motion corrected maps are generated by default as a separate Level 2 product. The new ssoMaps parameter allows to switch the generation of these products off. Since Level 1 was re-defined as being offset corrected, the default for parameter updateLevel1 was set to Yes. Please note that strictly speaking the tasks spiaLevel1 and spiaLevel1Repair don’t produce a Level1 product anymore but rather something that could be named Level 0.7. For practical reasons the SPIA tasks have not been renamed, but the removal of the offsets by the destriper only happens in the spiaLevel2 task. HIPE 10 will provide Planck based zero-point correction. Although the two Planck-HFI all-sky maps that are needed are not publicly available yet, SPIA is already prepared to do this processing step as well, once they become available. The parameter extendedMaps was added to allow disabling this processing step, which will be suppressed anyway if the necessary HIPE properties are not defined. The new parameters hybridMaps and hybridThreshold have been added for hybrid error map generation. If the number of readouts in a skybin is below the threshold, the error is calculated theoretically from the white noise levels of the contributing detectors rather than from the staistical distribution of the measured signals. Among other small improvements the parameter array in the task SpiaPlotPosition was changed to a pull-down menu.
1.9 plugin08-Aug-2012spia_1.9.jarThis version shows most changes in the Level 2 processing, however two changes were applied also to the Level 1 processing: 1) The electrical crosstalk correction module now appears before the jump detection module in the spiaLevel05 and spiaLevel1 tasks to stay in line with the updated HIPE 9 pipeline. 2) For HIPE V9 and higher the parameter names “reconstructionPointsBefore” and “reconstructionPointsAfter” were replaced by “reconPointsBefore” and “reconPointsAfter” respectively to stay compatible with changes to the wavelet deglitcher API. However, the API of the spia task didn’t change. In the task spiaLevel2 the default for “CopyObs” has been changed to “Yes”. Also the destriper was set to “active” by default and the baseline remover was set to “inactive”.The SSO proper motion correction received the options “Origin” and “Position” that set the center coordinate of the corrected map to either [0,0] or to the position of the SSO when the observation began. The default is “Position”. A check for presence of the auxiliary context in the observation was added to allow for a more graceful handling of the absence condition. The default mapper was set to “Destriper” to be in line with the updated HIPE 9 pipeline. The new scan speed limiting parameters minVel and maxVel, that the new naive mapper now offers were added to the spiaLevel2 task. Another new HIPE 9 parameter “noiseMethod” was added to the map maker tab of the task, allowing to choose between different methods to make the error map. This version of SPIA is meant to work with HIPE V7 through V10.
1.8 plugin20-Apr-2012spia_1.8.jarThis version contains a larger number of useful upgrades and is specifically compatible with the new HIPE 9 distribution. Due to a number of API changes in HIPE 9 the older SPIA version 1.7 is not anymore set to work with HIPE 9. Besides a few bug fixes, the processing up to Level 1 experienced the following changes: The signal jump detection module will now be automatically disabled for bright mode observations. The default for electrical cross talk correction was changed to “Active” anticpating a new calibration product for HIPE 9. The default for the concurrent glitch deglitcher was set to “Active” to be in line with the standard pipeline. The default of the parameter “optionReconstruction” in the wavelet deglitcher was set to the new value “linearAdaptive20″, and the default of parameter “reconstructionPointsAfter” was set to 3 for the same reason. Many changes in the task “SpiaLevel2″ are due to substantial changes in the destriper. The destriper part now supports threads, two different types of Level 2 deglitching, control over the number of iterations of Level 2 deglitching, control over use of the temporary storage, and feedback of old diagnostic products as start parameters. A list of observation contexts can now be marked with the mouse and dragged into a new input parameter “obsList” that allows to combine more than just 5 observations into one map (from HIPE 8 on). A new option “updateLevel1″ allows to replace the old Level 1 context by a new one that includes all changes like background subtraction etc, before the map is made. The Level2 task now also features correction for Solar System Object proper motion, and overall position correction for maps. This SPIA version brings a number of system improvements: The implementation and propagation of history information was substantially improved. Header data is now copied more completely, and history contexts are remembered from prior processing steps. The metadata creator item now includes the version and build numbers of both, SPIA and HIPE. All major processing tasks received a switch to control whether the temporary storage is used, allowing either for faster processing or for larger observations to fit into available memory. This version is meant to work with HIPE V7 through V9.
1.7 plugin30-Jan-2012spia_1.7.jarThis version contains a number of small updates and additional code to support the destriper upgrades that are being introduced for the future HIPE 9. A problem was fixed that prevented older Level 1 products to be processed because of a wrong product type. New parameters for glitch reconstruction were added to the wavelet deglitcher. For consistency the wavelet deglitcher was allowed as glitch reconstructor in spiaLevel1Repair. Unfortunately only the linear reconstruction is supported in repair only mode of the wavelet deglitcher. The default for DestL2DeglitchRepeat in spiaLevel2 was set to 0 for HIPE 8, because of issues with the Level2 deglitcher in the destriper that apply only to HIPE 8. For HIPE 9 a new destriper parameter DestL2DeglitchAlgorithm was added. This version should work with HIPE V7 through V8 (for HIPE 9 use SPIA 1.8).
1.6 plugin10-Nov-2011spia_1.6.jarThis version represents a major refactoring of the code that became unavoidable to ensure maintainability. However there were also a few upgrades. The task SpiaCopyHsa allows now exclusion of sub-contexts to save download time. The task SpiaLevel2 was changed to support the improved destriper, the new baseline remover, and the new extended source correction factor module. The new features in Level 2 processing will only work with HIPE 8, while everything else will continue to support HIPE V7 and V8.
1.5 plugin28-Jul-2011spia_1.5.jarThis version 1.5 supports the latest API of the destriper in HIPE 8. This is also the first SPIA that supports automatic version control and update service as offered by the plugin mechanism. This version comes only as a plugin and is supposed to work with HIPE V7, and V8.
1.4 plugin27-May-2011spia_1.4.jarThis version 1.4 is the first that was internally reorganized into a multi file distribution that is easier to maintain. It contains fixes to maintain compatibility with the new Jython 2.5 that will appear in HIPE 8. Further updates include accomodation of changes in the jump detector, inclusion of the new destriper available in HIPE 7, and introduction of name tags when saving observations into pools. SPIA 1.4 has also regained its compatibility with jylaunch. This version comes only as plugin and is supposed to work work with HIPE V7, and V8. Backwards compatibility with HIPE V5 and V6 could not be maintained for this version.
1.3 plugin21-Mar-2011spia_1.3.jarThis is the plugin version of 1.3 to work with HIPE V5, V6, and V7. Version 1.3 contains, besides a bugfix, a new validator that prevents certain SPIA tasks to show up as applicable if the data comes from other instruments or other observing modes than the photometer. Another significant improvement is the grouping of menu parameters into tabs with associated tooltips, so the GUIs become more organized. This new feature however is only available from HIPE V7 onwards. The script will work with HIPE V4, V5, V6, and V7.
1.2 plugin18-Feb-2011spia_1.2.jarThis is the plugin version of 1.2 to work with HIPE V5 and 6. Version 1.2 in the task spiaPlotPosition received a bugfix and some updates. The bug appeared with maps that include turnaround data. spiaLoadCal has an additional parameter now where the calibration pool from where to load can be specified. The creator is now set correctly in observation context and level 1 and 2 contexts. There were major changes to spiaLevel2: the useRemoveBaseline was replaced by removeBaselineType, the polynomial baseline removal was cleaned up and removal of a polynomial baseline per scan was added. The script will work with HIPE V4, V5 and V6.
1.1 plugin27-Jan-2011spia_1.1.jarThis is the plugin version of 1.1 to work with HIPE V5 and 6. 1.1 brings an overhauled “spiaPlotPosition” task, that works now correctly with turnaround data, places annotations correctly, indicates scan start positions and loci of thermistor jumps. The tasks “spiaLevel0_5″ and “spiaLevel1″ will now allow to update the calibration tree of the observation context. Containing the calibration tree that was actually used for processing the observation context is more consistent now. The defaults for copying the observation tree were adapted to common practises, i.e. the copyObs parameter was set to “No” for the tasks “spiaLevel1Repair” and “spiaLevel2″. Last but not least, all tasks were given a task description that appears in HIPE V6 as a tool-tip in the task view of HIPE. The script will work with HIPE V4, V5 and V6.
1.0 plugin06-Jan-2011spia_1.0.jarThis is the plugin version of 1.0 to work with HIPE V5.0. 1.0 contains several small changes and a bug fix. The most notable change is in the SpiaSaveObs task, that allows now to save partial observation contexts including only Level 0.5 and the auxiliary products needed for reprocessing from this level. The crosstalk correction was moved behind the jump detection to stay consistent with the official pipeline script. The script will work with HIPE V5 and HIPE V4 and disable the calls to the jump detector when it isn’t available in V4.
0.11 plugin04-Nov-2010spia_0.11.jarThis is the plugin version of 0.11 to work with HIPE CIB 5.0 655 and above. 0.11 has just a few fixes/updates. The default for running the wavelet deglicher a second time was set to “inactive”. The default values for the sigma kappa deglitcher were set according to the newest found optima and an obsolete filter type selection was removed. A fix was applied for compatibility with MS-Windows, although an issue still remains with the plug-in version.
0.10 plugin07-Oct-2010spia_0.10.jarThis is the plugin version of 0.10 to work with HIPE CIB 5.0 655 and above. Version 0.10 has three new tasks. spiaCalCopyHsa which replaces cal_import and is used to import a calibration context from the HSA into a local pool, spiaLevel0_5 that will run jump detector and deglitchers on Level 0.5 data but only set mask bits and save these back into Level 0.5, and spiaLevel1Repair. Masks can be inspected and - if necessary - corrected, at Level 0.5 with the SpireMaskEditorTask without having to deal with changed data yet. The other new task spiaLevel1Repair is supposed to run on such Level 0.5 data with masks for glitches and jumps set. It will only repair the gliches using the glitch remover of the Sigma Kappa deglitcher and process the data to Level 1 in the usual way. These last two new tasks, spiaLevel0_5 and spiaLevel1Repair, are an alternative to running the spiaLevel1 task. File selection dialogues were added to the parameters where files or directories are required. Apart from a few bug fixes, an option to re-run the wavelet deglitcher after the flux calibration was added too.
0.9 plugin28-Aug-2010spia_0.9.jarThis is the plugin version of 0.9 to work with HIPE CIB 5.0 655 and above. Version 0.9 comes with a new task spiaPlotPosition to overplot scan tracks of arbitrary detectors over the Level 2 map. The map display of spiaLevel2 was fitted with RA and DEC axes and a bug was fixed in the same task that occurred when processing from a Level1 only pool. Another improvement allows spiaLevel2 now to combine up to 5 observations into one map.
0.8 plugin02-Aug-2010spia_0.8.jarThis is the plugin version of 0.8 to work with HIPE CIB 5.0 655 and above. version includes two changes: 1) during mask generation display of the new maps can be switched off and 2) a new task spiaConcatL1 is introduced that concatenates Level 1 data from all building blocks into one table dataset for inspection in TablePlotter. This includes display of coordinates (track of detectors on the sky) and masks.
0.7 plugin28-Jul-2010spia_0.7.jarThis is a plugin and works only with HIPE CIB 5.0 655 and above. It contains a few more parameters and a fix for the mapmaker’s exclusion circle with SSOs version now works with HIPE V4 and beyond HIPE CIB 5.0 721. It contains a few more parameters and a fix for the mapmaker’s exclusion circle with SSOs. It is also available as a plugin that can be used with builds at and above CIB 5.0 655.
0.6 plugin spia_0.6.jarThe Version 0.6 packaged as a plugin by Paul Balm that is supported newly in the V5 build >= 655 development track. Installation instructions are below. Note that this plugin and also the script only work until build 721. For versions above a new version will be provided soon.
0.6 the global variables from cluttering the namespace and included baseline subtraction with avoidance circle.
0.5 of forgetting self in self.obs in spiaLevel1 and spiaSaveObs, and added optional browse product generation and map pixel sizes in spiaLevel2.
0.4 bug fix and additional parameters were applied to the wavelet deglitcher and the temperature drift correction in spiaLevel1
0.3 version has a cleaner and easier to read GUI with drop-down menus instead of tick boxes.
0.2 upgraded and debugged distribution version. This is HIPE V4 compatible. The comments tell which lines to comment out to make it compatible with HIPE V3.
0.1 for HIPE V4 (First distribution version)
0.1 for HIPE V3 (First distribution version)
Edit - History - Print - Recent Changes - Search
Page last modified on July 13, 2014, at 06:23 PM