Recent Changes - Search:


Destriper Module

Scan Map Destriper
Figure 1: Two maps of the Galactic Center region produced with standard pipeline processing (left) and with the HIPE 7 destriper. The destriper makes a particular difference in regions with structured non-uniform backgrounds.

1.  Module Description

1.1  Introduction

Due to the large telescope background, the fluxes measured by the SPIRE bolometers are very small differences on top of a large offset that is usually several orders of magnitude larger. As thermal and electronic stability are limited in a system like this, we observe residual offsets in the flux calibration from one detector to another, resulting in striping in reconstructed maps.

An easy way to remove these stripes is to determine from each detector timeline the median at Level 1 and then create a map. This works well as long as most of the readouts are seeing an overall flat background. In structured regions with cirrus clouds and gradients the method will produce less perfect results (see Fig. 1 and 5).

The destriper was developed to provide a generalized solution that should work independently of the map contents. It iteratively updates offsets until an optimal solution is found. There are also provisions for residual signal drifts and a Level 2 deglitcher is available as well in the newer version. The disadvantage is that it takes substantially longer to run than a simple median subtraction, so for large scale point source surveys it will not do better but take much longer. For structured regions however, like the galactic plane, such a tool is indispensable to create stripe free maps.

Other map makers that remove offsets as well are MadMAP, Scanamorphos, SanePic, and others. This destriper distinguishes itself from these as it uses a mathematically very simple method that in zero order does not make any assumptions about noise spectra or drifts in general, doesn’t change the signal, produces no FFT artefacts, and is available in Java within HIPE as a task that eventually may be suitable for standard pipeline processing. In essence it is probably most similar to Scanamorphos.

1.2  How does it work?

The destriper takes the flux timelines with positions of a list context (Level 1 context) as input. It will produce a preliminary naive map from this data and re-sample for each input timeline a map timeline at the same positions from the map. It will then fit an offset function that is a polynomial of user specified degree, to the difference between input timeline and map timeline. In the next iteration the difference between input timeline and offset function is used to construct the next map. For each timeline a Chi^2 is calculated as
Chi^2 = SUM_i[(mapTimeline_i - inputSignalTimeline_i + offsetFunction_i)^2] .
Each timeline is considered converged when the difference between the associated Chi^2 of two successive iterations sinks below a user selectable threshold. When all timelines have converged the iterations end, unless the maximum number of iterations is reached before. In addition the algorithm contains a suppression of bright sources, a Level 2 deglitcher, and a jump detector, that will be described in more detail later.


2.  Input Parameters

Figure 2: The default GUI for the destriper task in HIPE 10.

Figure 2 shows the default GUI for the destriper in HIPE 10. The parameters are described below.

HIPE 10 common parameters
level1objectThis is a list context or a Level 1 context containing all scans of one or more observations.
arraystringThis string selects the array that is to be processed; one of “PSW”, “PMW”, “PLW”. Only timelines of the selected detector array will be updated in the resulting list context. This paraneter is mandatory and has no default.
todobjectOptional time ordered data product to save the time to produce it. Note that this must be consistent with the pixelSize.
pixelSizedoubleSize of the skybins in arcsec. The default is set to 6, 10, 14 arcsec for PSW, PMW, PLW respectively depending on the chosen array. Note that the variable default is not displayed in the default GUI.
offsetFunctionstringThis parameter chooses between determining separate offset functions for each detector and building block (“perScan”) or one offset function per detector for the entire observation (“fullTimeline”). Note that if the list context contains scans of more than one observation, separate offset functions are calculated for different observations.
polyDegreeintegerDegree of polynomial that forms the offset functions. Default is 0.
kappadoubleNumber of standard deviations of the residual between signal, offset function and re-sampled map timeline, above which a Level 2 glitch is detected. Default is 5. There is still an issue with the HIPE 10 implementation that will be fixed in HIPE 11
iterThreshdoubleThreshold for Chi^2 to stop iterations. Default is 1.0E-10
l2DeglitchRepeatlongPeriod of iterations after which the Level 2 deglitcher is executed. This is a repetitive process. If set to zero the Level 2 deglitcher is never executed. If set to a larger value than the number of iterations necessary for convergence, the Level 2 deglitcher is executed once at the end. Default is 100.
l2DeglitchAlgorithmStringAlgorithm to use for Level2 deglitching. The default algorithm is “sigmaKappa”.
iterMaxlongMaximum number of destriper iterations that are allowed. Default is 100.
l2IterMaxlongThe maximum number of iterations that are allowed to do deglitching. The default is 5.
nThreadsIntegerIt defines how many threads the destriper will start. The default is one thread.
jumpThreshfloatThreshold for jump detection. Default is −1.
jumpIterlongPeriod of iterations after which the jump detector is executed. This is a repetitive process. If set to zero the Level 2 destriper is never executed. If set to a larger value than the number of iterations necessary for convergence, the jump detector is executed once at the end. Default is 100.
minVelfloatMinimum accepted scan speed. This parameter is handed through to the Naive Mapper. Readouts are not used if they are at a speed below the speed specified in arcsec/sec. Default is?.
maxVelfloatMaximum accepted scan speed. This parameter is handed through to the Naive Mapper. Readouts are not used if they are at a speed above the speed specified in arcsec/sec. Default is?.
methodobjectMap generation strategy This parameter is handed through to the Naive Mapper. This parameter defines specifics regarding how errors are weighted and how the error map is calculated.
startParametersobjectThis is a diagnostic product that is produced by a previous destriper run on the same dataset. Array, pixelSize, offsetFunction, and polyDegree must have the same values as in the previous iteration. The parameters found for the polynomial will be used as start parameters in the new destriper run. Changes applied to the vaules of column “Flags” will be taken into account. This way “bad” scans can be eliminated from the new destriper run.
chanNoiseobjectChannel noise calibration product. This parameter is handed through to the Naive Mapper. This is the calibration product found in the calibration tree at position .phot.chanRelGain, containing white noise levels of all SPIRE bolometers.
useOnlyMaskslongMask to change masking in Naive Mapper. This parameter is handed through to the Naive Mapper.
ignoreDefaultMaskslongMask to change masking in Naive Mapper. This parameter is handed through to the Naive Mapper.
withMedianCorrectedbooleanSwitch to decide if the first naive map should be made by median corrected scans. For the majority of all observations, it does not matter whether median corrected scans are used or not to make the first naive map. However, for parallel mode, we found that it gives better results if the median corrected scans. Default is “True”.
brightSourcebooleanSwitch to decide whether bright sources should be excluded. The default is off “True”.
fitWithWeightbooleanSwitch to decide whether to fit the offset function with weights. The default is “False”. The weighting algorithm as implemented does not work yet as of HIPE 10.
useSinkbooleanIf set to true, the ProductSink will be used in I/O. The default is “False”.
storeTodbooleanIf set to true, the tod associated files, such as signal, pixel and weight are stored in the temporary disk space of the Java virtual machine and can be retrieved from there. If not, the returned tod is null and the associated files will be deleted. Default is “False”. If you set to “True” you need to take care of the generated data or the temporary area will fill up.

3.  Output Parameters

The output parameter gives all result objects back as a list. A convenient method of assigning names immediately to the elements of this List in Jython is a call like this:

level1Corrected, PSWmap, PSWdiagProduct, PSWtod, signalMinusMap = destriper(level1=scans, array='PSW')

Output parameters
outputArrayListoutput=[Level1Corrected, map, diagnosticProduct,tod,signalMinusMapSignal]
The components of this list are the following
Level1CorrectedobjectThis is a list context or a Level 1 context containing the same scans as the input list context, except that the optimized offset functions have been subtracted from the timelines of the selected detector array. Note that the destriper updates only those scans that belong to the array selected in the “array” parameter.
mapobjectA map context containing a SimpleImage product that includes flux map, error map and coverage map.
diagnosticProductobjectThis is a product containing a Table Dataset that is described below.
todobjectTime ordered data product that can be used by a subsequent map maker run if the skybin size is not going to change.
signalMinusMapSignalobjectList context with signal minus the re-sampled map signal after the last iteration. This provides a check of the data the offset function is being fitted against.

3.1  Contents of the Diagnostic Product

The diagnostic product provides detailed results of the fitting process. The product contains a table dataset with a header and a table. The number of rows is equal to the product of scans in the map and number of detector that see the sky for the selected detector array.

Keywords in header

description:string“Diagnostic Tabledataset for Destriper”
arrayName:stringShows the detector array for the product. One of “PSW”,”PMW”,”PLW”.
chiSquareAll:doubleSum of all chi squares for all detectors contributing to the map.
validTimelines:integerNumber of all valid signal timelines in Level 1.
convergedTimelines:integerNumber of all channels that converged. Can not be larger than validTimelines.
iterThresh:doubleThreshold used to determine chi square convergence of timelines
maxIter:longMaximum number of iterations commanded.
numScans:longTotal number of scans in Level 1 data.
pGrade:longDegree of polynomial that represents the fit function.

Data in Table

Each row corresponds to the results for one detector scan.

1IndexintegerIndex number of detector channel. This can be the standard order of channels as they appear in the data. This will appear more than once if “perScan” was selected and more than one scan was commanded.
2channelNamestringStrings like “PSWE6″ or “PLWD2″. Name of detector channel. This will appear more than once if “perScan” was selected and more than one scan was commanded.
3detIndexlongIndex of each detector to simplify plotting against detectors.
4scanNumberlongScan number starting with 0. There will only be more than one scan number if “perScan” was selected and more than one scan was commanded.
5iterlongNumber of iterations for this scan until converged. If the scan didn’t converge this number is equal to parameter iterMax.
6chiSquaredoubleLast chi square achieved for this scan.
7convergencebooleanThe convergence status achieved for this scan.
8numberL2FlagslongNumber of Level 2 flags set in this scan.
9a0doubleFirst parameter of polynomial fit function
10a1doubleSecond parameter of polynomial fit function
11a2doubleThird parameter of polynomial fit function
12a3doubleFourth parameter of polynomial fit function
lastdeselectedintegerSet to 0 if the scan is valid and was included in the map. Set to 1 if the scan was not part of the map. This parameter can be manually changed and allows to deselect scans if the modified diagnostic product is given as “startParameter” to the destriper for another run.

4.  First Steps with the Destriper

The destriper can be tested in a straight forward way. To familiarize yourself with this task, try the following:

First load a scan map observation with the Product Browser into HIPE. The lower contexts of this observation will show up in the Observation View (Fig 3). By clicking on the level 1 context of a scan map observation as shown in Fig. 3 the level 1 context is highlighted and the applicable tasks appear in the Task View. Among those is the destriper that can be opened by double click. The default GUI will appear in the Editor View as shown in Fig 2.

To run processing for a single detector array, just run the module with the default setup by hitting the “Accept” button. Don’t forget to fill in the mandatory array parameter, i.e. “PSW” in order to run the destriper for this array. It helps replacing the output parameter “outArray” that appears by default by a list of output parameters like this one “level1Corrected, map, diagProduct, tod, signalMinusMap”, so you don’t have to extract the results one by one from the parameter list. It will take a little while to complete, depending on the size of the observation.

Figure 3: The Outline View of the observation context of a SPIRE scan map observation. Click on the level 1 entry to highlight it.

Once complete the 5 resulting products will appear either in a list or in the variables you provided in “Outputs”. Double click on the map will display the destriped map produced in the last iteration.

If you rather like scripts, then try the one provided in HIPE under “SPIRE Useful Scripts”→”Photometer Baseline Removal and Destriper”, or the following:

4.1  Script for basic tryout

obs = getObservation(myObsid, poolName=myDataPool)

level1Corrected, PSWmap, PSWdiagProduct,PSWtod,signalMinusMap = \
    destriper(level1=obs.level1, array='PSW')

There is a little more sophisticated script that applies relative gain correction for extended emission and deals with a few issues from older HIPE versions. You can view the resulting map with: Display PSWmap Figures 4 and 5 show the difference between a map where just medians were subtracted from the Level 1 data and a map that was built using the destriper.

Figure 4: Naive map with medians subtracted as offsets. Particularly strong striping can be seen in the upper half of the map.
Figure 5: Destriper map with offsets subtracted (polyDegree=0) that were optimized by iterations in the destriper.

5.  Running the Destriper with SPIA

The destriper is supported in the SPIRE Interactive Analysis plugin. The SPIA homepage contains further information. SPIA version 1.10.2 supports HIPE 10 at the time of writing. More information about SPIA can be found in the SPIRE Data Reduction Guide in Chapter 11. Since HIPE 10 the Level 2 results are derived using the destriper. The SPIA mirrors this behavior and provides a very simple way to re-reduce SPIRE photometer data with the destriper either through GUIs or scripts.

Here is a step-by-step procedure how to make a first test run of the destriper, provided the plugin is already installed in HIPE. Otherwise please refer to the SPIA homepage for instructions.

  1. Use the product browser view to load a SPIRE scan map or small map observation context into your HIPE session
  2. Make sure the observation context is selected in the variables view and open “Applicable” in the tasks view with double click.
  3. Double click spiaLoadCal and hit “Accept”. This creates a variable “cal” containing your calibration context. Make sure you have a pool with such a calibration context. If you don’t, you can set “FromObservation” to “Yes” to get the calibration context from the observation itself, however it is always recommended to use the newest calibration context.
  4. Select the observation context in the variables view again and double click the “spiaLevel2″ task in the tasks view.
  5. Drag the observation context onto the “obs” input button on the “spiaLevel2″ main tab and the calibration context onto the “cal” input button.
  6. The parameters in the “Destriper” panel can be left at their default values.
  7. Hit the “Accept” button to run the task. Although the processing speed was substantially shortened since HIPE 7, it is a good opportunity to go for a coffee if you are processing a large map.
  8. The resulting maps will appear on the screen and will be in the Level 2 of the new observation context, unless you had set “CopyObs” in the main panel to “No”. In this case the Level 2 context of the existing observation is just overwritten. Note that still a new variable name for the observation is created, however, this is just another Python pointer to the same data structure and can be deleted.

The equivalent script command is: obsOut = spiaLevel2(obs=observationContext, cal=cal)

6.  Running the Destriper in a Pipeline Environment

The destriper is written in such a way that it works on one detector array and its respective map at a time. To reconstruct the three maps that SPIRE normally produces, the task must be called three times with the respective array name. Since only the timelines for the respective detector array get changed, the same list context can be fed back into the task for processing the other two detector arrays. A construct similar to this one can be used.

level1 = obs.level1        #get level 1 context from observation context
arrays = ["PSW","PMW","PLW"]          #arrays
psize = [6.0,10.0,14.0]               #skybin sizes
for i in range(len(arrays)):
  level1Corrected, map, p1, p2, p3 = \
                 destriper(level1 = level1, pixelSize = psize[i], \
  level2.refs.put(arrays[i],ProductRef(map))   #update Level 2 context in original observation

Sometimes it is desired to save product versions and make them accessible for further analysis in a product pool. A browse product that appears in the outline view, which can be inspected separately, can be useful as well. We provide here a function that takes an observation context as input and provides a new observation context, that contains an updated (destriped) Level 1 context, the final maps produced by the destriper in Level 2, and the respective diagnostic products. In addition a browse product, i.e. an RGB colour map is produced as well according to pipeline standards. Level 0, 0.5 and calibration or auxiliary contexts were not copied to save space. Such a context can be saved easily into a pool.

Script: (for HIPE 10)

The script should be called with a command line like this,

obs = destripeObs(obsin)

where the observation context with at least Level 1 data is called obsin. A comparison of a standard processed level 2 result for PSW and a destriped Level 2 results is shown in Figures 4 and 5.

6.1  Parallel Mode Observations

The destriper can also be used for observations that were executed in parallel mode. In this case two separate observations were made, one for each scan direction. To eliminate striping and benefit from the constraints by scans taken in almost orthogonal directions, the Level 1 scans of both observations must be combined into one list context that can then be processed in the same way as that of a single observation. Some code like the following is the key to the combination.

  cmbLevel1 = Level1Context()             #make new list context
  level1 = obs_1.refs["level1"].product   #get Level 1 context from first observation
  for i in range(level1.count):           #Copy all scans (building blocks) from first observation
  level1 = obs_2.refs["level1"].product   #get Level 1 context from second observation
  for i in range(level1.count):           #Copy all scans (building blocks) from second observation

Now the variable cmbLevel1 contains a combined list context that can be run through the destriper in the same way as a normal Level 1 context.

For convenience we have prepared the following script that can be called on the command line with a list of observation contexts as argument.

Script: (This version is for HIPE 10).

For example, if four observation contexts were to be combined into one map, the script would be called as:

obslst = [observationContext_1,observationContext_2,observationContext_3,observationContext_4]
obsOut = destripePmodeObs(obslst)

7.  Frequently Asked Questions (FAQ)

  1. Question: How can I switch off the Level 2 deglitcher?
Answer: Set the parameter L2DeglitchRepeat to zero. Currently the default is off since the Level 2 deglitcher does not work right now.
  1. Question: How do I avoid getting stripes aligned with strong sources in the map?
Answer: Bright sources can outweight the influence of the other readouts belonging to a scan. Setting the brightSource flag will deselect brighter sources from being considered by the algorithm. This is the default.
  1. Question: How do I find which scans are responsible for residual stripes that the destriper can not resolve anymore?
Answer: The diagnostic product contains numbers that characterize the iterations that were performed. Problematic scans can be identified by looking for the conversion flag set to “false”, and high values of parameters “Iter” or “chiSquare” compared to the rest. A better method is to use the Bolometer Finder. Its use is described in the SPIRE Data Reduction Guide in Chapter 7.3
  1. Question: What is the quickest check to assess the quality of a destriper run?
Answer: In the HIPE image viewer select the error map for display and use the “Edit cut levels” function accessible by right clicking to re-adjust the display dynamic range. Any residual stripes will show prominently.
  1. Question: I find many holes in my destriped map while the naive map shows no holes. Why and what can I do?
Answer: The Level 2 deglitcher is probably flagging too many good points. Increasing the value of the parameter kappa or switching off the deglitcher entirely by setting the parameter L2DeglitchRepeat to zero will help.
  1. Question: Why does my destriper not run through with Level 1 data from an observation performed in bright source mode?
Answer: This happens when data in bright source mode is reduced to Level 1 with the jump detector module active. This module is not meant to be used for bright source data because the thermistors are saturated in this mode and jumps are generally not observed anyway with bright sources.

Information about the destriper in older HIPE versions: Older Destriper Versions

Edit - History - Print - Recent Changes - Search
Page last modified on March 11, 2013, at 04:54 PM