Building a product

This is the archive-creation stream, end to end: science arrays plus metadata and backplanes go in, one PDS4-shaped FITS product comes out.

The science arrays here are synthetic. The real stream reads them from a Cassini UVIS PDS product, and the metadata and backplanes arrive already computed from Showalter’s pipeline. Nothing below changes when that happens: only the Source is swapped. That is the whole point of the seam.

from pathlib import Path

import numpy as np
from astropy.io import fits

from cubegenpy import BuildConfig, CubeBuilder, SyntheticSource, read_product

OUT = Path("products")

1. The source

A Source produces one ProductInputs, the single payload the rest of the pipeline moves around. Its field names match the writer’s arguments one for one.

source = SyntheticSource()
inputs = source.load()

for name in ("raw_counts", "cal_factor", "wavelength", "cube"):
    value = getattr(inputs, name)
    shape = "None (the calibrator fills it)" if value is None else f"{value.shape} {value.dtype}"
    print(f"{name:12} {shape}")

print()
print("dims        ", inputs.dims)
print("geometry    ", {k: len(v) for k, v in inputs.geometry.items()})

raw_counts   (16, 8, 4) uint16
cal_factor   (16, 8) float32
wavelength   (16,) float32
cube         None (the calibrator fills it)

dims         Dims(NX=16, NY=8, NZ=4, NT=3)
geometry     {'SC_GEOM': 20, 'BODY_GEOM': 10, 'GENERAL_GEOM': 3, 'RING_GEOM': 12}

cube is the only field a source may leave empty. Everything else is input.

Note the conventions: raw_counts is uint16 in the PDS3 sense, where 65535 is the null rather than a count, and wavelength is in Angstrom even though pyuvis hands back nanometres.

2. Configure the build

BuildConfig carries the two knobs that matter: where the calibration factor comes from, and how much background to subtract first.

Here the synthetic source already carries a CAL_FACTOR, so "stored" reuses it verbatim. The archive stream will use "pyuvis" once the real reader lands.

config = BuildConfig(calibration="stored")
print(config.summary())
print("background mode:", config.background.mode)

cubegenpy[cal=stored, bg=none, v1.0]
background mode: none

3. Build

product = CubeBuilder(source, config=config).build(OUT)
print("fits :", product.fits_path)
print("label:", product.label_path)

fits : products/EUV2013_047_09_33_59.fits
label: products/EUV2013_047_09_33_59.xml

WARNING: VerifyWarning: Card is too long, comment will be truncated. [astropy.io.fits.card]

4. What landed

with fits.open(product.fits_path) as hdul:
    hdul.info()

Filename: products/EUV2013_047_09_33_59.fits
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU      33   (16, 8, 4)   float32   
  1  RAW_COUNTS    1 ImageHDU        11   (16, 8, 4)   int16   
  2  CAL_FACTOR    1 ImageHDU         9   (16, 8)   float32   
  3  BACKGROUND    1 ImageHDU         9   (16, 8)   float32   
  4  SC_GEOM       1 BinTableHDU    102   2R x 25C   [12A, 12E, 12E, 12E, 12E, 12E, 12E, 12E, 12E, 12E, 12E, 12E, 12E, 12E, 12E, 12E, 12E, 12E, 12E, 12E, 12E, E, E, E, L]   
  5  BODY_GEOM     1 BinTableHDU     57   1R x 13C   [12A, 480E, 480E, 480E, 480E, 480E, 480E, 480E, 480E, 480E, 480E, 480L, 480L]   
  6  GENERAL_GEOM    1 BinTableHDU     21   1R x 3C   [12D, 480E, 480E]   
  7  RING_GEOM     1 BinTableHDU     63   1R x 14C   [480E, 480E, 480E, 480E, 480E, 480E, 480E, 480E, 480E, 480E, 480L, 480L, 480E, 480E]   
  8  KERNELS       1 TableHDU        15   3R x 2C   [A32, A4]   
  9  WAVELENGTH    1 ImageHDU         8   (16,)   float32   

with fits.open(product.fits_path) as hdul:
    header = hdul[0].header
    for keyword in ("PROD_ID", "MISSION", "INSTRUME", "CHANNEL", "TARGET", "OBS_UTC", "INT_TIME"):
        print(f"{keyword:9} = {header[keyword]!r}")

PROD_ID   = 'EUV2013_047_09_33_59'
MISSION   = 'Cassini'
INSTRUME  = 'UVIS'
CHANNEL   = 'EUV'
TARGET    = 'TITAN'
OBS_UTC   = '2013-02-16T09:34:00.519'
INT_TIME  = 240.0

5. The calibration, in the open

The whole formula is one line, and it runs entirely in counts per second:

calibrated = (raw_counts / int_time - background_offset) * cal_factor

Worth reproducing by hand once, so nothing about the written product is mysterious. Undefined calibration entries arrive as NaN and are zeroed on the way out, which is what nan_to_num below mirrors.

written = read_product(product.fits_path)

int_time = float(inputs.header["INT_TIME"])
counts_per_sec = inputs.raw_counts.astype(np.float64) / int_time
expected = np.nan_to_num(counts_per_sec * inputs.cal_factor[:, :, np.newaxis]).astype(np.float32)

print("shapes match :", written.cube.shape == expected.shape)
print("values match :", np.array_equal(written.cube, expected))

shapes match : True
values match : True

What changes with real data

Only the first cell of section 1. SyntheticSource() becomes a source that reads the science arrays from a PDS product and merges in the externally computed metadata and backplanes, and calibration="stored" becomes "pyuvis". Sections 3 through 5 stay exactly as they are.