# REAL BSCAN signal processing

## Purpose of this document

This document describes the signal-processing steps used to convert raw REAL BSCAN waveforms into the relative aerosol-backscatter fields commonly displayed as REAL images.

It begins **after a BSCAN file has already been read correctly**.

For the raw file structure, header format, byte order, range coordinate, and examples of reading BSCAN files, see:

```text
REAL_bscan_data_file_format.md
```

For a description of the REAL instrument itself, see:

```text
REAL_instrument_description.md
```

The processing sequence documented here is based on the currently verified Python implementation derived from Pierre Dérian's REAL lidar toolbox and used by:

```text
M2HATS_REAL_bscan_PPI_viewer.py
```

That public viewer is a small, student-facing implementation. It does not require Pierre's full lidar toolbox, CUDA, SSH access, or accounts on the Chico State research servers.

The current implementation is especially well tested for the near-horizontal M2HATS PPI scans. Some parts of the code also identify and display RHI scans, but this document does not claim that every historical REAL deployment or scan strategy has been fully generalized.

---

# 1. What goes into the processing

After a BSCAN file has been read, the relevant quantities are:

- raw parallel-polarization digitizer-count profiles;
- raw perpendicular-polarization digitizer-count profiles;
- range for every sample;
- azimuth for every laser pulse;
- elevation for every laser pulse;
- time for every laser pulse.

For one scan, the two raw polarization channels can be thought of as arrays with dimensions:

```text
number of laser pulses × number of range bins
```

The raw values are digitizer counts.

They are **not yet relative scattering intensity**, and they are not an absolute aerosol backscatter coefficient or aerosol concentration.

---

# 2. Overview of the verified processing sequence

The current Python implementation follows this sequence:

```text
raw parallel and perpendicular profiles
        |
        v
sum the two polarization channels
        |
        v
identify pre-trigger / negative-range samples
        |
        v
estimate one background value for each laser pulse
        |
        v
retain the nonnegative-range portion of each profile
        |
        v
subtract the pulse-by-pulse background
        |
        v
multiply by range squared
        |
        v
add the current implementation's dB offset
        |
        v
clip nonpositive values to a small positive number
        |
        v
convert to 10 log10
        |
        +------------------------------+
        |                              |
        | unfiltered product           | filtered product
        | stops here                   v
        |                    short radial median filter
        |                              |
        |                              v
        |                    subtract long radial median
        |                              |
        |                              v
        |                    discard filter-edge regions
        +------------------------------+
        |
        v
map native range/angle samples to plotting coordinates
```

The exact implementation is described below.

---

# 3. Combining the two polarization channels

REAL records two elastic-backscatter profiles for every laser pulse:

1. parallel polarization;
2. perpendicular polarization.

The current processing code begins by summing them:

```python
profiles = np.asarray(raw["profiles"]).sum(axis=1)
```

Conceptually:

```text
combined signal = parallel signal + perpendicular signal
```

The result is one range profile per laser pulse.

This combined signal is appropriate when the purpose is to display the total elastic aerosol-return structure rather than analyze polarization or depolarization separately.

The original two channels should still be retained in the raw data. Analyses of polarization should begin with the separate channel measurements rather than this combined product.

---

# 4. Pre-trigger samples and the background estimate

The M2HATS BSCAN waveforms begin at a negative nominal range.

These negative-range samples occur before the atmospheric return associated with the outgoing laser pulse and provide a useful estimate of the electronic/background level.

The current Python implementation finds the first range bin for which:

```text
range >= 0
```

It then uses the preceding negative-range portion of the combined profile to estimate a separate mean background for each laser pulse.

In the current code, the sample immediately preceding the first nonnegative range bin is also excluded from the background average.

The background calculation can be summarized as:

```text
background for pulse i
    =
mean of selected pre-trigger samples for pulse i
```

The nonnegative-range part of the profile is then retained.

The same background value is subtracted from every retained range bin for that laser pulse:

```text
S_bg_corrected(i,r) = S_combined(i,r) - B(i)
```

where:

- `i` is the laser-pulse / beam index;
- `r` is range;
- `B(i)` is the mean pre-trigger background for that pulse.

The background is therefore estimated **pulse by pulse**, not once for an entire scan.

---

# 5. Range-squared correction

After background subtraction, the current processing compensates for the geometric range dependence of the lidar return.

The range is first converted from meters to kilometers:

```python
distances_km = distances * 1.0e-3
```

The signal is then multiplied by range squared:

```python
profiles *= np.square(distances_km)[np.newaxis, :]
```

Thus the implemented quantity at this stage is proportional to:

```text
(S - background) × r²
```

with `r` expressed in kilometers in the current code.

This operation compensates for the inverse-square geometric spreading term in the elastic lidar return.

It does **not** make the result an absolutely calibrated aerosol backscatter coefficient.

Atmospheric transmission, overlap behavior, aerosol optical properties, calibration factors, and other effects are not removed simply by multiplying by `r²`.

For this reason the displayed product is described as **relative backscatter intensity**, not absolute aerosol backscatter.

---

# 6. Offset and conversion to decibels

The current verified implementation next adds a constant:

```python
DEFAULT_DB_OFFSET = 500.0
```

so that:

```text
Q = (S - background) × r² + 500
```

where range is in kilometers in the implemented `r²` term.

Values are then clipped so that no value supplied to the logarithm is smaller than:

```text
1.0 × 10^-6
```

The code then calculates:

```text
10 log10(Q)
```

In Python this is:

```python
profiles += 500.0
np.clip(profiles, 1.0e-6, None, out=profiles)
np.log10(profiles, out=profiles)
profiles *= 10.0
```

The resulting values are the basis of the viewer's displayed:

```text
Relative backscatter intensity (dB)
```

## Important interpretation

The 500-unit quantity is an **implementation offset applied before the logarithm**.

It should not be interpreted as:

- an aerosol calibration constant;
- a physical aerosol concentration;
- an absolute backscatter coefficient;
- or an independently measured atmospheric quantity.

The exact numerical dB values produced by this processing therefore belong to this relative-display convention.

When comparing products generated by different software, it is important to know whether the same offset, range units, background treatment, and logarithmic convention were used.

---

# 7. What "unfiltered" means

In the public M2HATS viewer, the option called **unfiltered** does not display raw BSCAN digitizer counts.

The unfiltered product has already undergone:

1. addition of the two polarization channels;
2. pre-trigger background estimation;
3. background subtraction;
4. removal of negative-range samples;
5. multiplication by range squared;
6. addition of the 500-unit offset;
7. clipping to a positive minimum value;
8. conversion to `10 log10`;
9. coordinate conversion for display.

"Unfiltered" means only that the two radial median-filter operations described in the next section are not applied.

This distinction is important.

---

# 8. Radial median filtering used for fine-scale structure

The current filtered product is designed to emphasize relatively fine-scale aerosol structure while suppressing both small-scale noise and slowly varying radial structure.

The filtering is performed **only along the range dimension**.

It is not a two-dimensional image filter across neighboring azimuths.

## 8.1 Short-window median filter

The current default short filter length is:

```text
7 range bins
```

For M2HATS, with 1.5-m range sampling, this corresponds nominally to:

```text
7 × 1.5 m = 10.5 m
```

The operation is:

```text
S_short = median_filter(S_dB, 7 bins along range)
```

In the current Python code:

```python
profiles = median_filter(
    profiles,
    size=(1, 7),
    mode="constant",
    cval=0,
)
```

The purpose is to suppress very small-scale range-bin variability before the longer-scale background structure is removed.

## 8.2 Long-window median and high-pass operation

The current default long filter length is:

```text
333 range bins
```

For M2HATS:

```text
333 × 1.5 m = 499.5 m
```

A long-window median is calculated from the short-filtered field and subtracted:

```text
S_filtered
    =
S_short
    -
median_filter(S_short, 333 bins along range)
```

In the current Python code:

```python
profiles = profiles - median_filter(
    profiles,
    size=(1, 333),
    mode="constant",
    cval=0,
)
```

This removes slowly varying radial structure and emphasizes departures from the local large-scale radial background.

Examples of large-scale structure that may be reduced by this operation include:

- broad radial signal trends;
- effects associated with attenuation and shadows;
- gradual laser-power-related variations;
- large aerosol structures whose radial scale is much larger than the features of interest.

The result is commonly described as a **high-pass median-filtered** relative-backscatter field.

## 8.3 Edge removal

A finite-width median filter has unreliable edge regions because a full neighborhood is not available there.

For the 333-bin default filter:

```text
333 // 2 = 166 bins
```

The implementation therefore removes 166 range bins from each end after filtering.

At 1.5-m sampling, this corresponds to approximately:

```text
166 × 1.5 m = 249 m
```

from each radial edge of the retained filtered product.

The exact number changes if a different high-pass filter length is selected.

---

# 9. The filter lengths are processing choices, not instrument constants

The values:

```text
7 bins
333 bins
```

are the current defaults in the public M2HATS viewer.

They are **not fundamental specifications of REAL**.

They are processing choices used for the current scientific/display product.

The Python program permits the user to choose different positive odd filter lengths.

Any analysis that changes these values should record the chosen settings because the filter scales affect the structures that are retained or suppressed.

---

# 10. Scan-type identification in the current Python implementation

The public Python code attempts to identify whether a scan is:

- STARE;
- PPI;
- or RHI.

It unwraps the recorded azimuth and elevation angles and compares the total angular variation.

The current decision logic is approximately:

```text
if azimuth variation < 0.015 degree
and elevation variation < 0.015 degree:
    STARE

else if azimuth variation > elevation variation:
    PPI

else:
    RHI
```

This is a practical heuristic used by the current viewer.

It should not be treated as a universal definition of every REAL scan strategy ever used.

Historical REAL field campaigns may require more explicit scan metadata or campaign-specific logic.

---

# 11. PPI coordinate mapping

For a PPI scan, the current viewer uses the meteorological azimuth convention:

- 0 degrees points north;
- 90 degrees points east;
- azimuth increases clockwise.

For each native range/azimuth sample:

```text
x = r sin(azimuth)
y = r cos(azimuth)
```

Thus:

- `x` is positive toward the east;
- `y` is positive toward the north.

The current M2HATS viewer uses the measured slant range directly in this two-dimensional mapping rather than multiplying the range by `cos(elevation)`.

That approximation is appropriate for the nearly horizontal M2HATS scans for which the code was developed, but it should not automatically be generalized to PPI scans at substantial elevation angles.

---

# 12. RHI coordinate mapping

For a scan classified as RHI, the current viewer maps each range/elevation sample as:

```text
horizontal distance in scan plane = r cos(elevation)

height relative to lidar          = r sin(elevation)
```

The plotted horizontal coordinate is therefore distance in the vertical scan plane.

It is not automatically an east-west or north-south geographic coordinate.

The fixed azimuth of the RHI determines the geographic orientation of that vertical plane.

A more complete treatment of general REAL scan geometry should eventually be documented separately.

---

# 13. Coordinate conversion is not the same as Cartesian interpolation

This distinction is important.

The current public viewer does **not** resample the REAL data onto a uniformly spaced Cartesian grid before plotting.

Instead, it calculates an `x` and `y` (or horizontal-distance and `z`) coordinate for every native polar sample and supplies those curvilinear coordinates directly to Matplotlib's `pcolormesh`.

Conceptually:

```text
native range/angle samples
        |
        v
calculate physical coordinates of each native sample
        |
        v
plot the curvilinear grid directly
```

This is different from:

```text
native polar samples
        |
        v
interpolate onto a new regular x-y grid
        |
        v
plot the regular Cartesian grid
```

Some other REAL processing workflows may perform true Cartesian interpolation or gridding, but the current public M2HATS viewer does not.

That distinction should be kept clear when comparing products.

---

# 14. Computational acceleration and GPU/CUDA implementations

The scientific processing sequence should be distinguished from the hardware used to perform it.

The public:

```text
M2HATS_REAL_bscan_PPI_viewer.py
```

uses ordinary NumPy/SciPy processing and intentionally does not require CUDA or Pierre's full lidar toolbox.

For large REAL archives, however, some operations can become computationally expensive. In the broader REAL processing environment there are provisions for accelerating suitable operations with GPUs and CUDA.

Median filtering is an important candidate because it must be repeated over very large numbers of profiles and scans.

Other implementations may also use GPU acceleration for spatial interpolation or Cartesian gridding when those operations are part of the processing path.

The important conceptual point is:

> GPU/CUDA acceleration changes how the calculation is executed, not the scientific definition of the calculation.

A CPU and GPU implementation should produce scientifically equivalent results when they implement the same algorithm, boundary conditions, numerical conventions, and filter definitions.

The current public viewer should not be described as a CUDA program.

---

# 15. Reference Python implementation of the core processing

The following condensed function captures the essential current processing sequence.

It assumes that the BSCAN file has already been read and that `profiles` has dimensions:

```text
laser pulse × polarization channel × range bin
```

```python
import numpy as np
from scipy.ndimage import median_filter


def process_real_scan(
    profiles,
    distances_m,
    filtered=True,
    lowpass_length=7,
    highpass_length=333,
):
    # Add parallel and perpendicular polarization channels.
    signal = np.asarray(profiles).sum(axis=1).astype(np.float64, copy=True)
    r = np.asarray(distances_m, dtype=float).copy()

    # Find first nonnegative range bin.
    zero_indices = np.flatnonzero(r >= 0.0)
    if zero_indices.size == 0:
        raise ValueError("No nonnegative range bins.")

    zero_index = int(zero_indices[0])

    # Current Pierre-derived implementation excludes the final
    # negative-range bin from the background average.
    background_end = max(zero_index - 1, 1)

    background = signal[:, :background_end]

    # Retain nonnegative ranges.
    signal = signal[:, zero_index:]
    r = r[zero_index:]

    # Pulse-by-pulse background subtraction.
    signal -= background.mean(axis=1)[:, np.newaxis]

    # Range-squared correction.  Current implementation uses km.
    r_km = r * 1.0e-3
    signal *= np.square(r_km)[np.newaxis, :]

    # Current relative-dB convention.
    signal += 500.0
    np.clip(signal, 1.0e-6, None, out=signal)
    np.log10(signal, out=signal)
    signal *= 10.0

    if filtered:
        # Short radial median.
        signal = median_filter(
            signal,
            size=(1, lowpass_length),
            mode="constant",
            cval=0,
        )

        # High-pass median operation.
        signal = signal - median_filter(
            signal,
            size=(1, highpass_length),
            mode="constant",
            cval=0,
        )

        # Remove filter-edge regions.
        edge = highpass_length // 2
        signal = signal[:, edge:-edge]
        r = r[edge:-edge]

    return signal, r
```

This is intentionally a signal-processing example rather than a complete BSCAN reader or plotting program.

For the functioning public application, use the complete:

```text
M2HATS_REAL_bscan_PPI_viewer.py
```

---

# 16. Important scientific cautions

## 16.1 Relative backscatter is not aerosol concentration

The processed REAL field is a representation of elastic aerosol backscatter.

It should not be interpreted directly as aerosol mass concentration or number concentration.

Backscatter depends on particle properties and atmospheric conditions in addition to particle amount.

## 16.2 Range-squared correction is not full lidar inversion

Multiplying by `r²` compensates for geometric spreading.

It does not by itself correct for all atmospheric extinction, overlap effects, calibration factors, or other terms in the lidar equation.

## 16.3 The displayed dB scale is implementation dependent

The current code adds 500 before applying the logarithm.

Therefore the numerical dB values should be interpreted as the current relative-display quantity, not an absolute calibrated backscatter scale.

## 16.4 Filtering changes the scientific quantity being viewed

The high-pass median-filtered product intentionally removes slowly varying radial structure.

It is therefore excellent for emphasizing certain fine-scale aerosol structures, but it is not identical to the unfiltered relative-backscatter field.

Questions involving broad aerosol gradients or large structures may require the unfiltered product or a different processing choice.

## 16.5 Scan geometry matters

A PPI or RHI image is built from sequential laser pulses.

The coordinate conversion described here assigns a spatial position to each measured sample, but it does not make the scan instantaneous.

Atmospheric motion during the scan can still matter for quantitative analyses.

---

# 17. Relationship to the M2HATS public viewer

The current public viewer performs several jobs in one program:

1. locate a BSCAN file by date/time or direct path/URL;
2. read the BSCAN binary format;
3. identify individual scans where necessary;
4. construct range, azimuth, elevation, and time arrays;
5. identify STARE/PPI/RHI geometry;
6. perform the signal processing described in this document;
7. convert native polar samples to plotting coordinates;
8. display the result;
9. optionally save a PNG.

Those software-user tasks should not be confused with the scientific signal-processing definition itself.

The processing described in this file is meant to remain understandable even if the viewer interface changes later.

---

# 18. What this document does not yet attempt to generalize

This document does not yet provide a complete universal treatment of:

- every REAL scan strategy used in every field campaign;
- robust historical PPI/RHI/STARE classification;
- arbitrary elevated PPI projection;
- geographic coordinate transformations for all deployments;
- true interpolation to a regular Cartesian grid;
- scan-motion correction;
- optical-flow wind retrieval;
- absolute aerosol-backscatter calibration;
- depolarization retrieval from the two separate channels;
- MATLAB implementations;
- historical IDL implementations.

Those topics can be documented separately as verified code and source material are assembled.

It is better to document the current working calculation accurately than to make a more general claim that has not yet been tested.

---

# 19. Current source and related files

Current public M2HATS viewer guide:

```text
https://lidar.csuchico.edu/m2hats/M2HATS_REAL_bscan_PPI_viewer_guide.md
```

Current public Python viewer:

```text
https://lidar.csuchico.edu/m2hats/M2HATS_REAL_bscan_PPI_viewer.py
```

Related canonical documentation:

```text
REAL_instrument_description.md
REAL_bscan_data_file_format.md
```

A future document may be useful for general REAL scan geometry, PPI/RHI handling, and Cartesian gridding once those procedures have been assembled and verified across additional REAL data sets.

---

# 20. Compact processing summary

For the current Pierre-derived implementation, a concise description is:

> Add the parallel and perpendicular raw REAL signals, estimate and subtract a pulse-by-pulse background from pre-trigger samples, retain nonnegative ranges, multiply by range squared, add the current 500-unit display offset, convert to `10 log10`, and optionally apply a short radial median filter followed by subtraction of a long radial median to emphasize fine-scale aerosol structure. The processed native polar samples are then mapped to physical plotting coordinates.

The result is a **relative aerosol-backscatter intensity field**.

It is not an absolute aerosol backscatter coefficient, an aerosol concentration measurement, or a direct wind measurement.
