# REAL BSCAN raw data file format

## Purpose of this document

This document describes the custom **BSCAN** binary file format used to store original raw data from the Raman-shifted Eye-safe Aerosol Lidar (REAL).

It is intended to help students, collaborators, researchers, and AI-assisted analysis tools understand:

- what is stored in a REAL BSCAN file;
- how the file is organized;
- how to interpret the 30-word header;
- how to construct the range coordinate;
- how to read the files correctly in Python and MATLAB;
- which header fields are reliable for the 2023 M2HATS data set;
- and how to recognize common reading errors.

This document is about the **raw data file format and how to read it**.

It does **not** describe the subsequent signal-processing steps required to convert the raw digitizer counts into relative aerosol scattering intensity or into the high-pass-filtered images commonly used for REAL analysis. Those steps will be described in a separate Markdown document.

---

# 1. Quick mental model

A REAL BSCAN file can be understood with the following hierarchy:

```text
one BSCAN file
    |
    +-- normally one complete lidar scan
            |
            +-- record 1 = one laser pulse
            +-- record 2 = one laser pulse
            +-- record 3 = one laser pulse
            +-- ...
            +-- record N = one laser pulse
```

Each record contains:

```text
30-word header
+
parallel-polarization backscatter profile
+
perpendicular-polarization backscatter profile
```

All of these values are stored as **32-bit, big-endian floating-point numbers**.

For M2HATS, a typical profile contains 4200 samples in each polarization channel, with a native range spacing of 1.5 m.

The profile values are **raw digitizer counts**. They are not yet background-subtracted, range-corrected, converted to decibels, filtered, or gridded.

---

# 2. How BSCAN files are created

The original untouched REAL data are written in near-real time by the REAL data-acquisition system.

A LabVIEW program running on a Windows computer in the REAL laboratory shipping container records the waveform from every laser pulse together with the associated header information.

The important points are:

- the original waveforms are retained for every recorded laser pulse;
- one record normally corresponds to one laser pulse;
- one BSCAN file normally contains one complete scan;
- each pulse has its own 30-word header;
- each pulse has two recorded backscatter profiles, one for each polarization channel.

This means that a BSCAN file preserves the pulse-by-pulse structure of a REAL scan.

---

# 3. Why the format is called BSCAN

The name **BSCAN** comes from older radar and lidar display terminology.

An A-scope is essentially one-dimensional: signal strength as a function of range for one transmitted pulse or pointing direction.

A B-scope or B-scan is more image-like. Many range profiles are placed side by side so that one dimension is range and the other dimension is record number, time, pointing angle, aircraft position, or another scan coordinate.

That is also a useful way to think about a REAL BSCAN file.

Each laser pulse contributes one range-resolved record. If the records are stacked in acquisition order, the data form a rectangular array with range along one dimension and pulse number, time, or pointing angle along the other.

The format has historical roots in atmospheric lidar work performed at NASA Langley Research Center and later at NCAR. Some header fields still reflect the airborne-lidar origin of the format, including fields for platform altitude, latitude, longitude, roll, pitch, yaw, and transect number. REAL is normally operated from the ground, so many of those fields are unused or not meaningful in a particular deployment.

---

# 4. Binary representation

Every value in a REAL BSCAN file is stored as a 4-byte single-precision floating-point number.

| Property | BSCAN representation |
|---|---|
| Data type | 32-bit floating point |
| Bytes per word | 4 |
| Byte order | big-endian |
| IDL terminology | `real*4` |
| MATLAB type | `single` |
| NumPy type | `float32` with dtype `>f4` |

The **big-endian byte order is essential**.

If the file is read with the wrong byte order, familiar values such as year, month, hour, azimuth, elevation, range spacing, and profile length will appear nonsensical.

In MATLAB, open the file with:

```matlab
fopen(filename,'r','b')
```

where `'b'` specifies big-endian byte order.

In Python/NumPy, a convenient representation is:

```python
dtype=">f4"
```

where `>` means big-endian and `f4` means a four-byte floating-point number.

---

# 5. Overall record structure

A BSCAN file is a sequence of fixed-length records:

```text
Record 1
Record 2
Record 3
...
Record N
```

Each record has exactly three parts:

```text
30-word header
parallel-polarization profile
perpendicular-polarization profile
```

Let:

- `Lp` = number of words in one profile;
- `Lr` = number of words in one complete record.

Then:

```text
Lr = 30 + 2*Lp
```

Because every word occupies 4 bytes:

```text
record_length_bytes = 4 * (30 + 2*Lp)
```

The number of complete records in the file is:

```text
number_of_records = file_size_bytes / record_length_bytes
```

In practice, code should use integer division or otherwise verify that the file size is consistent with the calculated record length.

## Do not assume the number of records

For M2HATS, many scans contain roughly 150 records because a typical scan lasted about 15 seconds at a 10-Hz pulse repetition rate.

However, **the number of records should always be calculated from the file size and record length rather than hard-coded**.

---

# 6. The two polarization profiles

Each record contains two range-resolved arrays.

## Channel 1

**Parallel-polarization backscatter signal**

The received polarization is parallel to the transmitted linear polarization.

## Channel 2

**Perpendicular-polarization backscatter signal**

The received polarization is perpendicular to the transmitted linear polarization.

The profile values are stored in **digitizer counts**.

The recorded waveforms contain both:

- electronic/background signal; and
- the atmospheric lidar return.

For the M2HATS data, the profiles were sampled with a 14-bit, 100-megasample-per-second analog-to-digital converter.

The native range spacing is typically 1.5 m.

> **Important:** The two raw polarization profiles should be preserved separately when the file is read. Combining the channels, estimating and subtracting background, correcting for range, converting to dB, and filtering are later processing steps and are not part of simply reading the BSCAN file.

---

# 7. The 30-word header

Each laser-pulse record begins with a 30-word header.

The table below uses **zero-based indexing**, which matches Python and the historical BSCAN documentation.

MATLAB uses one-based indexing, so MATLAB users should add 1 to every index.

| Zero-based index | MATLAB word | General meaning |
|---:|---:|---|
| 0 | 1 | Hour, 0-23 |
| 1 | 2 | Minute, 0-59 |
| 2 | 3 | Second, 0.00-59.999 |
| 3 | 4 | GPS altitude, meters above sea level |
| 4 | 5 | Alternative altitude, such as pressure altitude |
| 5 | 6 | Alternative altitude, such as radar altimeter |
| 6 | 7 | Lidar beam-pointing azimuth, degrees |
| 7 | 8 | Lidar beam-pointing elevation, degrees |
| 8 | 9 | Platform roll, degrees with respect to earth horizon |
| 9 | 10 | Platform pitch, degrees with respect to earth horizon |
| 10 | 11 | Platform yaw or sideslip with respect to platform heading, degrees |
| 11 | 12 | Month, 1-12 |
| 12 | 13 | Day, 1-31 |
| 13 | 14 | Year, preferably full year such as 2023 |
| 14 | 15 | Pulse repetition frequency of lidar, Hz |
| 15 | 16 | Number of laser pulses intentionally not used in this record |
| 16 | 17 | Data type identifier |
| 17 | 18 | Number of hours to adjust header time to UTC |
| 18 | 19 | Platform latitude, degrees north |
| 19 | 20 | Platform longitude, degrees east of Greenwich |
| 20 | 21 | Range from lidar to first word of recorded profile data, meters |
| 21 | 22 | Range resolution, meters per word |
| 22 | 23 | Platform identifier |
| 23 | 24 | Number of words in header; should be 30 |
| 24 | 25 | No-data flag |
| 25 | 26 | Empty or unused |
| 26 | 27 | Number of laser shots used to derive this record |
| 27 | 28 | Number of laser shots intended to use to derive this record |
| 28 | 29 | Number of words in one profile array |
| 29 | 30 | Transect number, historically used for airborne lidar research |

These are the **general intended meanings** of the BSCAN header fields.

Not every deployment populated every field correctly.

---

# 8. M2HATS-specific header notes

For M2HATS, several header fields were not filled correctly by the data-acquisition system or have uncertain meanings.

The following table records the current working interpretation for the 2023 M2HATS files.

| Zero-based index | MATLAB word | M2HATS interpretation |
|---:|---:|---|
| 0 | 1 | Correct: hour |
| 1 | 2 | Correct: minute |
| 2 | 3 | Correct: second |
| 3 | 4 | All zeros |
| 4 | 5 | Meaning unclear; varies roughly around 0.375-0.378 |
| 5 | 6 | Meaning unclear; oscillates roughly between -0.120 and -0.095 |
| 6 | 7 | Correct: azimuth; often 60-120 degrees during sector scans |
| 7 | 8 | Correct: elevation; close to zero for near-horizontal scans |
| 8 | 9 | All zeros |
| 9 | 10 | All zeros |
| 10 | 11 | All zeros |
| 11 | 12 | Correct: month; 7, 8, or 9 for M2HATS |
| 12 | 13 | Correct: day of month |
| 13 | 14 | Correct: year; 2023 for M2HATS |
| 14 | 15 | All zeros |
| 15 | 16 | Constant at approximately 967; meaning uncertain |
| 16 | 17 | Constant at approximately 220; meaning uncertain |
| 17 | 18 | All zeros |
| 18 | 19 | **Incorrect: contains Chico latitude, not the M2HATS/Tonopah latitude** |
| 19 | 20 | **Incorrect: contains Chico longitude, not the M2HATS/Tonopah longitude** |
| 20 | 21 | Correct: range of first profile element; typically -567 m |
| 21 | 22 | Correct: range spacing; typically 1.5 m |
| 22 | 23 | Approximately 9.0001001; meaning uncertain |
| 23 | 24 | Correct: 30 words per header |
| 24 | 25 | All zeros |
| 25 | 26 | Constant at 2; meaning uncertain |
| 26 | 27 | Meaning unclear; varies roughly from 380 to 440; possibly transmit pulse energy |
| 27 | 28 | Meaning unclear; varies roughly from 1160 to 1220; possibly pump pulse energy |
| 28 | 29 | Correct: 4200 words per profile for M2HATS |
| 29 | 30 | Constant at 1 |

## Important M2HATS warning

Do **not** use header fields 18 and 19, zero-based, as the geographic location of the M2HATS REAL instrument.

Those fields contain Chico coordinates and are incorrect for the 2023 Tonopah deployment.

Use campaign-specific M2HATS documentation or verified metadata when geographic coordinates are needed.

Also, do not assign physical meanings to header values whose meanings are listed as uncertain unless they have been independently verified.

---

# 9. Constructing the range coordinate

The BSCAN header contains the information necessary to construct the range coordinate.

Using zero-based indexing:

```text
header[20] = range of the first profile element, meters
header[21] = range spacing, meters
header[28] = number of samples in each profile
```

Using MATLAB one-based indexing:

```text
header(21) = range of the first profile element, meters
header(22) = range spacing, meters
header(29) = number of samples in each profile
```

If:

```text
r0 = range of first element
dr = range spacing
Lp = profile length
```

then the range coordinate is:

```text
r[k] = r0 + k*dr
```

for:

```text
k = 0, 1, ..., Lp-1
```

For M2HATS, typical values are:

```text
r0 = -567 m
dr = 1.5 m
Lp = 4200
```

## Why are some ranges negative?

The recorded waveform begins before the atmospheric return from the transmitted laser pulse.

For M2HATS, the first profile samples are therefore assigned negative ranges.

Those pre-trigger samples are useful later for estimating the electronic/background signal.

A negative range in the raw waveform does not mean that the lidar measured atmosphere physically behind the instrument. It represents samples recorded before the atmospheric return begins.

---

# 10. Determining the file dimensions

The correct way to determine the dimensions of a BSCAN file is:

1. read the first 30 words as big-endian single-precision floats;
2. obtain the profile length from header index 28, zero-based;
3. calculate the record length;
4. obtain the file size in bytes;
5. calculate the number of complete records.

In Python:

```python
prof_len = int(first_header[28])
rec_len = 30 + 2 * prof_len
n_records = filesize // (rec_len * 4)
```

In MATLAB:

```matlab
prof_len = first_header(29);
rec_len = 30 + 2*prof_len;
n_records = floor(filesize/(rec_len*4));
```

This approach is preferable to assuming that all scans contain the same number of pulses.

---

# 11. Python example

The following Python function reads a complete REAL BSCAN file using NumPy.

It returns:

- all 30-word headers;
- all parallel-polarization profiles;
- all perpendicular-polarization profiles;
- the range coordinate.

```python
from pathlib import Path
import numpy as np


def read_bscan(filename):
    """
    Read a REAL BSCAN raw data file.

    Parameters
    ----------
    filename : str or pathlib.Path
        Path to a .bscan file.

    Returns
    -------
    headers : ndarray, shape (n_records, 30)
        Header values for every laser-pulse record.

    parallel : ndarray, shape (n_records, profile_length)
        Parallel-polarization raw digitizer-count profiles.

    perpendicular : ndarray, shape (n_records, profile_length)
        Perpendicular-polarization raw digitizer-count profiles.

    ranges : ndarray, shape (profile_length,)
        Range coordinate in meters.
    """

    filename = Path(filename)
    filesize = filename.stat().st_size

    first_header = np.fromfile(
        filename,
        dtype=">f4",
        count=30
    )

    if first_header.size != 30:
        raise ValueError("File is too short to contain a complete BSCAN header.")

    profile_length = int(first_header[28])
    record_length = 30 + 2 * profile_length
    record_bytes = record_length * 4

    n_records = filesize // record_bytes

    if n_records < 1:
        raise ValueError("File does not contain a complete BSCAN record.")

    if filesize % record_bytes != 0:
        print(
            "Warning: file size is not an exact multiple of the "
            "calculated BSCAN record length."
        )

    data = np.fromfile(
        filename,
        dtype=">f4",
        count=n_records * record_length
    )

    records = data.reshape((n_records, record_length))

    headers = records[:, 0:30]
    parallel = records[:, 30:30 + profile_length]
    perpendicular = records[
        :,
        30 + profile_length : 30 + 2 * profile_length
    ]

    r0 = first_header[20]
    dr = first_header[21]
    ranges = r0 + np.arange(profile_length) * dr

    return headers, parallel, perpendicular, ranges


if __name__ == "__main__":
    filename = "REAL.20230818_211736.bscan"

    headers, parallel, perpendicular, ranges = read_bscan(filename)

    print("records:", headers.shape[0])
    print("profile length:", parallel.shape[1])

    print("first date:", headers[0, 11:14])
    print("first time:", headers[0, 0:3])

    print("last date:", headers[-1, 11:14])
    print("last time:", headers[-1, 0:3])

    print("first range:", ranges[0], "m")
    print("range spacing:", ranges[1] - ranges[0], "m")
```

## Python indexing reminder

Python and NumPy use zero-based indexing.

Therefore:

```python
headers[:, 6]
```

is the azimuth field, while:

```python
headers[:, 7]
```

is the elevation field.

The profile length is:

```python
first_header[28]
```

not `first_header[29]`.

---

# 12. MATLAB example

The following MATLAB example reads the same BSCAN structure.

```matlab
% Simple MATLAB reader for REAL BSCAN raw data files.

fn = 'REAL.20230818_211736.bscan';

fid = fopen(fn,'r','b');   % 'b' means big-endian

if fid < 0
    error('Could not open file: %s', fn);
end

first_header = fread(fid,30,'single');

if numel(first_header) ~= 30
    fclose(fid);
    error('File is too short to contain a complete BSCAN header.');
end

fseek(fid,0,'eof');
filesize = ftell(fid);
fclose(fid);

% MATLAB uses one-based indexing.
profile_length = first_header(29);

record_length = 30 + 2*profile_length;
record_bytes = record_length * 4;

n_records = floor(filesize / record_bytes);

if mod(filesize,record_bytes) ~= 0
    warning(['File size is not an exact multiple of the ' ...
             'calculated BSCAN record length.']);
end

% Construct the range coordinate.
r0 = first_header(21);
dr = first_header(22);

range = r0 + (0:profile_length-1)*dr;

% Allocate arrays.
headers = nan(n_records,30);
parallel = nan(n_records,profile_length);
perpendicular = nan(n_records,profile_length);

% Read every record.
fid = fopen(fn,'r','b');

for i = 1:n_records

    record = fread(fid,record_length,'single');

    if numel(record) < record_length
        warning('Incomplete record encountered at record %d.',i);
        break
    end

    headers(i,:) = record(1:30);

    parallel(i,:) = ...
        record(31:30+profile_length);

    perpendicular(i,:) = ...
        record(31+profile_length:30+2*profile_length);

end

fclose(fid);

% Example header values from the first record.
hour      = headers(1,1);
minute    = headers(1,2);
second    = headers(1,3);

azimuth   = headers(1,7);
elevation = headers(1,8);

month     = headers(1,12);
day       = headers(1,13);
year      = headers(1,14);

fprintf('Date: %04.0f-%02.0f-%02.0f\n',year,month,day);
fprintf('Time: %02.0f:%02.0f:%06.3f\n',hour,minute,second);
fprintf('Azimuth: %.3f deg\n',azimuth);
fprintf('Elevation: %.3f deg\n',elevation);
fprintf('Number of records: %d\n',n_records);
fprintf('Profile length: %d\n',profile_length);
```

## MATLAB indexing reminder

MATLAB uses one-based indexing.

The profile length is:

```matlab
first_header(29)
```

The azimuth is:

```matlab
header(7)
```

and the elevation is:

```matlab
header(8)
```

Do not copy Python zero-based header indices directly into MATLAB code.

---

# 13. Reading a single record directly

It is sometimes useful to seek directly to a particular record rather than reading the entire file.

If `irec` is a MATLAB one-based record number:

```matlab
fid = fopen(fn,'r','b');

offset = (irec-1) * record_length * 4;

fseek(fid,offset,'bof');

record = fread(fid,record_length,'single');

fclose(fid);

header = record(1:30);

parallel = ...
    record(31:30+profile_length);

perpendicular = ...
    record(31+profile_length:30+2*profile_length);
```

The factor of 4 converts words to bytes.

---

# 14. Sanity-check file

A useful M2HATS test file is:

```text
REAL.20230818_211736.bscan
```

Known values from the first record are approximately:

```text
date: 18 August 2023
time: 21:17:38.880 UTC
```

Known values from the last record are approximately:

```text
date: 18 August 2023
time: 21:17:53.822 UTC
```

These values are useful for testing:

- byte order;
- record length;
- indexing;
- number of records;
- correct traversal from the first record to the last.

For the same M2HATS configuration, values such as the following are also reasonable sanity checks:

```text
profile length: 4200 samples
range spacing: 1.5 m
first range: approximately -567 m
```

If these quantities are wildly wrong, the most likely causes are an incorrect byte order, incorrect header indexing, or an incorrect record-length calculation.

---

# 15. Common mistakes

## 15.1 Reading with the wrong byte order

BSCAN files are **big-endian**.

MATLAB:

```matlab
fopen(fn,'r','b')
```

Python:

```python
np.fromfile(filename, dtype=">f4")
```

If the byte order is wrong, the header values will usually be obviously nonsensical.

## 15.2 Confusing zero-based and one-based indexing

| Quantity | Python | MATLAB |
|---|---|---|
| Azimuth | `header[6]` | `header(7)` |
| Elevation | `header[7]` | `header(8)` |
| Range of first sample | `header[20]` | `header(21)` |
| Range spacing | `header[21]` | `header(22)` |
| Profile length | `header[28]` | `header(29)` |

## 15.3 Assuming every file has the same number of records

Do not assume that every M2HATS BSCAN file contains exactly 150 records.

Calculate the number of complete records from the actual file size and record length.

## 15.4 Forgetting the second polarization profile

Each record contains **two** complete profiles:

```text
header
parallel
perpendicular
```

A reader that extracts only the first profile is not reading the complete REAL raw record.

## 15.5 Treating the M2HATS latitude and longitude header fields as valid

For the M2HATS files:

```text
header[18]
header[19]
```

contain Chico coordinates rather than the Tonopah/M2HATS deployment coordinates.

Do not use them for M2HATS geolocation.

## 15.6 Assuming the raw values are already processed backscatter

The profile arrays contain raw digitizer counts.

They still contain background signal and have not yet undergone the processing normally used to create scientific REAL aerosol images.

Reading a BSCAN file correctly is therefore **not the same thing as producing a processed REAL image**.

---

# 16. What is and is not in a raw BSCAN file

## A raw BSCAN file contains

- a sequence of laser-pulse records;
- one 30-word header for every pulse;
- pulse time information;
- scanner azimuth and elevation where correctly populated;
- range-coordinate information;
- two raw polarization-channel waveforms;
- the original digitizer-count profiles.

## A raw BSCAN file does not already contain

- background-subtracted signals;
- combined total-polarization signal;
- range-squared-corrected scattering intensity;
- decibel-converted scattering intensity;
- high-pass median-filtered imagery;
- Cartesian-gridded PPI images;
- optical-flow wind fields.

Those are derived products or processing steps.

---

# 17. The next step after reading the file

Successfully reading a BSCAN file produces:

1. pulse-by-pulse header information;
2. raw parallel-polarization digitizer counts;
3. raw perpendicular-polarization digitizer counts;
4. a range coordinate.

Additional processing is required before the raw profiles become the relative aerosol-scattering fields commonly displayed in REAL imagery.

That processing may include steps such as:

- estimating electronic/background levels;
- subtracting the background from each polarization channel;
- combining polarization channels when appropriate;
- compensating for the inverse-square range dependence of the lidar return;
- converting the resulting quantity to a logarithmic or decibel representation;
- applying filtering to emphasize spatial structure;
- and, when needed, mapping the polar measurements onto a Cartesian grid.

Those operations are deliberately **not documented here**, because they are conceptually separate from the BSCAN file format itself.

They should be described in a subsequent REAL data-processing Markdown document so that a reader or AI does not confuse:

> **what was originally recorded**

with:

> **what was later derived from the recorded signals**.

---

# 18. Related documentation

For a description of the REAL instrument itself, see:

```text
REAL_instrument_description.md
```

For the original technical note describing the BSCAN format, see:

https://lidar.csuchico.edu/How_To/reading_REAL_bscan_files.pdf

A separate future Markdown document should describe the processing of raw REAL BSCAN profiles into relative aerosol-scattering intensity and the image products used for scientific analysis.

---

# 19. Minimal checklist for a new BSCAN reader

Before trusting a new reader, verify all of the following:

- [ ] The file is being read as **big-endian**.
- [ ] Every value is being interpreted as a 32-bit floating-point number.
- [ ] The first 30 words are interpreted as the header.
- [ ] The profile length is obtained from header index 28 zero-based / word 29 MATLAB.
- [ ] Record length is calculated as `30 + 2*profile_length`.
- [ ] The number of records is calculated from the actual file size.
- [ ] Both polarization profiles are extracted.
- [ ] The range coordinate is built from the first-range and range-spacing header fields.
- [ ] Python/MATLAB indexing differences are handled correctly.
- [ ] M2HATS latitude and longitude header fields are not trusted.
- [ ] A known file such as `REAL.20230818_211736.bscan` produces reasonable date, time, range, and profile-length values.
- [ ] Raw digitizer counts are not mistaken for processed aerosol-scattering intensity.

If all of these checks pass, the BSCAN file is probably being interpreted correctly.
