# M2HATS public data and documentation

Welcome to the public documentation for the **Multipoint Monin-Obukhov Similarity Theory Horizontal Array Turbulence Study (M2HATS)**.

M2HATS was conducted near **Tonopah, Nevada, in 2023** and combined a dense 50-sonic horizontal array with scanning aerosol lidar, Doppler lidars, radar wind profilers, thermodynamic profilers, radiosondes, distributed temperature sensing, and other measurements.

This documentation is intended to help:

- researchers;
- students;
- collaborators;
- programmers;
- and AI-assisted analysis workflows

understand and use the public M2HATS data.

---

## Start here

If you are new to M2HATS, begin with:

### 1. Understand the experiment

[**M2HATS_overview.md**](M2HATS_overview.md)

Explains:

- why M2HATS was conducted;
- where and when it took place;
- the 50-sonic horizontal array;
- REAL and the intentional lidar/sonic geometry;
- the major NCAR observing systems;
- and how the measurements fit together scientifically.

### 2. Find the data

[**M2HATS_data_locations.md**](M2HATS_data_locations.md)

Explains:

- where the public data are located;
- which directories contain raw observations versus derived products;
- how the Chico State server differs from the NSF NCAR/EOL archive;
- and where to find REAL, ISFS, ISS, Typhoon, images, movies, and daily context products.

---

# Documentation by data type

## NCAR/EOL ISFS surface and turbulence data

[**M2HATS_ISFS_data.md**](M2HATS_ISFS_data.md)

Use this if you want to work with:

- the 50-sonic horizontal array;
- multilevel flux towers;
- high-rate turbulence data;
- 5-minute meteorological products;
- temperature, wind, pressure, humidity, fluxes, and related variables.

The guide explains:

- which ISFS product to use;
- `qc_geo_hr` versus `qc_geo_tiltcor_hr`;
- high-rate versus 5-minute files;
- variable naming;
- sonic sampling rates;
- time conventions;
- `_FillValue`;
- `ldiag_*` quality screening;
- and direct MATLAB access from the Chico server.

---

# REAL aerosol lidar documentation

## What REAL is

[**REAL_instrument_description.md**](REAL_instrument_description.md)

Explains:

- what REAL measures;
- what REAL does not measure;
- transmitter and receiver basics;
- polarization channels;
- PPI and RHI scanning;
- sequential scan timing;
- spatial sampling and resolution;
- and how aerosol structure can be used to estimate wind indirectly.

---

## Raw REAL BSCAN file format

[**REAL_bscan_data_file_format.md**](REAL_bscan_data_file_format.md)

Use this if you want to read the original REAL raw files.

Explains:

- BSCAN record structure;
- big-endian 32-bit floating-point storage;
- the 30-word header;
- the two polarization profiles;
- range construction;
- M2HATS-specific header caveats;
- Python and MATLAB reading examples;
- and common mistakes.

Raw M2HATS REAL BSCAN files are publicly available at:

```text
https://lidar.csuchico.edu/M2HATS-data/REAL/bscans/
```

---

## REAL signal processing

[**REAL_bscan_signal_processing.md**](REAL_bscan_signal_processing.md)

Explains how raw REAL digitizer counts are converted into the relative aerosol-backscatter fields commonly displayed as REAL images.

The documented processing includes:

- combining polarization channels;
- estimating pulse-by-pulse background;
- background subtraction;
- range-squared correction;
- logarithmic / dB conversion;
- optional radial median filtering;
- scan-type identification;
- PPI/RHI coordinate mapping;
- and the distinction between native coordinate mapping and true Cartesian interpolation.

---

# Typhoon wind-field documentation

## What Typhoon is

[**Typhoon_overview.md**](Typhoon_overview.md)

Explains:

- the scientific origin of Typhoon;
- Pierre Dérian's wavelet-based optical-flow work;
- dense motion estimation;
- aerosol-pattern tracking;
- regularization;
- valid-data masking;
- scan-motion correction;
- GPU/CUDA acceleration;
- validation against Doppler lidar;
- the distinction between the reference Typhoon implementation and public PyTyphoon.

---

## M2HATS Typhoon output data

[**M2HATS_Typhoon_data.md**](M2HATS_Typhoon_data.md)

Use this if you want to work with existing M2HATS Typhoon vector flow fields.

Explains:

- Typhoon netCDF structure;
- `x`, `y`, `vx`, `vy`, and `indices`;
- Cartesian grid reconstruction;
- reference times;
- source REAL scans;
- processing provenance;
- `FULL`, `MEAN`, `NONE`, `REF`, and `initREF`;
- Sasano scan-motion correction;
- Python and MATLAB examples;
- and how to compare different processing cases safely.

Public Typhoon-related products are currently available at:

```text
https://lidar.csuchico.edu/m2hats/images/winds/
https://lidar.csuchico.edu/m2hats/typhoon_nc/
```

---

# Quick navigation by task

| I want to... | Read this |
|---|---|
| Understand the experiment | [M2HATS_overview.md](M2HATS_overview.md) |
| Find public data | [M2HATS_data_locations.md](M2HATS_data_locations.md) |
| Use NCAR sonic / flux data | [M2HATS_ISFS_data.md](M2HATS_ISFS_data.md) |
| Understand REAL | [REAL_instrument_description.md](REAL_instrument_description.md) |
| Read raw REAL BSCAN files | [REAL_bscan_data_file_format.md](REAL_bscan_data_file_format.md) |
| Process REAL raw signals | [REAL_bscan_signal_processing.md](REAL_bscan_signal_processing.md) |
| Understand Typhoon | [Typhoon_overview.md](Typhoon_overview.md) |
| Read Typhoon vector fields | [M2HATS_Typhoon_data.md](M2HATS_Typhoon_data.md) |

---

# Public data access

The complete M2HATS field-project archive is maintained by **NSF NCAR/EOL**:

```text
https://data.eol.ucar.edu/project/636/datasets
```

The CSU Chico Atmospheric Lidar Research Group also provides direct public access to many M2HATS files and derived products:

```text
https://lidar.csuchico.edu/M2HATS-data/
https://lidar.csuchico.edu/m2hats/
```

The two access models are complementary.

NSF NCAR/EOL provides the authoritative long-term field-project archive.

The Chico State server emphasizes:

- direct file-level access over HTTPS;
- REAL raw data and derived products;
- frequently used NCAR data;
- documentation;
- images;
- movies;
- Typhoon results;
- and workflows where MATLAB or Python can read selected files directly from the web server.

Users therefore do not always need to download and store an entire large dataset before beginning an analysis.

---

# AI-assisted use

These Markdown files were written deliberately so that they can be read effectively by both humans and AI assistants.

For best results, give the AI the documents relevant to the task.

Examples:

## For raw REAL work

Provide:

```text
REAL_instrument_description.md
REAL_bscan_data_file_format.md
REAL_bscan_signal_processing.md
```

## For NCAR ISFS work

Provide:

```text
M2HATS_overview.md
M2HATS_data_locations.md
M2HATS_ISFS_data.md
```

For difficult sensor, calibration, or quality-control questions, also provide the authoritative NCAR/EOL ISFS data report.

## For Typhoon work

Provide:

```text
Typhoon_overview.md
M2HATS_Typhoon_data.md
```

and, when needed:

```text
REAL_bscan_signal_processing.md
```

A useful rule for AI-assisted analysis is:

> **Prefer exact file metadata, documented variable names, actual dimensions, and campaign-specific documentation over generic assumptions about lidar, turbulence, or netCDF data.**

---

# Raw observations versus derived products

Not everything on the server is the same kind of data.

## Original or near-original observations

Examples:

```text
REAL BSCAN files
REAL housekeeping files
NCAR ISFS netCDF files
NCAR ISS files
```

## Derived numerical products

Examples:

```text
Typhoon vector-flow netCDF files
```

## Derived visualization products

Examples:

```text
REAL PNG images
Typhoon vector-field PNG images
MP4 movies
daily meteorological context pages
```

For quantitative scientific analysis, use the numerical source data whenever possible.

---

# Main public locations

```text
M2HATS main page
https://lidar.csuchico.edu/m2hats/

Raw REAL BSCAN files
https://lidar.csuchico.edu/M2HATS-data/REAL/bscans/

REAL housekeeping files
https://lidar.csuchico.edu/M2HATS-data/REAL/hk/

NCAR ISFS data
https://lidar.csuchico.edu/M2HATS-data/NCAR/ISFS/

NCAR ISS data
https://lidar.csuchico.edu/M2HATS-data/NCAR/ISS/

High-pass-filtered REAL images
https://lidar.csuchico.edu/m2hats/images/scans/

Unfiltered REAL images
https://lidar.csuchico.edu/m2hats/images/scans_unfiltered/

Typhoon wind-field images
https://lidar.csuchico.edu/m2hats/images/winds/

REAL movies
https://lidar.csuchico.edu/m2hats/movies/

Typhoon netCDF archives
https://lidar.csuchico.edu/m2hats/typhoon_nc/

Daily meteorological context pages
https://lidar.csuchico.edu/m2hats/kgm/
```

---

# Recommended reading order

For someone approaching M2HATS for the first time:

```text
1. M2HATS_overview.md

2. M2HATS_data_locations.md

3. Choose a data path:

   ISFS:
       M2HATS_ISFS_data.md

   REAL:
       REAL_instrument_description.md
       REAL_bscan_data_file_format.md
       REAL_bscan_signal_processing.md

   Typhoon:
       Typhoon_overview.md
       M2HATS_Typhoon_data.md
```

---

# Documentation philosophy

The goal of this documentation is not to create an abstract software manual.

The goal is to make the real M2HATS data archive usable.

These documents therefore emphasize:

- exact public locations;
- actual filenames;
- real variable names;
- known working examples;
- experiment-specific caveats;
- provenance;
- scientific interpretation;
- and explicit warnings where assumptions could lead to incorrect results.

The documentation will continue to evolve as additional public M2HATS data, examples, and software are organized and verified.
