# M2HATS public data locations

## Purpose of this document

This file is a practical map of the public M2HATS data and derived products served by the CSU Chico Atmospheric Lidar Research Group at:

```text
https://lidar.csuchico.edu/
```

It is intended for people, students, collaborators, and AI-assisted workflows that need to answer questions such as:

- Where are the raw REAL data?
- Where are the NCAR ISFS netCDF files?
- Where can I find pre-made REAL images?
- Where are the Typhoon wind products?
- Where are the daily meteorological context pages?
- Which files are raw observations and which are derived products?

This document describes **where things are**. More detailed scientific and technical documentation is provided separately.

---

# 1. Two complementary ways to obtain M2HATS data

The complete M2HATS field-project archive is maintained by NSF NCAR's Earth Observing Laboratory (EOL):

```text
https://data.eol.ucar.edu/project/636/datasets
```

M2HATS data in the EOL archive are public. NCAR/EOL provides the long-term field-project archive, dataset documentation, and access to the complete collection of participating investigators' and NCAR observing-system data.

The Chico State lidar server provides a **complementary access method** for a growing subset of those data and for CSU Chico REAL products:

```text
https://lidar.csuchico.edu/M2HATS-data/
```

The practical difference is important.

## NCAR/EOL archive access

The NCAR/EOL archive is the appropriate place to find the complete M2HATS field-project data collection and its authoritative long-term archive.

For many large datasets, the usual workflow is approximately:

```text
select/request a dataset
        |
        v
EOL prepares or stages the requested files
        |
        v
user is notified when the request is ready
        |
        v
user downloads an archive or collection of files
        |
        v
files are stored and analyzed on the user's own computer or server
```

This is a good model for obtaining a complete dataset for long-term local use.

## Direct access from the Chico State lidar server

The CSU Chico server is intended to make many individual M2HATS files available directly over ordinary HTTPS.

Where the file format and software permit it, MATLAB, Python, and other tools can read data **directly from the web server** without first downloading and storing an entire dataset locally.

Conceptually:

```text
analysis program
        |
        v
https://lidar.csuchico.edu/...
        |
        v
open/read only the file or data needed
        |
        v
analyze in memory
```

This can be especially useful when:

- only a few times or events are needed;
- a user wants to inspect a dataset before deciding what to download;
- students are working on computers with limited local storage;
- scripts need reproducible access to individual public files;
- an AI assistant is helping a user write code to read a particular M2HATS file;
- copying a very large archive would be unnecessary.

The Chico server should **not** be assumed to contain every M2HATS dataset available from NCAR/EOL. It is a selectively organized public access point that emphasizes REAL data, frequently used NCAR data, derived products, examples, and direct file-level access.

When a complete or authoritative field-project archive is needed, use the NCAR/EOL M2HATS archive. When selective direct access is more convenient, the Chico State server may be the easier route.

---

# 2. Quick start

If you already know what you want, start here.

| If you want... | Go here |
|---|---|
| Raw REAL BSCAN files | `https://lidar.csuchico.edu/M2HATS-data/REAL/bscans/` |
| REAL housekeeping / instrument-health files | `https://lidar.csuchico.edu/M2HATS-data/REAL/hk/` |
| NCAR ISFS netCDF files | `https://lidar.csuchico.edu/M2HATS-data/NCAR/ISFS/` |
| NCAR ISS data | `https://lidar.csuchico.edu/M2HATS-data/NCAR/ISS/` |
| High-pass median-filtered REAL PPI PNG images | `https://lidar.csuchico.edu/m2hats/images/scans/` |
| Unfiltered REAL PPI PNG images | `https://lidar.csuchico.edu/m2hats/images/scans_unfiltered/` |
| PNG images of Typhoon horizontal wind-vector fields | `https://lidar.csuchico.edu/m2hats/images/winds/` |
| Full-day REAL time-lapse movies | `https://lidar.csuchico.edu/m2hats/movies/` |
| Existing Typhoon netCDF archives | `https://lidar.csuchico.edu/m2hats/typhoon_nc/` |
| Daily meteorological/context pages | `https://lidar.csuchico.edu/m2hats/kgm/` |
| Main human-readable M2HATS page | `https://lidar.csuchico.edu/m2hats/` |

---

# 3. Two parts of the public site

The public M2HATS material currently lives in two main parts of the web server.

## 2.1 `/M2HATS-data/`

This area is primarily the underlying public data repository:

```text
https://lidar.csuchico.edu/M2HATS-data/
```

It currently contains major branches for:

```text
REAL/
NCAR/
```

This area should be thought of mainly as the place to obtain observational data files.

## 2.2 `/m2hats/`

This area is the more human-facing part of the site:

```text
https://lidar.csuchico.edu/m2hats/
```

It contains material such as:

- pre-made images;
- animations;
- Typhoon products;
- daily context pages;
- documentation;
- software and guides.

The long-term goal is for `/m2hats/` to serve as the human-readable front door while `/M2HATS-data/` remains the underlying file repository.

---

# 4. Raw REAL BSCAN files

Public location:

```text
https://lidar.csuchico.edu/M2HATS-data/REAL/bscans/
```

These are the original raw REAL lidar data files.

The directory is organized first by month:

```text
202307/
202308/
202309/
```

and then by day:

```text
YYYYMMDD/
```

For example:

```text
https://lidar.csuchico.edu/M2HATS-data/REAL/bscans/202308/20230818/
```

The BSCAN filenames contain date and UTC time. A typical filename is:

```text
REAL.20230818_211736.bscan
```

The raw files contain pulse-by-pulse:

- a 30-word header;
- parallel-polarization digitizer data;
- perpendicular-polarization digitizer data.

For details of the binary structure and how to read the files, see:

```text
REAL_bscan_data_file_format.md
```

For the subsequent signal processing used to create relative-backscatter imagery, see:

```text
REAL_bscan_signal_processing.md
```

For a description of the lidar itself, see:

```text
REAL_instrument_description.md
```

---

# 5. REAL housekeeping files

Public location:

```text
https://lidar.csuchico.edu/M2HATS-data/REAL/hk/
```

These are REAL **housekeeping** files.

"Housekeeping" is the term used for instrument status or **lidar vital-sign information**. These data were useful during M2HATS for checking the health and operating status of the lidar, including times when the operators were not physically at the field site.

The public directory is organized in the same basic month/day pattern as the BSCAN archive:

```text
YYYYMM/
    YYYYMMDD/
```

For example:

```text
https://lidar.csuchico.edu/M2HATS-data/REAL/hk/202308/20230818/
```

A separate technical description of the housekeeping-file contents may be added later if there is a need for users to analyze these files directly.

At present, these files should primarily be regarded as **instrument-health and operational-support data**, not as the main atmospheric-science data product.

---

# 6. NCAR ISFS data

Public location:

```text
https://lidar.csuchico.edu/M2HATS-data/NCAR/ISFS/
```

ISFS stands for the **Integrated Surface Flux System**.

This directory contains high-rate and processed data from the dense near-surface in-situ observing system used during M2HATS, including the sonic-anemometer measurements that are central to much of the experiment.

The directory currently includes several generations or processing versions of the data, including directories such as:

```text
qc_geo_hr_07/
qc_geo_hr_08/
qc_geo_hr_09/

qc_geo_tiltcor_hr_07/
qc_geo_tiltcor_hr_08/
qc_geo_tiltcor_hr_09/

qc_geo_tiltcor_hr_5min/
```

It also contains an NCAR/ISFS data report:

```text
Data-report_M2HATS_ISFS_Flux.pdf
```

## Example high-rate filename

A typical hourly netCDF filename is:

```text
isfs_m2hats_qc_geo_hr_20230801_00.nc
```

The filename encodes:

```text
experiment + processing + date + hour
```

The high-rate August directory, for example, contains one large netCDF file for each UTC hour.

Some month directories also contain compressed archives of the complete monthly collection.

## Important caution

There are now both older `qc_geo_hr_*` directories and newer **tilt-corrected** `qc_geo_tiltcor_*` products.

A future document:

```text
M2HATS_ISFS_data.md
```

should explain which version should normally be used, what corrections have been applied, the important variable names and units, the sonic-array organization, and any quality-control caveats.

Do not choose among these products based only on the directory name if the scientific distinction matters.

---

# 7. NCAR ISS data

Public location:

```text
https://lidar.csuchico.edu/M2HATS-data/NCAR/ISS/
```

ISS stands for the **Integrated Sounding System**.

This area is still incomplete.

At present, the public ISS branch contains:

```text
PurpleAir/
```

with subdirectories including:

```text
10m/
Kims_work/
t44/
```

Additional NCAR ISS data products may be added in the future.

Therefore, this directory should currently be described as a **growing public collection**, not as the complete M2HATS ISS archive.

---

# 8. High-pass median-filtered REAL PPI images

Public location:

```text
https://lidar.csuchico.edu/m2hats/images/scans/
```

These are pre-made PNG images of REAL PPI scans processed to emphasize fine-scale aerosol structure.

The images use the high-pass median-filter processing described in:

```text
REAL_bscan_signal_processing.md
```

The directory is organized by month and day:

```text
YYYYMM/
    YYYYMMDD/
```

For example:

```text
https://lidar.csuchico.edu/m2hats/images/scans/202308/20230818/
```

These images are often the easiest place to begin if the goal is simply to inspect spatial aerosol structure without reading raw BSCAN files.

They are **derived products**, not raw observations.

---

# 9. Unfiltered REAL PPI images

Public location:

```text
https://lidar.csuchico.edu/m2hats/images/scans_unfiltered/
```

These are pre-made PNG images of REAL PPI scans without the final median-filter/high-pass operation.

The term **unfiltered** should not be interpreted as "raw."

The underlying BSCAN data have already undergone the essential processing needed to create relative-backscatter imagery, including background subtraction, range-squared correction, and logarithmic conversion.

For the exact distinction between the unfiltered and high-pass-filtered products, see:

```text
REAL_bscan_signal_processing.md
```

The current public collection begins in August 2023 and is organized by month and day.

---

# 10. Typhoon wind-vector PNG images

Public location:

```text
https://lidar.csuchico.edu/m2hats/images/winds/
```

These are PNG images showing horizontal wind-vector fields estimated from sequential REAL aerosol images using the **Typhoon** motion-estimation system.

The directory is organized by month and day.

For example:

```text
https://lidar.csuchico.edu/m2hats/images/winds/202308/20230818/
```

These wind fields are **derived products**.

REAL itself is not a Doppler lidar and does not directly measure wind velocity. Typhoon estimates horizontal motion by tracking aerosol structures between sequential REAL scans.

A future document should describe:

```text
Typhoon_overview.md
```

and another should document the M2HATS-specific Typhoon products:

```text
M2HATS_Typhoon_data.md
```

---

# 11. REAL time-lapse movies

Public location:

```text
https://lidar.csuchico.edu/m2hats/movies/
```

The most broadly useful current collection is:

```text
https://lidar.csuchico.edu/m2hats/movies/full_day_UTC_filtered_movies/
```

These are full-day MP4 animations made from high-pass median-filtered REAL scans.

A typical filename is:

```text
REAL_20230818_hpmf.mp4
```

where:

```text
hpmf = high-pass median filtered
```

These movies are particularly useful for seeing the time evolution and advection of coherent aerosol structures that may not be obvious from individual scans.

Other subdirectories contain movies prepared for particular papers, presentations, or meetings. These should not automatically be interpreted as the canonical full archive.

---

# 12. Existing Typhoon netCDF archives

Public location:

```text
https://lidar.csuchico.edu/m2hats/typhoon_nc/
```

This directory currently contains several large compressed archives of M2HATS REAL/Typhoon products.

Examples include filenames containing terms such as:

```text
alphaExperiments
stationnaryCases
FULL
MEAN
NONE
REF
initREF
```

and several values of an `alpha` parameter.

These files were produced during earlier Typhoon processing and experimentation and may include products prepared for collaborators, including work with Channing Tong's group.

## Important caution

This collection has **not yet been re-audited for this documentation project**.

At present, do not assume that:

- it is complete;
- every archive represents a final product;
- all archives use identical processing;
- every filename term is obvious or self-documenting;
- the directory contains the preferred Typhoon product for every M2HATS period.

Before these files are presented as a well-defined public scientific data product, the archives should be inspected and documented.

That later work should produce:

```text
M2HATS_Typhoon_data.md
```

with explicit descriptions of:

- file contents;
- netCDF variables;
- coordinate systems;
- time conventions;
- processing parameters;
- alpha values;
- differences among FULL, MEAN, NONE, REF, and initREF cases;
- completeness;
- known limitations;
- recommended products.

Until then, this directory should be treated as an **existing public but provisional Typhoon-product collection**.

---

# 13. Daily M2HATS context pages

Public index:

```text
https://lidar.csuchico.edu/m2hats/kgm/
```

These pages provide a day-by-day meteorological overview for much of the M2HATS field campaign.

They are organized by date and are intended mainly for visual context rather than as a primary numerical-data archive.

A daily page may include images from several observing systems and external meteorological products.

For example, the page for 18 August 2023 includes:

- 17 UTC radiosonde;
- 22 UTC radiosonde;
- 17 UTC GOES-West visible satellite image;
- 22 UTC GOES-West visible satellite image;
- 12 UTC 500-mb upper-air chart;
- 00 UTC 500-mb upper-air chart;
- micropulse DIAL aerosol time-height imagery;
- 915-MHz radar wind-profiler imagery;
- 449-MHz radar wind-profiler imagery;
- Doppler-lidar imagery;
- micropulse DIAL virtual-potential-temperature imagery.

The date-index page currently covers dates from late July through late September 2023.

These pages are useful for answering questions such as:

- What was the boundary-layer depth?
- Was convection present?
- What was the synoptic situation?
- What did the vertical wind profile look like?
- Were clouds or aerosol layers present?
- What broader atmospheric conditions surrounded an interesting REAL event?

They are therefore useful companions to the high-resolution near-surface REAL and ISFS observations.

---

# 14. Raw observations versus derived products

It is useful to distinguish among three broad categories.

## Original or near-original observations

Examples:

```text
REAL BSCAN files
REAL housekeeping files
NCAR ISFS netCDF files
NCAR ISS files
```

These are closest to the underlying measurements, although NCAR netCDF products may already include quality control, coordinate rotation, tilt correction, or other processing documented by NCAR.

## Derived numerical products

Examples:

```text
Typhoon netCDF wind fields
```

These have been produced by applying retrieval or analysis methods to original observations.

## Derived visualization products

Examples:

```text
high-pass REAL PNG images
unfiltered REAL PNG images
Typhoon vector-field PNG images
MP4 animations
daily context pages
```

These are convenient for inspection and interpretation but are not substitutes for the underlying data when quantitative analysis is required.

---

# 15. Suggested starting points

## If you simply want to look at REAL data

Start with:

```text
https://lidar.csuchico.edu/m2hats/images/scans/
```

or the full-day movies:

```text
https://lidar.csuchico.edu/m2hats/movies/full_day_UTC_filtered_movies/
```

## If you want to process REAL yourself

Read, in this order:

```text
REAL_instrument_description.md
REAL_bscan_data_file_format.md
REAL_bscan_signal_processing.md
```

Then obtain raw BSCAN files from:

```text
https://lidar.csuchico.edu/M2HATS-data/REAL/bscans/
```

## If you want near-surface turbulence measurements

Start with:

```text
https://lidar.csuchico.edu/M2HATS-data/NCAR/ISFS/
```

but consult the forthcoming ISFS documentation before choosing among the older and tilt-corrected processing versions.

## If you want environmental context for an interesting event

Start with the date-specific pages under:

```text
https://lidar.csuchico.edu/m2hats/kgm/
```

## If you want existing motion-estimated wind fields

Preview the PNG products under:

```text
https://lidar.csuchico.edu/m2hats/images/winds/
```

The existing netCDF archives are under:

```text
https://lidar.csuchico.edu/m2hats/typhoon_nc/
```

but should be treated cautiously until the Typhoon archive is documented.

---

# 16. Current documentation status

At the time this file was written, the following REAL documentation had been assembled:

```text
REAL_instrument_description.md
REAL_bscan_data_file_format.md
REAL_bscan_signal_processing.md
```

Likely next documentation priorities are:

```text
M2HATS_overview.md
M2HATS_ISFS_data.md
M2HATS_ISFS_examples.md
Typhoon_overview.md
M2HATS_Typhoon_data.md
```

The public archive will continue to evolve, especially the NCAR ISS area and the documentation of existing Typhoon products.

---

# 17. A note for AI-assisted use

When using an AI assistant with these data, provide the relevant Markdown documentation along with the scientific question.

For example:

- for raw REAL BSCAN work, provide the three REAL documents listed above;
- for NCAR ISFS work, provide the future ISFS data guide and examples;
- for Typhoon products, provide the future Typhoon overview and M2HATS Typhoon-data guide.

Exact URLs, filenames, variable names, units, and processing caveats are more reliable than asking an AI to infer the structure of the archive from generic lidar or boundary-layer knowledge.

If the documentation and a file disagree, inspect the actual file and campaign metadata rather than assuming the documentation is infallible.
