Long term North American PM data sets representative of urban and rural air quality are being created by integrating data from research and routine fine particle monitoring networks that are or have been in operation. The data sets will be composed of fine, coarse, and PM10 mass, elemental composition, organics and ions. In addition, visibility and extinction and scatting coefficinets will be included.

The process for creating the data sets involves gathering the data from various data suppliers, reformatting the data to a standard format and adding location, variable and data sampling metadata. The data and metadata will be available for each individual network including all locations and variables, as well as integrated datasets where select variables are grouped into one data set from select networks.

This is an on going project with intermediate data sets periodically created. This report describes the currently available data sets networks and providers, the processing of the data for integration, and format of the final data sets.

This project is being supported by EPA’s Office of Air Quality Planning and Standards (OAQPS).

Note: The elements included in this data set are only those that are include in the IMPROVE data. However, the organic variables from all data sets are included.

These data are provided as was received from their sources. No quality control of the data, beyond that done by the supplying organizations, has been performed on the data. Some networks corrected their data to a reference temperature and pressure, others do not. No attempt has yet been made to reconcile these differences.

The data networks and their suppliers to be included in the North American integrated data sets are listed in tables 1-3:

Table 1. Data sets processed.

1/1/87 – 2/18/87

Table 2. Data sets to be processed.

Table 3. Data sets on order.

A relational database for each network’s data is created consisting of a main Data table containing all of the data, a Location table containing the location metadata, and a Variable table containing the variable metadata. The data are then passed through a set of standardization routines that creat a data set ready for integration with all other processed data.

The standardization process includes:

• converting units to ng/m³
• converting NULL values to –9999
• converting dates to the format: mm/dd/yyyy hh:mm
• converting all known values below the instrument minimum detection (MDL) limit to ½ the MDL.
• creating a standard location code for each monitoring site (see below)
• creating a standard variable code for each variable (see below)
• converting the data set flags (if available) to:

A sample Data table is presented in Table 2. The first two columns, Loc_Code and Date are the Key or dimensional fields identifying the monitor and sample date and time for each data record respectively. Additional information about the monitoring site is contained in the Location table. Each of the remaining columns contain the data for a single variable. The variable code (Var_Code ) is used for the column name, additional variable metadata is located in the Variable Table.

Table 4. A sample Data Table.

(------------Key Fields----------------) (----------- ---------------Variable Fields----------------------------)

The Location table is made up of the location code (Loc_Code) followed by station name and location information (Table 5). Each station is assigned a unique Loc_Code based upon the "Station ID" format used in the Canadian NAtChem – Particle database. Encoded in the location code is the network as well as location information (see Table 6). The Loc-Code for monitoring sites with AIRS site codes use characters 1-4 for the Network abbreviation and 5-13 for the AIRS site code. The Location table’s StationID field contains the original location identifier that comes with the database.

Table 5. A sample Location Table.

Table 6. The Loc_Code encoding definitions.

The Variable table is made up of the variable abbreviation (Var_Abbr) followed by the variable name and sampler information (Table 7). Each variable is assigned a unique Var_Abbr. Encoded in the Var_Abbr is the species type and sampler and analysis methods. Table 8 lists the meaning of each character in the Var_Abbr.

Table 7. A sample Variable Table.

Table 8. The Var_Abbr encoding definitions. The meaning of the abbreviations and codes are listed in the Variable Table.

No quality control of the data beyond that done by the supplying organization has been performed on the data sets. However, as the data are used and problems identified appropriate procedures to remedy the problems in the data sets will be conducted.

The data are provided in three file formats:

• Voyager
• Fixed Length ASCII
• MS Access

Associated with each fixed length ascii file is a data dictionary which gives the field names, the number of bytes and NULL value. The order of the fields in the dictionary from top to bottom is the order of the fields in the ASCII files from left to right.