X2SYS_INIT

x2sys_init − Initialize data base for track data files

x2sys_init TAG −Ddeffile [ −Esuffix ] [ −F ] [ −Gd|g ] [ −Idx[/dy] ] [ −M[i|o][flag] ] [ −Rwest/east/south/north[r] ] [ −V ] [ −Wt|dgap ]

x2sys_init initializes a set of data bases that are particular to one kind of track data. These data, their associated data bases, and key parameters are given a short-hand notation called a system TAG. The TAG remembers settings such as file format, whether the data are geographic or not, and the binning resolution for track indexes. Running x2sys_init is a prerequisite to running x2sys_binlist which will create a crude representation of where data track go and which observations are available; this information serves as input to x2sys_put which updates the track data base. Then, x2sys_get can be used to find which tracks and data are available inside a given region.

TAG

The unique name of this data type system.

−D

Definition file for this data set.

No space between the option flag and the associated arguments.

−E

Specifies the file extension (suffix) for these data files. If not given we use the definition file prefix as the suffix (see −D).

−F

Force creating new files if old ones are present [Default will abort if old files are found].

−G

Selects geographical coordinates. Append d for discontinuity at the Dateline (makes longitude go from -180 to + 180) or g for discontinuity at Greenwhich (makes longitude go from 0 to 360 [Default]).

−I

x_inc [and optionally y_inc] is the grid spacing. Append m to indicate minutes or c to indicate seconds.

−M

Multiple segment file(s). Segments are separated by a special record. For ASCII files the first character must be flag [Default is ’>’]. For binary files all fields must be NaN.

−R

west, east, south, and north specify the Region of interest, and you may specify them in decimal degrees or in [+-]dd:mm[:ss.xxx][W|E|S|N] format. Append r if lower left and upper right map coordinates are given instead of wesn. The two shorthands −Rg −Rd stand for global domain (0/360 or -180/+180 in longitude respectively, with -90/+90 in latitude).

−V

Selects verbose mode, which will send progress reports to stderr [Default runs "silently"].

−W

Give t or d and append the corresponding maximum time gap (in user units; this is typically seconds [Infinity]), or distance gap (in user units; this is typically km [Infinity] ) allowed between the two points immediately on either side of a crossover.

The *.def files contain information about the data file format. The *.def files have two sections: (1) header information and (2) column information. All header information starts with the character # in the first column, immediately followed by an upper-case directive. If the directive takes an argument it is separated by white-space. You may append a trailing # comments. Five directives are recognized:

ASCII states that the data files are in ASCII format.
BINARY states that the data files are binary files.
SKIP takes an integer argument which is either the number of lines to skip (when reading ASCII files) or the number of bytes to skip (when reading binary files).
GEO indicates that these are geographic data set, with periodicities in the x-coordinate (longitudes).
MULTISEG means each track consists of multiple segments separated by a GMT multisegment header (alternatively, use −M when defining the system TAG).

The column information consists of one line per column in the order the columns appear in the data file. For each column you must provide seven attributes:

name type NaN NaN-proxy scale offset oformat

name is the name of the column variable. It is expected that you will use the special names lon (or x if Cartesian) and lat (or y) for the two required coordinate columns, and time when optional time data are present.
type is a for ASCII representations of numbers, whereas for binary files you may choose from c for signed 1-byte character (-127,+128), u for unsigned byte (0-255), h for signed 2-byte integers (-32768,+32767), i for signed 4-byte integers (-2,147,483,648,+2,147,483,647), f for 4-byte floating points and d for 8-byte double precision floating points.
NaN is Y if certain values are to be replaced by NAN, and N otherwise.
NaN-proxy is that special value.
scale is used to multiply the data after reading.
offset is used to add to the scaled data.
oformat is a C-style format string used to print values from this column.

For ASCII data the type format is always a. NaN should be Y if there is a certain value (e.g., -9999) that is used to represent missing data (NaNs). If so, the NaN-proxy value should be set accordingly (otherwise it is ignored). The scale and offset are applied to the data values after they are read (first scaled, then offset), whereas the oformat is used by some programs for formatted output. If you give - as the oformat then GMT’s formatting machinery will be used instead (i.e., PLOT_DEGREE_FORMAT, PLOT_DATE_FORMAT, PLOT_CLOCK_FORMAT). Some file formats already have definition files premade. These include mgd77 (for plain ASCII MGD77 data files), mgd77+ (for enhanced MGD77+ netCDF files), gmt (for old mgg supplement binary files), xy (for plain ASCII x, y tables), xyz (same, with z-column), geo (for plain ASCII longitude, latitude files), and geoz (same, with z-column).

If you have a large set of track data files you can organize them using the x2sys tools. Here we will outline the steps. Let us assume that your track data file format consist of 2 header records with text information followed by any number of identically formatted data records with 6 columns (lat, lon, time, obs1, obs2, obs3) and that files are called *.trk. We will call this the "line" format. First, we create the line.def file:

# Define file for the line format

#ASCII

# File is ASCII

#SKIP 2

# Skip 2 header records

#name

type

NaN

NaN-proxy

scale offset oformat

lon

a

N

0

1

0

%10.5f

lat

a

N

0

1

0

%9.5f

time

a

N

0

1

0

%7.1f

obs1

a

N

0

1

0

%7.2f

obs2

a

N

0

1

0

%7.2f

obs3

a

N

0

1

0

%7.2f

Next we create the TAG and the databases for these line track files. Assuming these contain geographic data and that we want to keep track of the data distribution at a 1 x 1 degree resolution, we may run

x2sys_init LINE −V −G −Dline −R0/360/-90/90 −I1/1 −Etrk

where we have used LINE as our system tag. When x2sys tools try to read your line data files they will first look in the current directory and second look in the file TAG_paths.txt for a list of additional directories to examine. Therefore, create such a file (here LINE_paths.txt) and stick the full paths to your data directories there. All TAG-related files (definition files, tag files, and track data bases built) will be expected to be in the directory pointed to by $X2SYS_HOME.

Create tbf file(s):

Once the (empty) TAG database have been initialized we go through a two-step process to populate them. First we run x2sys_binlist on all our track files to create one (or more) multi-segment track bin-index files (tbf). These contain information on which 1 x 1 degree bins (or any other blocksize; see −I) each track has visited and which observations (in your case obs1, obs2, obs3) were actually observed (not all tracks may have all three kinds of observations everywhere). For instance, if your tracks are listed in the file tracks.lis we may run this command:

x2sys_binlist −V −TLINE ‘cat tracks.lis‘ > tracks.tbf

Update index data base:

Next, the track bin-index files are fed to x2sys_put which will insert the information into the TAG databases:

x2sys_put −V −TLINE tracks.tbf

Search for data:

You may now use x2sys_get to find all the tracks within a certain sub-region, and optionally limit the search to those tracks that have a particular combination of observables. E.g., to find all the tracks which has both obs1 and obs3 inside the specified region, run

x2sys_get −V −TLINE −R20/40/-40/-20 −Fobs1,obs3 > tracks.tbf

MGD77[+] or GMT:

Definition files already exist for MGD77 files (both standard ASCII and enhanced netCDF-based MGD77+ files), the old *.gmt files manipulated by the mgg supplements, and plain x,y[,z] and lon,lat[,z] tracks. To initiate new track databases to be used with MGD77 data from NGDC, try

x2sys_init MGD77 −V −Dmgd77 −Emgd77 −Rd −Gd −I1/1 −Wt900 −Wd5

where we have chosen a 15 minute (900 sec) or 5 km threshold to indicate a data gap; the other steps are similar.

Binary files:

Let us pretend that your line files actually are binary files with a 128-byte header structure (to be skipped) followed by the data records and where lon, lat, time are double precision numbers while the three observations are 2-byte integers which must be multiplied by 0.1. Finally, the first two observations may be -32768 which means there is no data available. All that is needed is a different line.def file:

# Define file for the binary line format

#BINARY

# File is now binary

#SKIP 128

# Skip 128 bytes

#name

type

NaN?

NaN-proxy

scale offset oformat

lon

d

N

0

1

0

%10.5f

lat

d

N

0

1

0

%9.5f

time

d

N

0

1

0

%7.1f

obs1

h

Y

-32768

0.1

0

%6.1f

obs2

h

Y

-32768

0.1

0

%6.1f

obs3

h

N

0

0.1

0

%6.1f

The rest of the steps are identical.

x2sys_binlist(l) x2sys_get(l) x2sys_put(l)