Meta data

Overview

The scr_meta data structure associates various properties with files written by the application and with redundancy data files written by SCR. It tracks information such as the type of file (application vs. SCR redundancy data), whether the application marked the file as valid or invalid, the expected file size, its CRC32 checksum value if computed, and the original string the application used to register the file. Because the meta data is stored within a filemap (Section Filemap), there is no need to store the dataset id or rank id which owns the file.

The scr_meta data structure makes heavy use of the scr_hash data structure (Section Hash). The scr_hash is utilized in the scr_meta API and its implementation. Essentially, scr_meta objects are specialized scr_hash objects, which have certain well-defined keys (fields) and associated functions to access those fields.

Example meta data hash

Internally, meta data objects are implemented as scr_hash objects. Here is an example hash for a meta data object containing information for a file named “rank_0.ckpt”.

FILE
  rank_0.ckpt
TYPE
  FULL
COMPLETE
  1
SIZE
  524294
CRC
  0x1b39e4e4
CKPT
  6
RANKS
  4
ORIG
  ckpt.6/rank_0.ckpt
ORIGPATH
  /p/lscratchb/user3/simulation123/ckpt.6
ORIGNAME
  rank_0.ckpt

The FILE field records the file name this meta data associates with. In this example, the file name is recorded using a relative path. The TYPE field indicates whether the file is written by the application (FULL), whether it’s a PARTNER copy of a file (PARTNER), or whether it’s a redundancy file for an XOR set (XOR). The COMPLETE field records whether the file is valid. It is set to 1 if the file is thought to be valid, and 0 otherwise. The SIZE field records the size of the file in bytes. The CRC field records the CRC32 checksum value over the contents of the file. The CKPT field records the checkpoint id in which the file was written. The RANKS field record the number of ranks active in the run when the file was created. The ORIG field records the original string specified by the caller when the file was registered in the call to SCR_Route_File(). The ORIGPATH field records the absolute path to the original file at the time the file was registered, and the ORIGNAME field records just the name of the file when registered.

In this case, “rank_0.ckpt” was created during checkpoint id 6, and it was written in a run with 4 MPI tasks. It was written by the application, and it is marked as being complete. It consists of 524,294 bytes and its CRC32 value is 0x1b39e4e4. The caller referred to this file as “ckpt.6/rank_0.ckpt” when registering this file in SCR_Route_file(). Based on the current working directory at the time when SCR_Route_file was called, the absolute path to the file would have been “/p/lscratchb/user3/simulation123/ckpt.6” and its name would have been “rank_0.ckpt”.

These are the most common fields used in meta data objects. Not all fields are required, and additional fields may be used that are not shown here.

Common functions

This section describes some of the most common meta data functions. For a detailed list of all functions, see scr_meta.h. The implementation can be found in scr_meta.c.

Allocating, freeing, and copying meta data objects

Create a new meta data object.

scr_meta* meta = scr_meta_new()

Free a meta data object.

scr_meta_delete(&meta)

Make an exact copy of meta_2 in meta_1.

scr_meta_copy(meta_1, meta_2)

Setting, getting, and checking field values

There are functions to set each field individually.

scr_meta_set_complete(meta, complete)
scr_meta_set_ranks(meta, ranks)
scr_meta_set_checkpoint(meta, ckpt)
scr_meta_set_filesize(meta, filesize)
scr_meta_set_filetype(meta, filetype)
scr_meta_set_filename(meta, filename)
scr_meta_set_crc32(meta, crc32)
scr_meta_set_orig(meta, string)
scr_meta_set_origpath(meta, path)
scr_meta_set_origname(meta, name)

If a field was already set to a value before making this call, the new value overwrites any existing value.

And of course there are corresponding functions to get values.

scr_meta_get_complete(meta, complete)
scr_meta_get_ranks(meta, ranks)
scr_meta_get_checkpoint(meta, ckpt)
scr_meta_get_filesize(meta, filesize)
scr_meta_get_filetype(meta, filetype)
scr_meta_get_filename(meta, filename)
scr_meta_get_crc32(meta, crc32)
scr_meta_get_orig(meta, string)
scr_meta_get_origpath(meta, path)
scr_meta_get_origname(meta, name)

If the corresponding field is set, the get functions copy the value into the output parameter and return SCR_SUCCESS. If SCR_SUCCESS is not returned, the output parameter is not changed.

Many times one simply wants to verify that a field is set to a particular value. The following functions return SCR_SUCCESS if a field is set and if that field matches the specified value.

scr_meta_check_ranks(meta, ranks)
scr_meta_check_checkpoint(meta, ckpt)
scr_meta_check_filesize(meta, filesize)
scr_meta_check_filetype(meta, filetype)
scr_meta_check_filename(meta, filename)

Similar to the above functions, the following function returns SCR_SUCCESS if the complete field is set and if its value is set to 1.

scr_meta_check_complete(meta)