4.2.5  Tracing Activity on Target Devices with OMPT

A target device may or may not initialize a full OpenMP runtime system. Unless it does, it may not be possible to monitor activity on a device using a tool interface based on callbacks. To accommodate such cases, the OMPT interface defines a monitoring interface for tracing activity on target devices. Tracing activity on a target device involves the following steps:

• To prepare to trace activity on a target device, a tool must register for an ompt_callback_device_initialize callback. A tool may also register for an ompt_callback_device_load callback to be notified when code is loaded onto a target device or an ompt_callback_device_unload callback to be notified when code is unloaded from a target device. A tool may also optionally register an ompt_callback_device_finalize callback.
• When an OpenMP implementation initializes a target device, the OpenMP implementation dispatches the device initialization callback of the tool on the host device. If the OpenMP implementation or target device does not support tracing, the OpenMP implementation passes NULL to the device initializer of the tool for its lookup argument; otherwise, the OpenMP implementation passes a pointer to a device-specific runtime entry point with type signature ompt_function_lookup_t to the device initializer of the tool.
• If a non-null lookup pointer is provided to the device initializer of the tool, the tool may use it to determine the runtime entry points in the tracing interface that are available for the device and may bind the returned function pointers to tool variables. Table 4.3 indicates the names of runtime entry points that may be available for a device; an implementations may provide additional implementation-defined names and corresponding entry points. The driver for the device provides the runtime entry points that enable a tool to control the trace collection interface of the device. The native trace format that the interface uses may be device specific and the available kinds of trace records are implementation-defined. Some devices may allow a tool to collect traces of records in a standard format known as OMPT trace records. Each OMPT trace record serves as a substitute for an OMPT callback that cannot be made on the device. The fields in each trace record type are defined in the description of the callback that the record represents. If this type of record is provided then the lookup function returns values for the runtime entry points ompt_set_trace_ompt and ompt_get_record_ompt, which support collecting and decoding OMPT traces. If the native tracing format for a device is the OMPT format then tracing can be controlled using the runtime entry points for native or OMPT tracing.

  ---

  Table 4.3:

  OMPT Tracing Interface Runtime Entry Point Names and Their Type Signatures

  Entry Point String Name	Type Signature
  “ompt_get_device_num_procs”	ompt_get_device_num_procs_t
  “ompt_get_device_time”	ompt_get_device_time_t
  “ompt_translate_time”	ompt_translate_time_t
  “ompt_set_trace_ompt”	ompt_set_trace_ompt_t
  “ompt_set_trace_native”	ompt_set_trace_native_t
  “ompt_start_trace”	ompt_start_trace_t
  “ompt_pause_trace”	ompt_pause_trace_t
  “ompt_flush_trace”	ompt_flush_trace_t
  “ompt_stop_trace”	ompt_stop_trace_t
  “ompt_advance_buffer_cursor”	ompt_advance_buffer_cursor_t
  “ompt_get_record_type”	ompt_get_record_type_t
  “ompt_get_record_ompt”	ompt_get_record_ompt_t
  “ompt_get_record_native”	ompt_get_record_native_t
  “ompt_get_record_abstract”	ompt_get_record_abstract_t

  ---

• The tool uses the ompt_set_trace_native and/or the ompt_set_trace_ompt runtime entry point to specify what types of events or activities to monitor on the device. The return codes for ompt_set_trace_ompt and ompt_set_trace_native use the ompt_set_result_t enumeration type. If the ompt_set_trace_native /or the ompt_set_trace_ompt runtime entry point is called outside a device initializer, registration of supported callbacks may fail with a return code of ompt_set_error.
• The tool initiates tracing on the device by invoking ompt_start_trace. Arguments to ompt_start_trace include two tool callbacks through which the OpenMP implementation can manage traces associated with the device. One allocates a buffer in which the device can deposit trace events. The second callback processes a buffer of trace events from the device.
• If the device requires a trace buffer, the OpenMP implementation invokes the tool-supplied callback function on the host device to request a new buffer.
• The OpenMP implementation monitors the execution of OpenMP constructs on the device and records a trace of events or activities into a trace buffer. If possible, device trace records are marked with a host_op_id—an identifier that associates device activities with the target operation that the host initiated to cause these activities. To correlate activities on the host with activities on a device, a tool can register a ompt_callback_target_submit callback. Before the host initiates each distinct activity associated with a structured block for a target construct on a device, the OpenMP implementation dispatches the ompt_callback_target_submit callback on the host in the thread that is executing the task that encounters the target construct. Examples of activities that could cause an ompt_callback_target_submit callback to be dispatched include an explicit data copy between a host and target device or execution of a computation. This callback provides the tool with a pair of identifiers: one that identifies the target region and a second that uniquely identifies an activity associated with that region. These identifiers help the tool correlate activities on the target device with their target region.
• When appropriate, for example, when a trace buffer fills or needs to be flushed, the OpenMP implementation invokes the tool-supplied buffer completion callback to process a non-empty sequence of records in a trace buffer that is associated with the device.
• The tool-supplied buffer completion callback may return immediately, ignoring records in the trace buffer, or it may iterate through them using the ompt_advance_buffer_cursor entry point to inspect each record. A tool may use the ompt_get_record_type runtime entry point to inspect the type of the record at the current cursor position. Three runtime entry points (ompt_get_record_ompt, ompt_get_record_native, and ompt_get_record_abstract) allow tools to inspect the contents of some or all records in a trace buffer. The ompt_get_record_native runtime entry point uses the native trace format of the device. The ompt_get_record_abstract runtime entry point decodes the contents of a native trace record and summarizes them as an ompt_record_abstract_t record. The ompt_get_record_ompt runtime entry point can only be used to retrieve records in OMPT format.
• Once tracing has been started on a device, a tool may pause or resume tracing on the device at any time by invoking ompt_pause_trace with an appropriate flag value as an argument.
• A tool may invoke the ompt_flush_trace runtime entry point for a device at any time between device initialization and finalization to cause the device to flush pending trace records.
• At any time, a tool may use the ompt_start_trace runtime entry point to start tracing or the ompt_stop_trace runtime entry point to stop tracing on a device. When tracing is stopped on a device, the OpenMP implementation eventually gathers all trace records already collected on the device and presents them to the tool using the buffer completion callback.
• An OpenMP implementation can be shut down while device tracing is in progress.
• When an OpenMP implementation is shut down, it finalize each device. Device finalization occurs in three steps. First, the OpenMP implementation halts any tracing in progress for the device. Second, the OpenMP implementation flushes all trace records collected for the device and uses the buffer completion callback associated with that device to present them to the tool. Finally, the OpenMP implementation dispatches any ompt_callback_device_finalize callback registered for the device.

RestrictionsTracing activity on devices has the following restriction:

• Implementation-defined names must not start with the prefix ompt_, which is reserved for the OpenMP specification.

Cross References

• ompt_callback_device_initialize_t callback type, see Section 4.5.2.19 on page 1369.
• ompt_callback_device_finalize_t callback type, see Section 4.5.2.20 on page 1372.
• ompt_get_device_num_procs runtime entry point, see Section 4.6.2.1 on page 1449.
• ompt_get_device_time runtime entry point, see Section 4.6.2.2 on page 1451.
• ompt_translate_time runtime entry point, see Section 4.6.2.3 on page 1453.
• ompt_set_trace_ompt runtime entry point, see Section 4.6.2.4 on page 1455.
• ompt_set_trace_native runtime entry point, see Section 4.6.2.5 on page 1458.
• ompt_start_trace runtime entry point, see Section 4.6.2.6 on page 1461.
• ompt_pause_trace runtime entry point, see Section 4.6.2.7 on page 1464.
• ompt_flush_trace runtime entry point, see Section 4.6.2.8 on page 1465.
• ompt_stop_trace runtime entry point, see Section 4.6.2.9 on page 1467.
• ompt_advance_buffer_cursor runtime entry point, see Section 4.6.2.10 on page 1469.
• ompt_get_record_type runtime entry point, see Section 4.6.2.11 on page 1471.
• ompt_get_record_ompt runtime entry point, see Section 4.6.2.12 on page 1473.
• ompt_get_record_native runtime entry point, see Section 4.6.2.13 on page 1475.
• ompt_get_record_abstract runtime entry point, see Section 4.6.2.14 on page 1478.