If we run this example, it will create a file named 00_Main.trc and several files named *_Compute.trc (each thread creates ts own file). If we then run TimeTrac in this directory, we will see a display similar to the that below. In ths case we have multiple threads each with multiple events instrumented. Each of the "compute" threads run the same code and should produce similar results for 25 iterations.

Note: At this point when running TimeTrac, you should set some viewer defaults (failure do do so may cause images on your system to appear different compared to that described here and in the following examples). First adjust the screen size for convenient viewing and then:

• In the task control bar, select Sort and then Sort by Increasing Name
• In the task control bar, select File and then Start Up Properties.
• Then select Save in the lower left corner.

At first glance, it appears that the processing is not uniform. We will use the time bars to expand a region for closer examination.

Looking at the region around 0.2 seconds, we have several threads that have been interrupted. We can visually see that some certain instances of some functions take much longer than usual. In the first case (below), one of the threads has been interrupted by the Linux scheduler in the midst of one of its operations (thread for rank #3, the "vinput" function is taking much longer than expected), and its processor was given to another thread, most likely that of rank #1 or #2. In this example, no harm has been done; these threads are free running with no dependencies between threads. In a time critical application however, this might not be the case and the unexpected switching of threads might be important.

Nor was this an isolated case; threads for ranks #1 and #2 later have a similar issue. This system demonstrates a large amount of thread "unfairness" that is clearly demonstarted by TimeTrac.

Here we are looking at the startup sequence of the 4 threads. Even though they are directed to start at very nearly the same time and in order starting with number 1, thread with rank #4 started first and got an unfair portion of the processing power.

In this example, startup sequence does not matter but it is not what we expected. In another application, this could cause a failure before we even get started. Debugging this without TimeTrac is very painful; the use of printf's in the code could change the application's timing and printf's do not always occur in the order that they appear on the screen. Using TimeTrac is a much better method of finding these issues.

This example shows TimeTrac outputs for an application using multiple threads with multiple operations.

• A parent thread starting / controlling multiple child threads
• 25 executions of the same code sequence in each of 4 additional threads.
• Multiple events within each sequence
• How to spot unexpected (non-uniform) events.
• How to evaluate startup and unfairness issues.