[NBLUG/talk] Dramatic slow down moving from gcc 2.96 to 4.1.1

[Roger House]

Recently I worked on moving a large, complex app written in C and C++
from gcc 2.96 running on RHEL 2.1 to gcc 4.1.1 running on RHEL 4.  Code
changes were kept to a minimum, basically cleaning up code which gcc 2.96
allowed but gcc 4.1.1 does not allow.  Rather extensive testing has been
done on the app, and it seems to be functionally equivalent to the original.
However, the gcc 4.1.1 version of the app is running significantly slower
than the old app in certain cases, most of the time being used by the system
rather than by the user.  Here is an example, where the times are in 
seconds:

                   gcc     gcc     percent
                   2.96    4.1.1   increase
        -----------------------------------
        total      8.99    30.46      239%
        user       7.20     9.43       31%
        system     1.79    21.03     1075%

In this case the new version is running over 3 times slower than the old
version, with system time almost 12 times slower.  Both sets of timings
above were obtained on the same machine (a four-way 3GHz Intel Xeon running
RHEL 4).  The only difference is that different compilers were used, and
the 2.96 version uses compatibility libraries so it can run on RHEL 4.

Does anyone have any ideas why the system time has increased so 
dramatically?
What profiling tools might I use to see which system calls are taking so
much time?

Thanks in advance,

Roger House