[Linux] strace: live system call tracing in Linux

original article: http://linuxhelp.blogspot.com/2006/05/strace-very-powerful-troubleshooting.html

Many times I have come across seemingly hopeless situations where a program when compiled and installed in GNU/Linux just fails to run. In such situations after I have tried every trick in the book like searching on the net and posting questions to Linux forums, and still failed to resolve the problem, I turn to the last resort which is trace the output of the misbehaving program. Tracing the output of a program throws up a lot of data which is not usually available when the program is run normally. And in many instances, sifting through this volume of data has proved fruitful in pin pointing the cause of error.

For tracing the system calls of a program, we have a very good tool in strace. What is unique about strace is that, when it is run in conjunction with a program, it outputs all the calls made to the kernel by the program. In many cases, a program may fail because it is unable to open a file or because of insufficient memory. And tracing the output of the program will clearly show the cause of either problem.

The use of strace is quite simple and takes the following form:
$ strace
For example, I can run a trace on 'ls' as follows :
$ strace ls
And this will output a great amount of data on to the screen. If it is hard to keep track of the scrolling mass of data, then there is an option to write the output of strace to a file instead which is done using the -o option. For example,
$ strace -o strace_ls_output.txt ls
... will write all the tracing output of 'ls' to the 'strace_ls_output.txt' file. Now all it requires is to open the file in a text editor and analyze the output to get the necessary clues.

It is common to find a lot of system function calls in the strace output. The most common of them being open(),write(),read(),close() and so on. But the function calls are not limited to these four as you will find many others too.

For example, if you look in the strace output of ls, you will find the following line:
open("/lib/libselinux.so.1", O_RDONLY) = 3
This means that some aspect of ls requires the library module libselinux.so.1 to be present in the /lib folder. And if the library is missing or in a different path, then that aspect of ls which depends on this library will fail to function. The line of code signifies that the opening of the library libselinux.so.1 is successful.