Analyzing SECONDO Crashes
This page describes how a crash of Secondo can be analyzed.
For the demonstration of a crash, the faultcrash operator is used, which is
part of the AuxiliaryAlgebra. This operator forwards a stream of tuples and
crashes the system with a certain probability. This operator is primarily used in
Secondo to demonstrate the error recovery capability of the
distributed algebras. For example, the query
query plz feed faultcrash[100] count crashes the system with a probability of
(1/100) = 1% per processed tuple.
Table of Contents
- Automatically Generated Stacktrace
- Reading a Stacktrace from a Logfile
- Using Valgrind
- Using the GNU Debugger (GDB) and Core Dumps on Linux
- Using the Low-Level Debugger (lldb) on macOS
Automatically Generated Stacktrace
After Secondo has crashed, a signal handler is called, which
dumps the current stack. To be more precise: in
Tools/Utilities/Application.cpp, a signal handler is defined and called on
signals
like SIGINT, SIGTERM, or SIGFPE.
The stacktrace is symbolized directly in the crashing process using libbacktrace.
When a signal arrives, WinUnix::stacktrace() walks the stack and, for every frame,
resolves the function name and the source file and line directly against the debug info of the
shared object that frame belongs to — the main Secondo
binary, an algebra .so, or libc. This means every frame is fully resolved by the
time it is printed.
Resolution can still fail in two situations: (1) when the stack becomes corrupted before the
crash occurs, it can not be unwound, and (2) a frame without debug info available (e.g., code
built without -g, or a build configured with
SECONDO_NO_LIBBACKTRACE=1) is printed as a bare address with no function name or
source location.
$ ./SecondoTTYBDB [...] Secondo => query plz feed faultcrash[100] count; command 'query plz feed faultcrash[100] count' started at: Thu Oct 22 08:31:50 2020 noMemoryOperators = 0 perOperator = 0 Generating stack trace ... ************ BEGIN STACKTRACE ************ 0x55d4f31ca5b1 _ZN7WinUnix10stacktraceEPKc at WinUnix.cpp:247 0x55d4f31c1860 _ZN11Application20AbortOnSignalHandlerEi at Application.cpp:330 0x7fc65d4bcfd0 ?? 0x55d4f425dc6b _ZN9auxiliary14FaultLocalInfoILNS_9CrashTypeE0EE12forwardTupleEv at AuxiliaryAlgebra.cpp:376 0x55d4f425d944 _ZN9auxiliary11InjectFaultILNS_9CrashTypeE0EEEiP4WordRS2_iS4_Pv at AuxiliaryAlgebra.cpp:421 0x55d4f31f71f3 _ZN14QueryProcessor4EvalEPvR4Wordi at QueryProcessor.cpp:4491 0x55d4f31f794c _ZN14QueryProcessor7RequestEPvR4Word at QueryProcessor.cpp:4691 0x55d4f335e1a0 _Z12TCountStreamP4WordRS_iS1_Pv at RelationAlgebra.cpp:2096 [...] 0x55d4f3175314 main at MainTTY.cpp:164 *********** END STACKTRACE ********************** *** Signal SIGILL (4) caught! Calling default signal handler ...
Note: A stacktrace has to be read from the bottom up. At the bottom you will find the entry point into the application, whereas at the top are the most recently called functions and methods.
In the example, the first three entries are the most important ones.
_ZN7WinUnix10stacktraceEPKc at WinUnix.cpp:247 shows that the
stacktrace method of the class WinUnix was run as the last method.
Before this method was called, the AbortOnSignalHandler method of the class
Application was called
(_ZN11Application20AbortOnSignalHandlerEi at Application.cpp:330). The third
entry on the stack is the most important one:
_ZN9auxiliary14FaultLocalInfoILNS_9CrashTypeE0EE12forwardTupleEv at
AuxiliaryAlgebra.cpp:376. It can be seen that the forwardTuple method of a
class called auxiliary was called before the crash handler was executed, at line
376 of AuxiliaryAlgebra.cpp. This is exactly the method of the
faultcrash operator in the AuxiliaryAlgebra.
It can also be seen that Secondo has crashed with an Illegal Instruction Exception (SIGILL).
Demangled Stacktrace
The function names above are DWARF linkage (mangled) names —
libbacktrace does not demangle them. When
Secondo is started through bin/SecondoTTYBDB, the
stacktrace is captured to a temporary file, and once Secondo
exits, the script prints that file piped through c++filt (when available) to
demangle the names:
======== SECONDO has crashed, printing stack trace.... ======== 0x55d4f31ca5b1 WinUnix::stacktrace(char const*) at WinUnix.cpp:247 0x55d4f31c1860 Application::AbortOnSignalHandler(int) at Application.cpp:330 0x7fc65d4bcfd0 ?? 0x55d4f425dc6b auxiliary::FaultLocalInfo<(auxiliary::CrashType)0>::forwardTuple() at AuxiliaryAlgebra.cpp:376 0x55d4f425d944 int auxiliary::InjectFault<(auxiliary::CrashType)0>(Word*, Word&, int, Word&, void*) at AuxiliaryAlgebra.cpp:421 0x55d4f31f71f3 QueryProcessor::Eval(void*, Word&, int) at QueryProcessor.cpp:4491 [...] 0x55d4f3175314 main at MainTTY.cpp:164 ========
Reading a Stacktrace from a Logfile
In certain situations (e.g., when working with the Distributed Algebra and
Secondo workers), only the stacktrace that ended up in a logfile
is available, without the interactive bin/SecondoTTYBDB wrapper around it. This
is not a problem: the stacktrace was already fully resolved to function names and source
locations by libbacktrace on the machine where
Secondo crashed, so the logfile can simply be read as is.
Note: Resolving the trace only requires the debug info to be available on the
crashing machine at the time of the crash (i.e., the binary that crashed must have been
built with libbacktrace support and with debug symbols). The machine you are
reading the logfile on does not need a matching copy of the binary.
The only thing that may still be missing is demangling: a stacktrace read straight out of a
logfile still contains DWARF linkage (mangled) names, since only
bin/SecondoTTYBDB pipes its captured trace through c++filt
automatically. Piping the logfile content through c++filt demangles it:
$ c++filt < stacktrace.txt 0x55a0ec18bdc6 WinUnix::stacktrace(char const*) at WinUnix.cpp:247 0x55a0ec18316d Application::AbortOnSignalHandler(int) at Application.cpp:330 0x7f0df2aacfd0 ?? 0x55a0ed25c20b auxiliary::FaultLocalInfo<(auxiliary::CrashType)0>::forwardTuple() at AuxiliaryAlgebra.cpp:376 [...] 0x55a0ec136c66 main at MainTTY.cpp:164
Using Valgrind
Valgrind is a tool to profile and debug C
and C++ software. It allows us to detect memory leaks and also shows debug data when software
crashes. Secondo automatically calls Valgrind when the option
--valgrind is provided. A drawback is that Valgrind performs many checks, which
causes some overhead during the execution of Secondo and
significantly slows down the execution.
$ ./SecondoTTYBDB --valgrind [...] Secondo => query plz feed faultcrash[100] count; [...] *** Signal SIGILL (4) caught! Calling default signal handler ... ==14132== ==14132== Process terminating with default action of signal 4 (SIGILL) ==14132== at 0x5EA2715: raise (raise.c:46) ==14132== by 0x139C8AF: Application::AbortOnSignalHandler(int) (Application.cpp:311) ==14132== by 0x81A2FCF: ??? (in /lib/x86_64-linux-gnu/libc-2.27.so) ==14132== by 0x2438C5F: auxiliary::FaultLocalInfo<(auxiliary::CrashType)0>::forwardTuple() (AuxiliaryAlgebra.cpp:373) ==14132== by 0x2438943: int auxiliary::InjectFault<(auxiliary::CrashType)0>(Word*, Word&, int, Word&, void*) (AuxiliaryAlgebra.cpp:420) ==14132== by 0x13D21F2: QueryProcessor::Eval(void*, Word&, int) (QueryProcessor.cpp:4491) [...] ==14132== by 0x1350313: main (MainTTY.cpp:164) ==14132== ==14132== HEAP SUMMARY: ==14132== in use at exit: 38,668,949 bytes in 57,824 blocks ==14132== total heap usage: 241,221 allocs, 183,397 frees, 170,810,525 bytes allocated ==14132==
Valgrind produces a cleaner stacktrace with line numbers. The stack can be read from the top
to the bottom to see which methods have been called before the crash occurred. The most
important lines show that the AbortOnSignalHandler of the class
Application was called before the application crashes. Before that, the
forwardTuple method of the class auxiliary was executed. This method
can be found in the file AuxiliaryAlgebra.cpp at line 373.
Using the GNU Debugger (GDB) and Core Dumps on Linux
To overcome the slowdown that Valgrind introduces, the
GNU debugger can be used
together with core dumps.
Core dumps are generated on Unix systems after a software crashes. The core dumps contain the
memory image of the crashed software; the call stack is a part of this memory image. To save
disk space, most Unix systems don't generate core dumps by default. To store these files on
disk, the command ulimit -c unlimited has to be executed. This sets the maximal
size of core dumps (-c) to an unlimited amount of bytes. After the
command is called, the core dumps of all programs that are started from this shell are written
to disk.
$ ulimit -c unlimited $ ./SecondoTTYBDB [...] Secondo => query plz feed faultcrash[100] count; *** Signal SIGILL (4) caught! Calling default signal handler ... ./SecondoTTYBDB: line 34: 14367 Floating point exception(core dumped) $runner $*
After Secondo has crashed, a file named core was
written:
$ ls -l core -rw------- 1 nidzwetzki nidzwetzki 51212288 Okt 22 08:41 core
This file can now be loaded into the GNU Debugger (gdb). In addition, the path to the binary
file that has generated this core dump has to be provided. In most cases, this is
SecondoBDB. However, when a problem in the server component of
Secondo has caused the crash, the appropriate binary (e.g.,
SecondoListener) has to be provided as a parameter.
$ gdb SecondoBDB core [...] Core was generated by `./SecondoBDB'. Program terminated with signal SIGILL, Arithmetic exception. #0 0x000055e1bfb17c6b in auxiliary::FaultLocalInfo<(auxiliary::CrashType)0>::forwardTuple (this=0x55e1c210c280) at AuxiliaryAlgebra.cpp:376 376 __builtin_trap(); [Current thread is 1 (Thread 0x7f6b742f37c0 (LWP 14367))]
After the core dump has been processed by gdb, two important pieces of
information can be seen: (1) the method forwardTuple is shown together with the
file and line number AuxiliaryAlgebra.cpp:376 that has caused the crash, and (2)
the line of code that has caused the crash (__builtin_trap();) is also shown.
This line executes an illegal instruction, which crashes
Secondo.
GDB provides a lot of
useful commands.
For example, the command bt (short for back trace) dumps the complete stacktrace.
(gdb) bt #0 0x000055e1bfb17c6b in auxiliary::FaultLocalInfo<(auxiliary::CrashType)0>::forwardTuple (this=0x55e1c210c280) at AuxiliaryAlgebra.cpp:376 #1 0x000055e1bfb17944 in auxiliary::InjectFault<(auxiliary::CrashType)0> (args=0x55e1c2413430, result=..., message=2, local=..., s=0x55e1c24133f0) at AuxiliaryAlgebra.cpp:420 #2 0x000055e1beab11f3 in QueryProcessor::Eval (this=0x55e1c1ec96c0, node=0x55e1c24133f0, result=..., message=2) at QueryProcessor.cpp:4491 [...] #15 0x000055e1bea2f314 in main (argc=1, argv=0x7ffe4780b4e8) at MainTTY.cpp:164
Note: Don't forget to delete the core file after the examination of the crash is done.
This saves space and allows Secondo to generate a new core dump
on the next crash. If the core file is not generated on a Linux system, have a look at the
file /proc/sys/kernel/core_pattern (e.g.,
cat /proc/sys/kernel/core_pattern). Some Linux distributions write the core dumps
to other directories, append the process id (i.e., core.12345), or forward the
core dumps to a crash handler software.
Using the Low-Level Debugger (lldb) on macOS
macOS uses a slightly different toolchain for debugging software. To debug Secondo on macOS, we recommend starting Secondo directly in the low-level debugger (lldb). A crash is captured and handled by the debugger automatically, and a readable stacktrace can be generated.
To start Secondo in lldb, please execute the command
lldb ./SecondoBDB in the bin directory of
Secondo. After the debugger is running, the path to prolog needs
to be provided. This can be done by executing
env LD_LIBRARY_PATH="/Applications/SWI-Prolog.app/Contents/Frameworks". As the
last step, run needs to be executed to start
Secondo.
In the following example, Secondo is started in the debugger, and
a query containing the faultcrash operator is executed:
$ lldb ./SecondoBDB
(lldb) target create "./SecondoBDB"
Current executable set to '/Users/kristofnidzwetzki/secondo/bin/SecondoBDB' (x86_64).
(lldb) env LD_LIBRARY_PATH="/Applications/SWI-Prolog.app/Contents/Frameworks"
(lldb) run
[...]
Secondo => query plz feed faultcrash[100] count;
[...]
Process 33348 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
frame #0: 0x00000001018bbef2 SecondoBDB`auxiliary::FaultLocalInfo<(auxiliary::CrashType)0>::forwardTuple(this=0x0000000107235bf0) at AuxiliaryAlgebra.cpp:372:14
369 int randValue = rand();
370 if(randValue % crashAfter == 0) {
371 if(crashType == CRASH) {
-> 372 __builtin_trap();
373 exit(1);
374 } else {
375 // Loop forever!
Target 0: (SecondoBDB) stopped.
It can be seen that the resulting crash is captured and handled by the debugger. The operation
that caused the crash is marked with a ->. In this example, this is line 372 in
the file AuxiliaryAlgebra.cpp.
By typing in bt (short for back trace), a full stacktrace can be generated. lldb
provides almost the same commands as gdb. More information about the differences can be found
here.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
* frame #0: 0x00000001018bbef2 SecondoBDB`auxiliary::FaultLocalInfo<(auxiliary::CrashType)0>::forwardTuple(...) at AuxiliaryAlgebra.cpp:372:14
frame #1: 0x00000001018b8dc7 SecondoBDB`int auxiliary::InjectFault<(auxiliary::CrashType)0>(...) at AuxiliaryAlgebra.cpp:415:13
frame #2: 0x000000010016d95f SecondoBDB`QueryProcessor::Eval(...) at QueryProcessor.cpp:4492:29
[...]
frame #15: 0x00000001000851cd SecondoBDB`main(argc=1, argv=0x00007ffeefbff050) at MainTTY.cpp:164:10
frame #16: 0x00007fff2035d631 libdyld.dylib`start + 1
(lldb) quit