When I run my C program on Ubuntu 20.04, I get this run-time error:
Segmentation fault (core dumped)
I really need to find and view the core file, but I can't find it anywhere. Where is it, and how do I view the backtrace in it?
Tested in Ubuntu 20.04.
First off, run ulimit -c to see what the max allowed size is for core files on your system. On Ubuntu 20.04 for me, mine returns 0, which means no core file can be created.
ulimit --help shows the meaning of -c:
-c the maximum size of core files created
So, set the allowed core file size to unlimited, as shown below. Note that I think this only applies to the one terminal you run this in, and I do not think it's persistent across reboots, so you have to run this each time you want core files to be created, and in each terminal you want it to work in:
# set max core dump file size to unlimited
ulimit -c unlimited
# verify it is now set to "unlimited"
ulimit -c
That's it! Now, run your program and if it crashes and does a "core dump" it will dump the core as a core file into the same directory you were in when you called the executable. The name of the file is simply "core".
I tested all of my code and examples here, and they work for me. However, if you do not get a core file locally as described above, apparently you are not alone. So, try the following:
/var/lib/apport/coredump, as explained in this other answer by @guyr here./var/crash/cat /var/log/apport.logcat /proc/version), and where your core dump files are.ulimit -c unlimited causes core dump files to appear right in the dir where I am. I have to manually delete old ones for new ones to be created. On my Ubuntu 18.04 machine, cat /proc/version shows Linux version 4.15.0-194-generic (buildd@lcy02-amd64-052) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)).You should have built your C or C++ program with debug symbols on, in order to see useful information in your core file. Without debug symbols, you can only see the addresses of the functions called, not the actual names or line numbers.
In gcc, use -ggdb -O0 to turn on debug symbols optimized for the gdb GNU debugger. You can also use -g -O0, -g3 -O0, etc, but -ggdb -O0 is best. Do we really need optimization level 0 (-O0) for this? Yes, yes we do. See my answer here: Stack Overflow: What's the difference between a compiler's -O0 option and -Og option?
Example build and run commands in C and C++: so, your full build and run commands in C or C++ might look like this:
# C build and run command for "hello_world.c"
gcc -Wall -Wextra -Werror -ggdb -O0 -std=c11 -o hello_world hello_world.c
&& ./hello_world
# C++ build and run command for "hello_world.c"
g++ -Wall -Wextra -Werror -ggdb -O0 -std=c++17 -o hello_world hello_world.c
&& ./hello_world
Open the core file in gdb like this:
gdb path/to/my/executable path/to/core
Assuming you just ran path/to/my/executable, then the core file will be in the same directory you were just in when the core was dumped, so you can just run this:
gdb path/to/my/executable core
In gdb, view the backtrace (function call stack at the time of the crash) with:
bt
# or (exact same command)
where
# OR (for even more details, such as seeing all arguments to the functions--
# thanks to Peter Cordes in the comments below)
bt full
# For gdb help and details, see:
help bt
# or
help where
IMPORTANT: when a core dump occurs, it does NOT automatically overwrite any pre-existing core file in your current directory with a new one, so you must manually remove the old core file with rm core PRIOR TO generating the new core file when your program crashes, in order to always have the latest core file to analyze.
sleep 30 to start a process sleeping for 30 seconds.core file in the directory you are in.gdb -c core to open the core file just created by the forced crash.sleep 30) when the core dump occurred:
Core was generated by `sleep 30'.
Program terminated with signal SIGQUIT, Quit.
#0 0x00007f93ed32d334 in ?? ()
(gdb)
bt or where to see the backtrace. You'll see this:
(gdb) bt
#0 0x00007f93ed32d334 in ?? ()
#1 0x000000000000000a in ?? ()
#2 0x00007f93ed2960a5 in ?? ()
#3 0x0000000000000000 in ?? ()
(gdb)
#10 0x00007fc1152b8ebf in __printf (format=<optimized out>) at printf.c:33
#11 0x0000562bca17b3eb in fast_malloc (num_bytes=1024) at src/fast_malloc.c:225
#12 0x0000562bca17bb66 in malloc (num_bytes=1024) at src/fast_malloc.c:496
As @Peter Cordes states in the comments below, you can also just run the program inside gdb directly, letting it crash there, so you have no need to open up a core file after-the-fact! He stated:
Those GDB commands are not specific to core files, they work any time you're stopped at a breakpoint. If you have a reproducible crash, it's often easier / better to run your program under GDB (like
gdb ./a.out) so GDB will have the process in memory instead of a core file. The main advantage is that you can set a breakpoint or watchpoint somewhere before the thing that crashed, and single-step to see what's happening. Or with GDB's record facilities, you may be able to step backwards and see what led up to the crash, but that can be flaky, slow, and memory-intensive.
As stated above, you should have compiled your program with debugging symbols on and with Optimization Level 0, using -ggdb -O0. See the full example build and run commands in C and C++ above.
Now run the program in gdb:
# Open the executable in gdb
gdb path/to/my/executable
# Run it (if it's still crashing, you'll see it crash)
r
# View the backtrace (call stack)
bt
# Quit when done
q
And if you ever need to manually log the backtrace to a log file to analyze later, you can do so like this (adapted from notes in my eRCaGuy_dotfiles repo here):
set logging file gdb_log.txt
set logging on
set trace-commands on
show logging # prove logging is on
flush
set pretty print on
bt # view the backtrace
set logging off
show logging # prove logging is back off
Done! You've now saved the gdb backtrace in file "gdb_log.txt".
-O0 option and -Og option?LD_PRELOAD with gdb: https://stackoverflow.com/questions/10448254/how-to-use-gdb-with-ld-preload
