Debugger
About
This page is about using a debugger with FreeSWITCH. This is an advanced topic. For an introduction to troubleshooting please see Troubleshooting FreeSWITCH
Troubleshooting
Handy Troubleshooting Links
UUID Stamp
The update in the code to enable a UUID Stamp at each debug line was kindly provided by Mathieu Rene and is referred to in the "Finding session pointers" section below. You can enable this by a simple option at the logfile.conf.xml:
<param name="uuid" value="true" />
XML errors
In a message like "error near line X" the line number refers to the concatenation of all the individual configuration files named freeswitch.xml.fsxml and is kept in memory by FreeSWITCH™. The default location is ~freeswitch/log/freeswitch.xml.fsxml
Do not edit this file. Edit the individual configuration file, then issue the reloadxml command in fs_cli or the FS console.
Example:
freeswitch@localhost.localdomain>reloadxml API CALL [reloadxml()] output: +OK [[error near line 1622]: unexpected closing tag </extension>] freeswitch@localhost.localdomain>
In the example above, line 1622 refers to the offending line number in freeswitch.xml.fsxml so open that file to determine where the error lies, but correct the problem in the source file, whether it be dialplan/default.xml or directory/default/1000.xml or wherever. Do not edit the freeswitch.xml.fsxml file directly.
Doing it Better
Some of these can be hard to track down especially if one has update a number of XML files. In such situation an external tool such as 'xmllint' (part of libxml2-utils on Debian/Ubuntu) should be used. It will narrow down the error to exact line with a more information error message.
Example of cryptic FreeSWITCH output:
root@shared-hosting-1:/usr/local/freeswitch/log# ../bin/freeswitch
Error: stacksize 8192 is not optimal: run ulimit -s 240 from your shell before starting the application.
auto-adjusting stack size for optimal performance...
2013-01-12 23:41:29.410698 [INFO] switch_event.c:589 Activate Eventing Engine.
2013-01-12 23:41:29.421120 [WARNING] switch_event.c:563 Create additional event dispatch thread 0
Cannot Initialize [[error near line 4455]: unexpected <]
Example of xmllint output find the error on line 4465; please note that you will also have to filter through errors generated by some of the configuration included in FreeSwith XML files assuming you haven't deleted much of those configs.
root@shared-hosting-1:/usr/local/freeswitch/log# xmllint freeswitch.xml.fsxml
freeswitch.xml.fsxml:224: parser error : Unescaped '<' not allowed in attributes values
<api name="user_name" description="Return Name for extension" syntax="<exten>"
^
freeswitch.xml.fsxml:224: parser error : attributes construct error
<api name="user_name" description="Return Name for extension" syntax="<exten>"
^
freeswitch.xml.fsxml:224: parser error : Couldn't find end of Start Tag api line 224
<api name="user_name" description="Return Name for extension" syntax="<exten>"
^
freeswitch.xml.fsxml:225: parser error : Opening and ending tag mismatch: exten line 224 and apis
</apis>
^
freeswitch.xml.fsxml:226: parser error : Opening and ending tag mismatch: apis line 223 and configuration
</configuration>
^
freeswitch.xml.fsxml:428: parser error : EntityRef: expecting ';'
am name="url" value="http://query.voipcnam.com/query.php?api_key=MYAPIKEY&number
^
freeswitch.xml.fsxml:4450: parser error : Opening and ending tag mismatch: configuration line 222 and section
</section>
^
freeswitch.xml.fsxml:4465: parser error : StartTag: invalid element name
<! -- this is custom added -->
^
freeswitch.xml.fsxml:11591: parser error : Opening and ending tag mismatch: section line 221 and document
</document>
^
freeswitch.xml.fsxml:11592: parser error : Premature end of data in tag document line 19
^
Recompiling with debug symbols on
Note: FreeSWITCH is compiled with debug symbols on by default !
\export CFLAGS="-g -ggdb" \export MOD_CFLAGS="-g -ggdb" ./configure
Creating core files
For core files to be created when a crash occurs, set the core ulimit to unlimited before starting FreeSWITCH
$ ulimit -c unlimited $ ./freeswitch
The core file should be located in the directory where FreeSWITCH was started (i.e., if you were in the /tmp directory when you typed the command to start FreeSWITCH, then there should be a file called something like /tmp/core.xxx).
NOTE: On macOS, core files are dumped to a hidden directory called /cores by default, not the current directory!
Loading FreeSWITCH in GDB
From /usr/local/freeswitch:
$ gdb bin/freeswitch
Once you have reproduced the error you run the following command:
run gdb with core dump file
$ gdb bin/freeswitch core.xxx
Getting a Backtrace
Once you load the core file into GDB you can collect a backtrace typing
gdb command
set logging on
set pagination off
bt
bt full
info threads
thread apply all bt
thread apply all bt full
Default output to the file gdb.txt
Collect the output when reporting bugs to Jira.
If you have a FS source tree available, please run instead:
./scripts/backtrace-from-core /path/to/corefile
Finding session pointers
Sometimes hunting stuff down can be a royal pain. The following code, submitted by Mathieu Rene, will print out all the session information (pointer information, etc.) for each session in a core file. After loading gdb with a core file (see above) do this in gdb:
set $x=session_manager.session_table->table->first while($x != 0x0) printf "uuid %s is at %p\n", $x->pKey, $x->data set $x = $x->next end
NB: $x->data is the switch_core_session_t*, you can dereference the channel with ->channel and can print some other stuff you might be looking for. This also work to print the content of a hashtable (use hash_table_name->table->first on the first set)
Simple bash script to make debug easy
In the source directory there is a support-d sub-directory. Inside this directory is a handy shell script called fscore_pb. The script will collect the appropriate trace (and some other information), post it to pastebin.freeswitch.org, and then give you the URL to give to the developers.
Usage:
support-d/fscore_pb gcore [user] support-d/fscore_pb [user]
If gcore is given it will take a gcore of a running instance of FreeSWITCH (this will pause the process during the dump). If gcore is not given it will use the most recent core.* file in the directory.
User is optional, and is the name given when submitting the pastebin. If it is omitted it will use your local username.
gdb issues
On Ubuntu (Dapper/Hardy/Jaunty) gdb does not show symbols for some modules (e.g. mod_portaudio), while CentOS 5.3 haven't got such issue.
The reason of the issue is that gdb in Ubuntu (probably Debian is affected too) is buggy. As a quick fix you can install development version of gdb from sources (install flex, bison and texinfo and get sources from git repository) which solves the issue. gdb trunk 20090626 is proven to work.