FAQ

This page collects frequently asked questions, known problems and their solutions when building, running, and operating Secondo.

A operation in SECONDO is slow. How can I investigate this?

You can use a profiler like callgrind to investigate the problem.

In one shell, execute:

watch "callgrind_control -e -b"

In another shell, execute:

./SecondoTTYBDB --profile

You will now see in the first shell the current stack trace of Secondo while it is running. Execute the slow operation and have a look at the stack trace, and you see in which operations the CPU time is spent.

After you have finished your operation and you have quit Secondo, a file named callgrind.out.<pid> is written into the current directory. This file contains the stack trace and can be analyzed by executing callgrind_annotate <filename>.

ld: [...] building for macOS-arm64 but attempting to link with file built for macOS-x86_64

It seems you are using a Mac with an ARM-based CPU, while your Secondo-SDK was built on an Intel CPU. Please delete your old Secondo-SDK (e.g., rm ~/secondo-sdk) and reinstall the latest Secondo-SDK. Please also re-install your BerkeleyDB installation.

E_SMI_BDB sysErrCode=12 Cannot allocate memory

---------------------------
  Secondo-SMI Error Stack
---------------------------
E_SMI_BDB sysErrCode=12 Cannot allocate memory -> [bdbFile.cpp:1622]

This error is most likely caused by a too small memory area for lock objects (more information can be found here and here). To solve this problem, two known solutions exist:

• Disable the transactions in Secondo. This can be done by setting RTFlags += SMI:NoTransactions in your SecondoConfig.ini.
• Increase the memory area for locks. This is controlled by the following settings in the SecondoConfig.ini:

  [BerkeleyDB]
  
  MaxLockers=100
  MaxLocks=5000
  MaxLockObjects=5000

  In most cases the values for MaxLocks and MaxLockObjects need to be increased.

Compile error 'syntax error, unexpected string, expecting =' in *.y files

This error indicates that you are using an outdated version of Bison. Please ensure that you have updated your project dependencies. At least Bison 3.0.4 is required to build the project.

Compile error ''register' storage class specifier is deprecated and incompatible with C++17'

This error indicates that you are using an outdated version of Flex. Please ensure that you have updated your project dependencies. At least Flex 2.6.4 is required to build the project.

Compile error 'undefined reference to `TempRelation::BasicType[abi:cxx11]()'

A part of the project was built with an older C++ standard, another part with the C++17 standard (-std=c++17). The linker is unable to link these files. Please update your Secondo to the most recent version and run:

make clean
make

Compile error 'fatal error: libxml/xmlreader.h: No such file or directory'

The GNOME XML library was not found on the system. Please install the library and the needed development files on the system. On Debian/Ubuntu based systems, this can be done by executing apt-get install libxml2-dev.

How to run checkpd in git as a pre-commit check

Commits on GitHub are tested asynchronously by checkpd and a GitHub Action. Due to the asynchronous execution, it may take some time before an error is reported. To execute checkpd locally before a commit is made, use the included pre-commit check. This check can be enabled by executing the command cp -av CM-Scripts/pre-commit .git/hooks/pre-commit from the root directory of your git repository.

How to speed up the compilation process?

You can use ccache to cache compilation units (even make clean / make -j $(nproc) is quite fast afterwards). Simply add the following lines to your .secondorc file:

export SECONDO_CC="ccache gcc"
export SECONDO_CPP="ccache g++"

The cache can be checked by running ccache -s.

How can I keep long-running queries alive on a remote system?

Executing queries on a large data set in Secondo can take some time. Usually, the secure shell (SSH) is used to connect to a remote system. Keeping an SSH connection open for a long time can be challenging. The following tips can be used to keep a connection open or prevent the termination of Secondo when the SSH connection is terminated.

Enabling the 'ServerAliveInterval' in the SSH Client

IP gateways (such as firewalls) can terminate idle TCP connections after some time. To prevent a TCP connection from becoming idle, SSH can send some keep-alive packets in the background on the TCP connection automatically. To enable the feature, add the following lines to your SSH client configuration (e.g., /etc/ssh/ssh_config or ~/.ssh/ssh_config):

Host *
    ServerAliveInterval 60

The setting will send a keep-alive packet every 60 seconds. If the connection is terminated earlier, please reduce the setting to a lower value (e.g., 15). Please note this is a setting for the SSH client ssh_config, not for the SSH server sshd_config.

Using screen to Execute SECONDO

screen is a terminal multiplexer for Linux. It allows the execution of multiple terminal sessions within one window or SSH connection. Besides, the screen session (and all programs started from the session) is not terminated when the SSH connection is lost. Another SSH session can be used to re-attach to the previously opened session. screen can be used to start Secondo and to re-attach to the same Secondo session when the connection was lost. A complete overview of the functionality of screen can be found here.

Example:

# Connect to the remote system
$ ssh user@remote.system.tld

# Start a new screen session and run SECONDO
$ screen
$ cd ~/secondo/bin
$ ./SecondoTTYBDB

# You can now detach from the SECONDO session by pressing CTRL-a and d
# (press CTRL-a first, then press d)

# Re-attach to the previously opened screen session
# This can also be done from another SSH session if the connection was lost
$ screen -x