GAAS Q&A: Tutorials, Compilations etc. | 教程常见编译问题汇总


#1

The Basics of Compiling Programs Under Linux | Linux 程序编译基础

You may have noticed that almost 70 per cent of the GAAS repository on GitHub is C++ code. Even though in the tutorials we are programming our drones mainly with Python, it is C++ that does all the heavy lifting behind the scene.

As you may have already known, C++ is a compiled language, i.e. C++ code needs to be translated into binary format to run. This process is not particularly straightforward and may remain a mystery even for some veteran coders. So here is a quick-and-dirty, tl;dr style of introduction to the basics of compiling C++ programs under Linux, especially those relevant to GAAS.

Intended Audience

You should have some entry-level experience with Linux, and be relatively comfortable with the command line interface. Basic C/C++ programming experience would be greatly helpful too, even if it was not on Linux.

Compiler, Build Tool, and CMake

GCC

If you find the names gcc, cc, or CC confusing, it may help to know a bit of history: the earliest C compilers in the 1970s were plainly named cc and could be found under /usr/bin/cc on Unix systems. Then GNU made their own C compiler in the late 80s, named it GCC (GNU C Compiler, later renamed to GNU Compiler Collection as it supports a lot more than just C), which is normally soft-linked to /usr/bin/cc on GNU/Linux, as well as some other Unix-like systems. Similar story for C++: the early C++ compiler in the mid 80s is simply c++, GNU named their version as g++ and soft-linked to /usr/bin/c++.

GCC might be the most popular compiler on Linux, but it is not the only choice. There is Clang/LLVM or Intel C++ Compiler. It is possible to have more than just one C++ compiler installed on your system (and it is also possible that a system does not have a compiler installed at all), so by convention, environment variables CC and CXX, if defined, point to the compiler that should be used for the compilation. It is possible that when you compile a program by calling cc mycode.c under command line, the /usr/bin/gcc is used, but since your CC environment variable is set to /usr/loca/bin/clang, when you call cmake or make, Clang will be used instead of GCC. Similar problems may happen if you have multiple versions of the same compiler too.

The simplest way to compile the simplest programs is a direct call to cc for C or c++ for C++ programs ( Actually, nowadays g++ is nothing more than just a faster way of calling gcc -xc++ -lstdc++ -shared-libgcc).

cc helloworld.c
# compiles helloworld.c, generates 'a.out' executable

cc helloworld.c -o helloworld
# compiles helloworld.c, generates 'hellworld' executable

However, the calls to compilers quickly get messy if the project gets more complex than just several single files. Below is a random excerpt of real world GCC call:

/usr/loca/gcc-4.9/bin/gcc -maix32 -c -DPERL_CORE -D_ALL_SOURCE -D_ANSI_C_SOURCE \
-D_POSIX_SOURCE -DUSE_NATIVE_DLOPEN -fno-strict-aliasing -pipe -I/usr/local/include \
-D_LARGE_FILES -D_FORTIFY_SOURCE=2 -std=c89 -O -Wall -Werror=declaration-after-statement \
-Wextra -Wc++-compat -Wwrite-strings caretx.
/usr/loca/gcc-4.9/bin/gcc -maix32 -o miniperl -Wl,-brtl -Wl,-bdynamic -L/usr/local/lib -Wl,-b32 \
-Wl,-bmaxdata:0x80000000 perlmini.o opmini.o miniperlmain.o  gv.o toke.o perly.o pad.o regcomp.o \
dump.o util.o mg.o reentr.o mro_core.o keywords.o hv.o av.o run.o pp_hot.o sv.o pp.o scope.o pp_ctl.o \
pp_sys.o doop.o doio.o regexec.o utf8.o taint.o deb.o universal.o globals.o perlio.o perlapi.o numeric.o \
mathoms.o locale.o pp_pack.o pp_sort.o caretx.o   -lpthread -lbind -lnsl -lgdbm -ldbm -ldl -lld -lm -lcrypt -lc

Since it is unreasonable to manually type in these commands every time you want to compile a program, build tools are introduced to generate and run these commands automatically.

Make

Make is a build tool invented in the late 1970s, almost as old as cc itself. The idea of Make is very simple: the programer writes down some targets, as well as the commands that generate these targets, in a file called Makefile. A target defined in Makefile can either depend on another target or depend on files; make will read this Makefile, check if the targets are generated / up-to-date, and run the commands if they are not. Here is an example:

app : main.o utils.o
        cc -o app main.o utils.o
main.o : main.c defs.h
        cc -c main.c
utils.o : utils.c defs.h
        cc -c utils.c
clean :
        rm app main.o utils.o

So the Makefile above defines four targets: app, main.o, utils.o, and clean, while the target app depends on the targets main.o and utils.o, main.o depends on main.c and defs.h, utils.o depends on utils.c and defs.h, and clean depends on nothing. Targets and their dependencies are written in the same line, and the rules about how to generate them are written directly under it. For example, if we modify utils.c and invoke make app, make will notice that the timestamp of utils.c is newer than that of utils.o, thus calling automatically cc -c utils.c to update utils.o, then re-compile app, since utils.o is now newer than app.

Autotools

Makefile is a primitive tool. As a program becomes larger, especially when it has to be compiled on different machines or even different platforms, maintaining a flexible Makefiles could easily become a pain. A standardized way to solve this problem was established in the early 90s, that is, using a shell script called configure to generate/update the Makefile automatically. This is how the famous ./configure && make started, or as the GNU calls it, a GNU Build System was born. Here is an example from GNU, showing the Build System in action:

~ % tar zxf amhello-1.0.tar.gz
~ % cd amhello-1.0
~/amhello-1.0 % ./configure
…
config.status: creating Makefile
config.status: creating src/Makefile
…
~/amhello-1.0 % make
…
~/amhello-1.0 % make check
…
~/amhello-1.0 % su
Password:
/home/adl/amhello-1.0 # make install
…
/home/adl/amhello-1.0 # exit
~/amhello-1.0 % make installcheck
…

The GNU Coding Standards specifies some common targets, like all, install, install-strip, uninstall, clean, check, etc. for the Makefile, as well as some parameter or variables that should be accepted by the configure script. If you implement them, other people can simply build your code by following muscle memory. This convention is easy to learn, and flexible enough for many Unix developers, so it is still popular almost thirty years later today. It is not uncommon to see projects that maintain the configure and Makefile (or its template, Makefile.in) by hand. Yet some developers still don’t feel the process automated enough, so they went a further step forward and wrote the Autotools, which are basically tools that generate configure and Makefile for you. So the whole source code distribution process with Autotools can be summarized as the following steps:

  1. The developer writes/generates Makefile.am and configure.ac
  2. The developer calls automake which generates Makefile.in by reading Makefile.am
  3. The developer calls autoconf which generates configure by reading configure.ac
  4. The developer distributes the generated configure and Makefile.in as part of the source code
  5. The user calls configure which examines the system and converts Makefile.in into Makefile
  6. The user calls make that build the source code

CMake

GNU Autotools has its limitations, one being that it is not straightforward to learn, but the major problem is that it is (almost) Unix-only. So in the year 1999, CMake was invented to support real multi-platform build for C/C++ projects.

Despite the name, CMake has little to do with Make, the only relationship between the two is that CMake can be used to generate Makefiles on Unix, i.e. it is a tool which generates files that can be used to build C/C++ projects: Makefile or Ninja for Linux, Xcode projects for macOS, and Microsoft Visual Studio projects for Windows. It is a separated step to actually start the build process by opening/calling these generated files, you can either do this by hand or as the CMake to call them for you. Here is an example:

cmake_minimum_required(VERSION 2.8.9)
project (prj)
add_executable(app main.c)

By adding three lines, we defined a project named prj, with a target app which should be an executable, and it should be compiled from source main.c. Now if you save these three lines into a file named CMakeLists.txt and put it together with a reasonable main.c, then run cmake . on a Linux machine with GCC and build tools properly installed, you will see an output similar to the following:

-- The C compiler identification is GNU 4.9.2
-- The CXX compiler identification is GNU 4.9.2
-- Check for working C compiler: /usr/local/gcc-4.9.2/bin/gcc
-- Check for working C compiler: /usr/local/gcc-4.9.2/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/local/gcc-4.9.2/bin/g++
-- Check for working CXX compiler: /usr/local/gcc-4.9.2/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/metaphox/code/gaas-example

If you check the directory, among other files, a Makefile is created. If you run make in this directory, an app executable will be built.

What happens if we run cmake . for exactly the same code on Windows? The following is the output of cmake . on my Windows with Visual Studio 2015 installed:

-- Building for: Visual Studio 14 2015
-- The C compiler identification is MSVC 19.0.24210.0
-- The CXX compiler identification is MSVC 19.0.24210.0
-- Check for working C compiler using: Visual Studio 14 2015
-- Check for working C compiler using: Visual Studio 14 2015 -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler using: Visual Studio 14 2015
-- Check for working CXX compiler using: Visual Studio 14 2015 -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: C:/repo/cmaketest

And if we check the current directory:

C:\repo\cmaketest>dir
 Datenträger in Laufwerk C: ist System
 Volumeseriennummer: 1AAA-3821

 Verzeichnis von C:\repo\cmaketest

22.08.2019  12:50    <DIR>          .
22.08.2019  12:50    <DIR>          ..
22.08.2019  12:50            55.795 ALL_BUILD.vcxproj
22.08.2019  12:50               272 ALL_BUILD.vcxproj.filters
22.08.2019  12:50            65.773 app.vcxproj
22.08.2019  12:50               545 app.vcxproj.filters
22.08.2019  12:50            11.679 CMakeCache.txt
22.08.2019  12:50    <DIR>          CMakeFiles
22.08.2019  12:49                82 CMakeLists.txt
22.08.2019  12:50             1.268 cmake_install.cmake
22.08.2019  12:49                71 main.c
22.08.2019  12:50             3.132 prj.sln
22.08.2019  12:50            39.654 ZERO_CHECK.vcxproj
22.08.2019  12:50               509 ZERO_CHECK.vcxproj.filters
              11 Datei(en),        178.780 Bytes
               3 Verzeichnis(se), 59.097.427.968 Bytes frei

So CMake generated prj.sln solution file, as well as the ALL_BUILD, ZERO_CHECK, and app project files. These are Visual C++ project files that can be open and built with Visual Studio, but we can also simply build them with the msbuild command-line tool shipped with Visual Studio by calling msbuild app.vcxproj, and an app.exe will be compiled.

You can specify what kind of build tools you would like to use with CMake, these are called “Generators” and can be passed with -G option, for example cmake . -G "Unix Makefiles" will generate Makefiles (which is the default option on Linux), cmake . -G "Visual Studio 14 2015 Win64" will generate Visual Studio solution files for x86_64 architecture,cmake . -G "XCode"will generate Apple XCode project files, andcmake . -G “Eclipse CDT4 - Ninja”` will generate Ninja-based Eclipse CDT4 project files (whatever that is).

Do notice that, although we ran make and msbuild by hand, it was just for illustration purpose. CMake does support calling the build tool for you, which is simply cmake --build <build-dir>.

As you may see, CMake is powerful for its ability to generate build-tool specific files across multiple platforms, which is of great importance for portable C++ projects, and that is why it has become the de-facto standard for such projects.

Out-of-Source Build

A common practice with CMake worth mentioning here is out-of-source build. Citing a documentation we recommend you to read:

“Out-of-source build” is a good practice of keeping separately generated files from binary tree and source files from source tree. CMake do support contrary “in-source build” layout but such approach has no real benefits and unrecommended.

That is, if you are using CMake, it is better to create a dedicated directory for just the build process, so the source code would not be littered with unnecessary files produced by the build process. This directory needs not to be really “out” of the source tree, you can just create a build directory directly under the root of the source code. For example, in our tutorial E06, we built the Kernelized Correlation Filters (KCF) algorithms with:

cd (GAAS-Object-Tracking_PATH)/KCF
mkdir build
cd build
cmake ..
make -j6

So here we are doing out-of-source build, and calling make in a separated step for clarity.

Headers and Libs

A major and maybe the most common headache of compiling C++ projects is missing header files and/or missing libraries.

The GNU Coding Standards define a set of standard configuration variables used during the build. Here are some:

  • CC: C compiler command
  • CFLAGS C compiler flags
  • CXX C++ compiler command
  • CXXFLAGS C++ compiler flags
  • LDFLAGS linker flags
  • CPPFLAGS C/C++ preprocessor flags

Troubleshooting | 常见问题排查


How to Ask Questions | 论坛提问指南