Commit 1aed175e authored by Oliver Horst's avatar Oliver Horst
Browse files

[add] Initial draft of the overall toki documentation

Pipeline #18197 failed with stages
in 3 minutes and 42 seconds
## Introduction to toki
toki, Japanese for "when an action occurs", is a build- and test-environment constructed around the real-time operating system FreeRTOS. toki's main goal is to compose an easy to use rapid prototyping platform to develop and evaluate new operating system and inter-task communication concepts for embedded and cyber-physical systems on commodity micro controller hardware. Therefore, toki combines the following open-source projects into a unified build system:
* [FreeRTOS v10.x](
* ESFree
* EDFxxx
* [newlib vX.Y](
* [libXil v2019.1](
* [lwIP v2.x](
All components of the toki platform were chosen under the aspect of simplicity and easiness of modifications and extensions to the code base. A special attention was put on the fact that the utilized components, on the one hand, closely mimic the software stacks found in comparable industrial systems, and on the other hand still do not impede the prototypical realization of ideas with conceptional obstacles. Hence, the operating system of choice is FreeRTOS, a widely supported, well documented, and industry ready real-time operating system, which is also available in a safety certified version SafeRTOS.
At the overall platform level, the toki build-system is utilizing Yocto to provide a self contained build environment that not only produces the final image that can be used to flash the actual hardware platform, but also all of its dependencies, such as a GCC cross-compiler. At the project level, the source code of the contained open-source projects was restructured and extended with a CMake based build-system to support include prefixes, where needed. This reconstruction is conducted in an automatic fashion by dedicated restructuring script, to allow fetches of upstream changes. More details on that restructuring process and the scripts behind it are discussed in Section XXX. In addition, the generated cross-compiler together with a CMakeToolchain file, both provided by Yocto, and an additionally provided top-level CMake file allow you to enable full code insight and debugging capabilities within the IDE of your choice.
Besides the changes made to support an integrated build environment for the software components mentioned above, toki extends the functionality of the components. Namely, toki provides a
* AMP multi-core boot for FreeRTOS,
* newlib syscalls for FreeRTOS,
* user and kernel space separation via a syscall interface in FreeRTOS,
* memguard implementation for FreeRTOS on ARMv8, and
* STM tracing support.
The aforementioned toki build-environment is seamlessly integrated with a test-environment based on Qemu. All images built with Yocto in the toki build environment can either be run directly on the target hardware via flashing an SD card, or emulated and debugged via Qemu and its GDB stub support. The build-environment already provides you with an adequate built of Qemu.
Currently, the toki build- and test-environment is solely targeted at the Xilinx Zynq UltraScale+ MPSoC ZCU102 Evaluation Kit, a contemporary ARMv8 multi-core SoC with an integrated FPGA. Nevertheless, toki was designed with portability in mind, hence, it can easily be ported to other target platforms, such as XXX by STmicro. The Xilinx ZCU102 evaluation kit was chosen as the designated toki hardware platform, due to the fact that Xilinx provides a bare-metal board support package with drivers for all integrated peripherals, and the integrated FPGA allows to extend the hardware for instance to support a more accurate performance evaluation of the system.
## Using toki a.k.a. Setting up your own toki build- and test-environment
To use toki, the following dependencies need to be installed on your system:
* curl
* bash
* python >= 3.6
* dialog (pip + system)
* git >= ?
* cmake >= 3.7
* docker
* ?
For Ubuntu based systems, please run:
$ sudo apt-get install
To prepare the setup you shall furthermore obtain a personal access token from GitLab with at least 'api' and 'read_repository' access rights. This token is needed in the following steps of the setup process.
The actual setup of the toki build- and test-environment is conducted by an automatic setup script. Please download and execute the script by executing:
$ curl -s --request GET --header 'PRIVATE-TOKEN: [YOUR PRIVATE TOKEN]' '' | bash
The script will download and setup the toki build- and test-environment within the folder you are executing it in, it will guide you through the setup process. The script will ask you to provide once again your personal GitLab access token, to clone all repositories, this token will be stored in a file called 'secrets.txt' (only accessible by your user) for future calls to the toki helper scripts.
Currently, Yocto only supports Ubuntu 16.04 based systems and thus toki copies this requirement. Nevertheless, we have tested the toki environment successfully on Ubuntu 18.04 and do support other distributions through the means of a tailored Docker container. Please read Section CCC for details on how-to setup the Docker container for toki.
In the next Section YYY, the folder structure created by the setup script is explained. Section ZZZ continues with instructions on how-to use toki and Section AAA provides you with an insight on how-to setup your IDE for developing and debugging with toki.
## toki Folder Structure
##### _build
The build folder contains all build related, temporary files for both build systems CMake and Yocto. It is safe to delete all files underneath this folder, however, as this induces a re-initialization of the Yocto build environment, we advise you to use the reset functionality of the toki setup script instead. It will delete all temporary files, except for the downloaded external sources and correctly set up the Yocto environment again. Just run the following command from within your toki environment:
( cd $TOKI_SRC ; ./auxiliary/ --reset )
There are three sub-folders located within the '_build' folder. 'cmake' and 'yocto' do keep the build specific files of the two build systems, and 'yocto_downloads' keeps all downloaded external sources required by the Yocto build process.
##### applications
##### auxiliary
More details on the available toki helper scripts can be found in the dedicated Section AAA.
##### bootimages
##### build-systems
* cmake
* yocto
##### components
##### configs
##### hardware
##### init
## How-to use toki
Every time you would like to use the toki build- and test-environment you shall open a new terminal and source the toki environment by running the following command with the toki source folder as working dir (see Section BBB on how to correctly setup the toki source folder):
source toki-env
This initializes Yocto for you and defines the following environment variables
* _TOKI_SRC_DIR_. Points to the root dir of your toki setup.
* _CMAKE_BUILD_DIR_. Points to the CMake build directory, the default is '$TOKI_SRC_DIR/_build/cmake'.
* _YOCTO_BASE_DIR_. Points to the directory that holds the Yocto main sources, i.e., '$TOKI_SRC_DIR/build_systems/yocto/poky'.
* _YOCTO_BUILD_DIR_. Points to the directory where Yocto places its (temporary) build files, the default is '$TOKI_SRC_DIR/_build/yocto'.
* _YOCTO_DOWNLOAD_DIR_. Do allow an easy deletion of the Yocto build folder without the necessity to download all external sources again during the next build, toki advises Yocto to use a separate download directory, its default location is '$TOKI_SRC_DIR/_build/yocto_downloads'.
## Setting up your IDE for toki
If you have no done during the setup process you now need to first build the IDE support with Yocto before you proceed
After the setup script finished its task, you will find a
usage of build env
source oe...
die eigentlichen usage istructions werden dann automatisch angezeigt
hier dann auch noch mal wiederholen, idealerweise einfach so wie am bildschirm angezeigt auch hier zeigen
erstmal nur am minimal beispiel
setup script
manual process
as soon as you have initialized your toki environment by sourcing the toki-env script, you can use git toki xxx to execute git commands on all registered projects
## Further Reading
##### CMake
[A brief introduction to CMake and its concepts](
[A more detailed introduction](
[CMake cross compilation / toolchain files]()
[The CMake reference manual](
##### Yocto
[Overview Manual](
[Reference Manual](
[BitBake User Manual](
##### Qemu
QEMU is a generic and open source machine emulator and virtualizer.
we use it for emulating the target platform hw on a comodity Linux PC, such enabling devleopers to develop most of the time cost efficiently on their Linux desktop pc.
Qemu GDB Stub
incl. suitable default configurations
toki provides an easy to use build system based on Yocto for buiulding the cross-compilation toolchain and full-blown images to flash the target with, as well as CMake for building the individual components.
toki extends the original feature set of FreeRTOS with features like
toki das zeit os
gedacht für rapid prototyping mit einem echten embedded os auf echter embedded hw
einige Features der toki build umgebung machen aber evtl. keinen sinn für andere plattformen. so setzen wir fürs debuggen das selbe sd karten image ein, wie für das flashen des boards verwendet werden kann. so geben wir dem nutzer die möglichkeit 4 kerne zu spezifizieren und ein fpga hardware image auszuwälen. der gesamt image bau prozess ist dabei xilinx spezifisch. mit vertretbarem aufwand lassen sich diese sachen aber auch generalisierne und somit für andere platform lösungen zugänglich machen.
für die einfache nutzung stellen wir ein docker image zur verfügung welches es ermöglicht den yocto build prozess auch auf nicht ubuntu machinen einsetzen zu können.
die cmake umgebung kann aber auch standalone genutzt werdne um binaries zu erstellen, die dann mittels eines jtag debuggers aufs target geflasht werden können. dies ermöglicht auch die einbindung in entsprechende IDEs inkl. einem komplettebn code insight. allerdings hat der cmake bau prozess die einschränkung, dass immer nur eine applikation für einen kern gebaut werden kann. kompliziertere setups mit mehreren applikationen auf verschiedenen kernen und mit verschiedenenen os konfigurationen werden nur mit yocto unterstützt.
hier muss entsprechend auch etwas sorgfalt walten gelassen werden, denn es lässt sich auch immer nur eine applikation entwickeln zur gleichen zeit. so dass man sorgfältig die CMake Parameter anpassen muss, wenn man zu einem anderen anwendungsszenario wechselt.
designt als flexibles AMP System, bei dem jeder core eine eigene OS Instanz mit einem eigenen, unabhängigen Scheduler ausführt, diese Instanzen sich aber über einen Kommunikationskanal austauschen/synchronisieren. referenz auf EWC paper. wichtig war dabei das augenmerk auf eine einfache erweiterung der code basis um neue features bzw. das ersetzen existierender funktionen durch neue konzepte im bereich scheduling und inter prozess kommunikation. aber gleichzeitig auch eine hoch präsize messplatform für die evaluierung von neuen ideen.
die high-level struktur des toki projekts gliedert sich in folgende Kategorien auf:
- hier sollten dann die einzelnen ordner und unterordner erklärt werden, was man wo findet und wieso das so strukturiert ist.
provides you with hardware specific boot images that can be used to initialize the target hardware
fpga configurations/projects that can be used to extend the hw platform
all software components that toki is based on
sub.divded into cmake and yocto this sub-project holds all necessary files for the build systems to work. the cmake project contains only auxiliary files that are used to build the toki environment from within an ide solely with cmake.
toki Build and Test system:
toki Components
hier sollte auch erwähnt werden, dass abweichend von den upstream konfigurationen die projekte umstruktiert wurden, so dass man einen sauberen, gekapselten buuild prozess mit header rpefixen hinbekommt, da es ansonsten z.B. bis zu 3 time.h Dateien gibt, etc. mehr zu dem umstrukturierungsprozess verraten die "pull upstream changes" und "adding additional components" abschnitte.
- linker script
toki Extensions
FreeRTOS Syscall interface
kurz und knapp die beduetung und den aufbau skizzieren
getcurrenttime, etc. syscalls
man pages und RFC referenzieren
EDF scheduler
original arbeiten referenzieren
paper referenzieren
STM Tracing
auf die arm doku verweisen und den sinn und nutzen beschreiben
configure your own deployment
menuconfig script und was man dort einstellen kann
helper scripts
git toki xxx
pull upstream changes
pull script options
pull configs
adding additional components:
cmake scripts
yocto recipes
JSON package config for CMake
existing and extending pull scripts
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment