Gerard Braad’s Knowledge Base

A personal reference for tools, platforms, languages, and workflows collected over time.

Contents

• Containers & Cloud - Docker, Kubernetes, OpenShift, OpenStack, Ceph
• Linux - ARM, RISC-V, Raspberry Pi, storage, virtualization
• Development - Rust, Python, Go, Git, shell, audio
• Networking - nginx, SSH, traefik, curl
• Security - tools and hardening
• Books - reading notes
• Language - Chinese tech vocabulary, English

Containers & Cloud

Notes on container runtimes, orchestration, and cloud platforms.

• Containers - container runtimes and fundamentals
• Docker - Docker CLI, Compose, and Containerfiles
• Atomic - Project Atomic / rpm-ostree based systems
• cloud-init - cloud instance initialization
• Cloud Foundry - PaaS platform
• Ceph - distributed storage
• ManageIQ - cloud management platform
• Kubernetes - container orchestration
• OpenShift - Kubernetes distribution by Red Hat
• OpenStack - open source cloud infrastructure

Containers

A software packaging concept that typically includes the application and all of the runtime dependencies

Implementations

• Docker
• runc
• rkt

Docker

Containers and registries

Docker hub

• Personal repos

GitLab

• devenv
• Jupyter
• Kubernetes client
• Mono
• OpenStack client
• ostreetests
• Openshift client

Proxies (at GitLab)

• Alpine
• CentOS + custom
• Debian
• Fedora + custom
• openSUSE
• Photon
• Ubuntu

Test wrappers (at GitLab)

• CentOS min
• Fedora min

Get private IP address of container

docker inspect --format="{{.NetworkSettings.IPAddress}}" [container id or name]

Use LVM thin pool

$ systemctl stop docker
$ rm -rf /var/lib/docker
$ pvcreate /dev/vdb
$ vgcreate docker_vol /dev/vdb
$ vi /etc/sysconfig/docker-storage-setup 

VG="docker_vol"

$ docker-storage-setup

Use OverlayFS

Docker container engine

If running Docker engine from docker.io:

systemctl stop docker
vi /usr/lib/systemd/system/docker.servic

and add --storage-driver=overlay to ExecStart.

rm -rf /var/lib/docker/  # This will remove your existing images/containers
systemctl daemon-reload
systemctl start docker

Verify with:

docker info | grep Storage

overlay will be shown instead of devicemapper.

Fedora/CentOS

For Fedora/CentOS you can use docker-storage-setup to setup the storage driver.

To use OverlayFS

$ systemctl stop docker
$ vi /etc/sysconfig/docker-storage

`DOCKER_STORAGE_OPTIONS= -s overlay`

$ rm -rf /var/lib/docker
$ docker-storage-setup
$ systemctl start docker

Note:

• This involves removing existing images and containers!
• SELinux is not supported with the OverlayFS driver

Source, more info GitHub

Registry

• Registry API v2

SELinux errors

If you get ‘permission denied’ from the container and you use CentOS/Fedora, you likely ran into SELinux permission not given:

Relabel the target with:

$ chcon -Rt svirt_sandbox_file_t [location]

Remove untagged images

$ docker rmi $(docker images -a | grep "^<none>" | awk '{print $3}')

$ docker rmi $(docker images -f "dangling=true" -q)

Export / Import

Used for containers

$ docker run -d --name devenv gbraad/c9ide:f24
$ docker stop devenv
$ docker export devenv > devenv-f24.tar

Save / Load

Used for images

$ docker save gbraad/c9ide:c7 > c9ide-c7.tar 

My repositories (on GitHub; after rename)

Source for images

• https://github.com/gbraad/dockerfile-openshift-client
• https://github.com/gbraad/dockerfile-gitbook-server
• https://github.com/gbraad/dockerfile-openstack-keystone
• https://github.com/gbraad/dockerfile-gollum
• https://github.com/gbraad/dockerfile-c9ide
• https://github.com/gbraad/dockerfile-flatpak-builder-gnome
• https://github.com/gbraad/dockerfile-flatpak-builder-freedesktop
• https://github.com/gbraad/dockerfile-jupyter
• https://github.com/gbraad/dockerfile-mono
• https://github.com/gbraad/dockerfile-openstack-client
• https://github.com/gbraad/dockerfile-kubernetes-client
• https://github.com/gbraad/dockerfile-glusterfs
• https://github.com/gbraad/dockerfile-flatpak
• https://github.com/gbraad/dockerfile-httpd

Containerized libvirt

• https://github.com/gbraad/dockerfile-image-libvirt
• https://github.com/gbraad/dockerfile-image-libvirt-client

Brew of wellknown distributions

• https://github.com/gbraad/dockerfile-brew-debian
• https://github.com/gbraad/dockerfile-brew-ubuntu-core
• https://github.com/gbraad/dockerfile-brew-fedora
• https://github.com/gbraad/dockerfile-alpine

Source for GitLab proxies/container registries

• https://github.com/gbraad/container-registry-cirros
• https://github.com/gbraad/container-registry-arm
• https://github.com/gbraad/container-registry-steam
• https://github.com/gbraad/container-registry-fedora
• https://github.com/gbraad/container-registry-centos
• https://github.com/gbraad/container-registry-opensuse
• https://github.com/gbraad/container-registry-photon
• https://github.com/gbraad/container-registry-debian
• https://github.com/gbraad/container-registry-docker
• https://github.com/gbraad/container-registry-ubuntu

Project Atomic

• See also ostree
• Getting started

Fedora Atomic

• Landingpage
• Download images

CentOS Atomic

• Landingpage
• Download images

Kubernetes on Atomic

Deploy kubernetes on CentOS and CentOS Atomic hosts using Ansible

• About Kubernetes
• Ansible playbook
• Getting started with containers
• Getting started guide at Project Atomic

Atomic Reactor

On Fedora 24 or CentOS 7

$ dnf install -y atomic-reactor
$ docker pull registry.gitlab.com/gbraad/fedora:atomicreactor
$ docker tag registry.gitlab.com/gbraad/fedora:atomicreactor buildroot-hostdocker
$ atomic-reactor build git --method hostdocker \
             --build-image buildroot-hostdocker \
             --image hello-world \
             --target-registries 10.5.0.2:5000 \
             --uri "https://github.com/TomasTomecek/docker-hello-world.git"

More info

Note: currently FROM:scratch builds are not supported.

Build your own Atomic

• BYO Atomic
• BYO Atomic - base
  • Registry
• BYO Atomic - Toolbox

Creating a custom image

You can add repos simply by including them in the same folder as your myostree.json definition.

In the json file you define:

{
    "include": "centos-atomic-host.json",
    "ref": "centos-atomic-host/7/x86_64/my-software",
    "automatic_version_prefix": "2016.0",
    "repos": ["myrepo"],
    "packages": [
        "my-sofware"
     ]
}

and create a file myrepo.repo containing:

[myrepo]
name=My software repo
baseurl=http://gbraad.gitlab.io/mysoftware
gpgcheck=0

Variant: CentOS

Automated GitLab CI runner is available: BYO Atomic - CentOS

Instructions

Use Fedora as basis for the buildserver. In this example F24-Cloud is used.

dnf install -y git rpm-ostree rpm-ostree-toolbox python

git clone https://github.com/CentOS/sig-atomic-buildscripts
cd sig-atomic-buildscripts
git checkout downstream

mkdir -p /srv/repo
ostree --repo=/srv/repo init --mode=archive-z2

rpm-ostree compose tree --repo=/srv/repo sig-atomic-buildscripts/centos-atomic-host.json

cd /srv/repo
python -m SimpleHTTPServer

sudo ostree remote add foo http://$YOUR_IP:8000/repo --no-gpg-verify
sudo rpm-ostree rebase foo:centos-atomic/7/x86_64/docker-host

Variant: Fedora

Automated GitLab CI runner is available: BYO Atomic - Fedora

Instructions

git clone https://git.fedorahosted.org/git/fedora-atomic.git
cd fedora-atomic
git checkout f23

Upstream ostree

Using CentOS continuous ostree

If you are running an existing Atomic image

ostree remote add --set=gpg-verify=false centos-atomic-continuous https://ci.centos.org/artifacts/sig-atomic/rdgo/centos-continuous/ostree/repo/
rpm-ostree rebase centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous
systemctl reboot

Source

Using Fedora Workstation ostree

From inside an existing system:

yum -y install ostree
ostree admin init-fs /
ostree remote add --set=gpg-verify=false fedora-ws-centosci https://ci.centos.org/artifacts/sig-atomic/rdgo/fedora-workstation/ostree/repo/
ostree --repo=/ostree/repo pull fedora-ws-centosci:fedora/24/x86_64/desktop/continuous
ostree admin os-init fedora
ostree admin deploy --os=fedora --karg-proc-cmdline --karg=enforcing=0 fedora-ws-centosci:fedora/24/x86_64/desktop/continuous
cp /etc/fstab /ostree/deploy/fedora/deploy/$checksum.0/etc
#  (You may need to modify the fstab)
grub2-mkconfig -o /boot/grub2/grub.cfg

If you already deployed F23 and want to rebase to F24:

  rpm-ostree rebase fedora/24/x86_64/desktop_continuous

Source, See also

PXE boot Atomic image

Steven Dake describes the process of converting an Atomic image to be used by Ironic for PXE-boot.

Grow the / filesystem

lvremove /dev/mapper/atomicos-docker--pool
lvresize -l +100%FREE /dev/atomicos/root
xfs_growfs /dev/atomicos/root

Note: this trashes the docker storage pool

Atomic on OpenStack

$ wget http://http://cloud.centos.org/centos/7/atomic/images/CentOS-Atomic-Host-7-GenericCloud.qcow2
$ openstack image create "CentOS7 Atomic" --progress --file CentOS-Atomic-Host-7-GenericCloud.qcow2 --disk-format qcow2 --container-format bare --property os_distro=centos

$ wget https://download.fedoraproject.org/pub/alt/atomic/stable/CloudImages/x86_64/images/Fedora-Atomic-24-20160809.0.x86_64.qcow2
$ openstack image create "Fedora24 Atomic" --progress --file Fedora-Atomic-24-20160809.0.x86_64.qcow2 --disk-format qcow2 --container-format bare --property os_distro=fedora

Note: for Ceph you might have to convert-img to RAW format.

Atomic using Vagrant

CentOS

$ vagrant init centos/atomic-host && vagrant up --provider virtualbox 
# or  --provider libvirt

Source

Fedora

$ vagrant init fedora/24-atomic-host && vagrant up --provider virtualbox
# or  --provider libvirt

Source

Using OpenStack DiskImage Builder (Magnum)

Create the ostree

git clone https://pagure.io/fedora-atomic.git -b f24
ostree init --repo=/srv/repo --mode=archive-z2
rpm-ostree compose tree --repo=/srv/repo ~/fedora-atomic/fedora-atomic-docker-host.json

ostree trivial-httpd -d -P 9001 /srv/repo

Create the image

git clone https://git.openstack.org/openstack/magnum
git clone https://git.openstack.org/openstack/diskimage-builder.git
git clone https://git.openstack.org/openstack/dib-utils.git

export PATH="${PWD}/dib-utils/bin:$PATH"
export PATH="${PWD}/diskimage-builder/bin:$PATH"

export ELEMENTS_PATH="${PWD}/diskimage-builder/elements"
export ELEMENTS_PATH="${ELEMENTS_PATH}:${PWD}/magnum/magnum/drivers/common/image"

export DIB_RELEASE=24
export DIB_DEBUG_TRACE=1
export DIB_IMAGE_SIZE=3.0

export FEDORA_ATOMIC_TREE_URL=http://localhost:9001
export FEDORA_ATOMIC_TREE_REF=$(cat /srv/f24ah/refs/heads/fedora-atomic/24/x86_64/docker-host)

mkdir -p ~/output

disk-image-create fedora-atomic -o ~/output/fedora-atomic

Ref

Other resources

• Cloud-init
• Compose server
• Photon OS, wiki
• Testing Atomic for OpenStack

Cloud-init

Post-creation script

A simple example of using a post-creation script, as possible with OpenStack, is as follows:

#cloud-config
password: passw0rd
chpasswd: { expire: False }
ssh_pwauth: True

This will set a password for the login user (differs per image?!) and allow this user to login using ssh.

Force run of cloud-init script

You can just run it like this:

/usr/bin/cloud-init -d init

This runs the cloud init setup with the initial modules. Note: the -d option is for debug.

If want to run all the modules you have to run:

/usr/bin/cloud-init -d modules

Keep in mind that the second time you run these it doesn’t do much since it has already run at boot time. To force to run after boot time you can run from the command line:

( cd /var/lib/cloud/ && sudo rm -rf * )

Always run cloud-init scripts

In 11.10, 12.04 and later, you can achieve this by making the ‘scripts-user’ run ‘always’. In /etc/cloud/cloud.cfg you’ll see something like:

cloud_final_modules:
 - rightscale_userdata
 - scripts-per-once
 - scripts-per-boot
 - scripts-per-instance
 - scripts-user
 - keys-to-console
 - phone-home
 - final-message

This can be modified after boot, or cloud-config data overriding this stanza can be inserted via user-data. Ie, in user-data you can provide:

#cloud-config
cloud_final_modules:
 - rightscale_userdata
 - scripts-per-once
 - scripts-per-boot
 - scripts-per-instance
 - [scripts-user, always]
 - keys-to-console
 - phone-home
 - final-message

More info

Config drive

Most virtual platforms will provision instances with no root password, so it’s necessary to supply a password or key to log in using cloud-init. If you’re using a virtualization application without cloud-init support, such as virt-manager or VirtualBox, you can create a simple iso image to provide a key or password to your image when it boots.

To create this iso image, you must first create two text files.

Create a file named meta-data that includes an “instance-id” name and a “local-hostname.” For instance:

instance-id: instance0
local-hostname: instance-00

The second file is named user-data, and includes password and key information. For instance:

#cloud-config
password: secrete
chpasswd: {expire: False}
ssh_pwauth: True
ssh_authorized_keys:
  - ssh-rsa AAA...EFus user1@mydomain.com
  - ssh-rsa AAB...RDie user2@mydomain.com

Once you have completed your files, they need to packaged into an ISO image. For instance:

# genisoimage -output instance0-cidata.iso -volid cidata -joliet -rock user-data meta-data

You can boot from this iso image, and the auth details it contains will be passed along to your instance.

For more information about creating these cloud-init iso images, see config-drive.

Automated creation of configdrive

• Project at GitLab and output

Cloud Foundry

Ceph

Deployment

• ceph-ansible
• ceph-docker

Play ceph-ansible with CentOS cloud hosts

site.yml

- hosts: mons # gather mon facts first before we process ceph-mon serially
  tasks: []

- hosts: mons
  become: True
  gather_facts: false
  roles:
  - ceph-mon
  serial: 1

- hosts: osds
  become: True
  roles:
  - ceph-osd

hosts

[mons]
centos-01 ansible_ssh_host=10.3.0.15
centos-02 ansible_ssh_host=10.3.0.18
centos-03 ansible_ssh_host=10.3.0.14

[osds]
centos-04 ansible_ssh_host=10.3.0.16
centos-05 ansible_ssh_host=10.3.0.17
centos-06 ansible_ssh_host=10.3.0.19

group_vars/all

cluster_network: 10.3.0.0/24
generate_fsid: 'true'
ceph_origin: upstream
ceph_stable: true
monitor_interface: eth0
public_network: 10.3.0.0/24
journal_size: 5120

group_vars/mons

cephx: false

group_vars/osds

journal_collocation: true
devices:
   - /dev/vdb
cephx: false

Play ceph-ansible with Atomic hosts

hosts

[mons]
atomic-01 ansible_ssh_host=10.3.0.15
atomic-02 ansible_ssh_host=10.3.0.18
atomic-03 ansible_ssh_host=10.3.0.14

[osds]
atomic-04 ansible_ssh_host=10.3.0.16
atomic-05 ansible_ssh_host=10.3.0.17
atomic-06 ansible_ssh_host=10.3.0.19

ansible.cfg

[defaults]
action_plugins = plugins/actions
retry_files_enabled = False
host_key_checking = False
remote_user = centos

$ ansible -i hosts all -m ping
atomic-05 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
atomic-01 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
atomic-04 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
atomic-03 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
atomic-06 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
atomic-02 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

$ cp site.yml.sample site.yml
$ cp group_vars/all.sample group_vars/all
$ cp group_vars/mons.sample group_vars/mons
$ cp group_vars/osds.sample group_vars/osds

group_vars/all

mon_containerized_deployment: true
osd_containerized_deployment: true
mds_containerized_deployment: true
rgw_containerized_deployment: true
nfs_containerized_deployment: true
restapi_containerized_deployment: true
rbd_mirror_containerized_deployment: true
cluster_network: 10.3.0.0/24
ceph_docker_on_openstack: true
ceph_rgw_civetweb_port: 8080
generate_fsid: 'true'

ceph_docker_dev_image: false
ceph_origin: distro
monitor_interface: eth0
public_network: 10.3.0.0/24
journal_size: 5120

group_vars/osds

journal_collocation: true
devices:
   - /dev/vdb
ceph_osd_docker_devices: ['/dev/vdb']
ceph_osd_docker_extra_env: "CEPH_DAEMON=OSD_CEPH_DISK_ACTIVATE,OSD_JOURNAL_SIZE=100"
cephx: false

group_vars/mons

cephx: false
ceph_mon_docker_interface: eth0
ceph_mon_docker_subnet: 10.3.0.0/.0/24

# upgrade and reboot the nodes
$ ansible -i hosts all -m shell -a "rm -rf /etc/ceph/*.conf" -u centos -s
$ ansible -i hosts all -m shell -a "rm -rf /var/lib/ceph/*" -u centos -s
$ ansible-playbook -i hosts site.yml --skip-tags with_pkg

Ceph-ansible; Ceph Atomic

• https://gitlab.com/gbraad/ceph-atomic
• https://gitlab.com/gbraad/byo-atomic-ceph

$ git clone https://github.com/ceph/ceph-ansible.git
$ cp site.yml.sample site.yml
$ cp group_vars/all.sample group_vars/all
$ cp group_vars/mons.sample group_vars/mons
$ cp group_vars/osds.sample group_vars/osds

group_vars/mons

cluster_network: 10.3.0.0/24
generate_fsid: 'true'
skip_tags: 'with_pkg'
ceph_origin: distro
monitor_interface: eth0
public_network: 10.3.0.0/24
journal_size: 5120
cephx: false

group_vars/osds

journal_collocation: true
cephx: false
devices:
   - /dev/vdb

$ ansible -i hosts all -u centos -s -m shell -a "setenforce 0"
$ ansible -i hosts all -u centos -s -m shell -a "ostree remote add --set=gpg-verify=false byo-atomic-ceph https://gbraad.gitlab.io/byo-atomic-ceph/"
$ ansible -i hosts all -u centos -s -m shell -a "rpm-ostree rebase byo-atomic-ceph:centos-atomic-host/7/x86_64/ceph-jewel"
$ ansible -i hosts all -u centos -s -m shell -a "systemctl reboot"  # will throw errors as connection is dropped
$ ansible -i hosts all -u centos -s -m ping
$ ansible-playbook -i hosts site.yml --skip-tags with_pkg,package-install

Gist

Ceph-ansible: installation source

group_vars/all

#ceph_origin: distro
ceph_origin: upstream
ceph_stable: true

distro does not install any repos, while upstream will and needs you to specify and installation type; ceph_stable used here.

Building

• Build wrapper: GitLab, GitHub
  Runs automated builds of the master branch at ceph/ceph.

Standard build using cmake

The following script performs a basic build using cmake on CentOS 7 (Cloud image).

# prepare
$ yum install -y git
$ git clone git://github.com/ceph/ceph --depth 1
$ cd ceph
$ git submodule update --init --recursive
$ ./install-deps.sh
# build
$ ./do_cmake.sh
$ cd build
$ make

Links

• Ceph homepage

ManageIQ

Deployment

As container

$ docker pull manageiq/manageiq:latest-darga
$ docker run --privileged -di -p 80:80 -p 443:443 manageiq/manageiq:latest-darga

Credentials: admin/smartvm

Source

As image (OpenStack/KVM)

$ curl -O -L http://manageiq.org/download/manageiq-openstack-darga-3.qc2
$ openstack image create --name "manageiq" --disk-format qcow2 \
  --container-format=bare --file manageiq-openstack-darga-3.qc2

Kubernetes

Container orchestration platform originally developed by Google, now governed by the CNCF.

• OpenShift - enterprise Kubernetes distribution by Red Hat

Quick start

# Install kubectl
curl -LO "https://dl.k8s.io/release/$(curl -Ls https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl && mv kubectl /usr/local/bin/

# Basic commands
kubectl get nodes
kubectl get pods -A
kubectl apply -f manifest.yaml
kubectl logs <pod> -f
kubectl exec -it <pod> -- bash

Build from source

git clone https://github.com/kubernetes/kubernetes.git
cd kubernetes
make

Ansible roles

• gbraad.kubernetes-master
• gbraad.kubernetes-node
• gbraad.kubernetes-client
• Deployment playbook

Resources

• Hands-on-labs

OpenShift

Red Hat’s enterprise Kubernetes distribution. OpenShift Origin (now OKD) is the upstream community version.

• minishift - local single-node OpenShift for development

Client installation

# CentOS/RHEL
yum install -y centos-release-openshift-origin origin-clients

# Fedora
dnf install -y origin-clients

Start a local cluster

oc cluster up

oc login http://localhost:8443

Web console: http://localhost:8443

Ansible deployment

git clone https://github.com/openshift/openshift-ansible -b master

• openshift-ansible

Resources

• OpenShift Origin releases
• OKD (community)

Minishift

Build environment

#!/bin/sh
GO_TARGET=$HOME
GLIDE_TARGET=$HOME
SRC_TARGET=$HOME

if [ ! -x "${SRC_TARGET}/minishift" ]
then
    cd ${SRC_TARGET}
    git clone https://github.com/minishift/minishift.git
    cd ${SRC_TARGET}/minishift
else
    cd ${SRC_TARGET}/minishift
    git pull --reb
fi

if [ ! -x "${GO_TARGET}/go" ]
then
    OS=linux
    ARCH=amd64
    GO_VERSION=1.7.1
    GO_RELEASE_URL="https://storage.googleapis.com/golang/go${GO_VERSION}.${OS}-${ARCH}.tar.gz"
    curl -sSL ${GO_RELEASE_URL} -o /tmp/go.tar.gz
    mkdir -p ${GO_TARGET}/go
    tar --directory=${GO_TARGET} -xvf /tmp/go.tar.gz
    #rm /tmp/go.tar.gz
fi
export GOROOT=${GO_TARGET}/go/
export PATH=$PATH:${GO_TARGET}/go/bin

GLIDE_OS_ARCH=`go env GOHOSTOS`-`go env GOHOSTARCH`
if [ ! -x "${GLIDE_TARGET}/glide" ]
then
    GLIDE_TAG=$(curl -s https://glide.sh/version)
    GLIDE_LATEST_RELEASE_URL="https://github.com/Masterminds/glide/releases/download/${GLIDE_TAG}/glide-${GLIDE_TAG}-${GLIDE_OS_ARCH}.tar.gz"
    curl -sSL ${GLIDE_LATEST_RELEASE_URL} -o /tmp/glide.tar.gz
    mkdir -p ${GLIDE_TARGET}/glide
    tar --directory=${GLIDE_TARGET}/glide -xvf /tmp/glide.tar.gz
    #rm /tmp/glide.tar.gz
fi

export PATH=$PATH:${GLIDE_TARGET}/glide/${GLIDE_OS_ARCH}

export GOPATH=$PWD/_gopath
glide install -v
make

OpenStack

Open source cloud infrastructure platform. Originally developed by NASA and Rackspace; now governed by the OpenInfra Foundation.

• Keystone - identity service
• Client - OpenStack CLI client
• Puppet - Puppet modules for OpenStack
• RDO - community OpenStack distribution for RHEL/Fedora
• TripleO - OpenStack-on-OpenStack deployment
• Devstack - development/test deployment
• Fuel - Mirantis deployment tool
• PackStack - quick all-in-one deployment via Puppet

Core services

Infrastructure components

• Ceph - storage backend for Cinder, Glance, Nova
• Open vSwitch / Linuxbridge - Neutron backends
• RabbitMQ - message bus between services
• MariaDB + Galera - highly available database
• HAProxy + Keepalived - service high availability

Contributions

• Devstack configurations
• Packstack answerfiles
• Heat templates
• Hands-on-labs

OpenStack TripleO

Original TripleO (fix needed)

OpenStack client

• Documentation
• clouds.yaml: usage, support

Setup

Docker image

The easiest way is to use the Docker container I created: OpenStack client

$ alias stack='docker run -it --rm -v $PWD:/workspace -v ~/.stack:/root/.stack registry.gitlab.com/gbraad/openstack-client:centos stack'
$ alias openstack='docker run -it --rm -v $PWD:/workspace -v ~/.config/openstack:/root/.config/openstack registry.gitlab.com/gbraad/openstack-client:centos openstack'

Then create a configuration file at .stack for your environment, and prepend commands with:

$ stack trystack [command]

Install

Alternatively you can use the following installation methods

Using PIP

$ pip install python-openstackclient

On Fedora/RHEL/CentOS

$ yum install -y python-openstackclient   # dnf

On Ubuntu/Debian

$ apt-get install -y python-openstackclient

Using Ansible playbook

$ curl -sSL https://raw.githubusercontent.com/gbraad/ansible-playbooks/master/playbooks/install-openstack-client.yml > /tmp/install.yml
$ ansible-playbook /tmp/install.yml

Managing users and projects

$ openstack project list

$ openstack user list

$ openstack role list

Add new user

$ openstack user create <username> --project <project> --password <password>

Grant role to user

$ openstack role add --user <username> --project <project> <role>

Managing SSH keys

Create a new key-pair

$ openstack keypair create mykey > mykey.pem

Upload existing key

$ openstack keypair create --public-key ~/.ssh/id_rsa.pub mykey

Managing images

$ openstack image list

Upload image

# openstack image create <name> --disk-format <dformat> --container-format <cformat> --file <file> --property os_distro=<||>

Managing instances

$ openstack flavor list

$ openstack server list

Create instance

$ openstack server create <name> --flavor <flavor> --image <image> --key-name <key-name>

Lifecycle

$ openstack server reboot <name>

$ openstack server delete <name>

Managing volumes

$ openstack volume create <volume-name> --size <size>

$ openstack volume create <volume-name> --image <image> --size <size>

$ openstack server add volume <server-name> <volume-name>

Managing networks

$ openstack network list

$ openstack network show <network>

Create network

$ openstack network create [--prefix prefix] [--enable | --disable] [--share | --no-share] <name>

Examples

wget -q http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 -O /tmp/centos7.qcow2
qemu-img convert /tmp/centos7.qcow2 /tmp/centos7.raw
#openstack image create --disk-format qcow2 --file /tmp/centos7.qcow2 centos7
openstack image create --disk-format raw --container-format bare --file /tmp/centos7.raw centos7
net_id=$(openstack network list -f value |awk '{print $1}')
openstack server create --flavor m1.small --image centos7 --nic net-id=${net_id} --key-name mykey test-instance

Dealing with values from the table output

for i in $( [command] | grep [filter] | awk ' { print $2 } '); do
  [command] $i
done

OpenStack Devstack

• Devstack configurations

Installation

On Ubuntu 14.04

$ sudo apt-get update
$ sudo apt-get install git
$ git clone https://git.openstack.org/openstack-dev/devstack

$ tools/create-stack-user.sh; su stack
$ vi local.conf

[[local|localrc]]
ADMIN_PASSWORD=secret
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD

$ ./stack.sh

Fuel

Development

• https://ide.c9.io/gbraad/openstack-fuel

Source

$ git clone https://github.com/openstack/fuel-main
$ git clone https://github.com/openstack/fuel-web
$ git clone https://github.com/openstack/fuel-ui
$ git clone https://github.com/openstack/fuel-agent
$ git clone https://github.com/openstack/fuel-astute
$ git clone https://github.com/openstack/fuel-ostf
$ git clone https://github.com/openstack/fuel-library
$ git clone https://github.com/openstack/fuel-docs

Make ISO

On Ubuntu 14.04

$ git clone https://github.com/openstack/fuel-main
$ cd fuel-main
$ ./prepare-build-env.sh
$ make iso

Source

Virtualized environment (for testing)

Preparation

Enable KVM nested virtualization

Packages

$ sudo apt-get install -y git postgresql postgresql-server-dev-all \
    libyaml-dev libffi-dev python-dev qemu-kvm libvirt-bin \
    libvirt-dev ubuntu-vm-builder bridge-utils \
    libpq-dev libgmp-dev \
    python-pip python-virtualenv

Database

$ sudo -u postgres createuser -P fuel_devops  # use "fuel_devops" as the password!
$ sudo -u postgres createdb fuel_devops -O fuel_devops

Create environment

$ git clone https://github.com/openstack/fuel-qa
$ cd fuel-qa
$ virtualenv .venv
$ source .venv/bin/activate
$ pip install -r ./fuelweb_test/requirements.txt

Setup the database

$ django-admin.py syncdb --settings=devops.settings
$ django-admin.py migrate devops --settings=devops.settings

To create the Fuel node use an ISO as described in the previous topic, or download from here

$ export NODES_COUNT=5
$ ./utils/jenkins/system_tests.sh -t test -w $(pwd) -j fuel_test -k -K \
    -i MirantisOpenStack-9.0.iso -V .venv -e "Deployment test" -o \
    --group=prepare_release

Create slaves

$ ./utils/jenkins/system_tests.sh -t test -w $(pwd) -j fuel_test -k -K \
    -i MirantisOpenStack-9.0.iso -V .venv -e "Deployment test" -o \
    --group=prepare_slaves_5

You should be able to login to the Fuel UI, at http://10.109.0.2 using admin/admin.

OpenStack Keystone

Development

• https://ide.c9.io/gbraad/openstack-keystone#openfile-README.md

Container

• https://gitlab.com/gbraad/openstack-keystone
• https://github.com/gbraad/docker-openstack-keystone
• https://hub.docker.com/r/gbraad/openstack-keystone

Keystone development environment

Requirements

Distribution packages

$ apt-get update
$ apt-get install -y mariadb-server mariadb-client
$ apt-get install -y python-pip python-virtualenv
$ apt-get install -y python-dev python3-dev libxml2-dev libxslt1-dev libsasl2-dev libsqlite3-dev libmysqlclient-dev libssl-dev libldap2-dev libffi-dev

Python packages

$ pip install -U pip
$ pip install -U pyopenssl ndg-httpsclient pyasn1
$ pip install repoze.lru pbr mysql-python
$ pip install python-keystoneclient python-openstackclient
$ pip install uwsgi

Setup database

$ service mysql start

$ mysql  -u root -ppass -e "create database keystone;"
$ mysql  -u root -ppass -e "GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' IDENTIFIED BY 'keystone';"
$ mysql  -u root -ppass -e "GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'keystone';"

Setup keystone for development

Getting the source

$ git clone https://github.com/openstack/keystone.git -b stable/mitaka

Choose: Setup development environment in virtualenv

$ tox -e venv --notest
$ source .tox/venv/bin/activate
$ pip install MySQL

or: Setup development outside virtualenv

$ pip install -r requirements.txt
$ pip install -r test-requirements.txt
$ python setup.py develop

Verify installation

$ python -c "import keystone"

Configuration

$ export TOKEN=$(openssl rand -hex 10)
$ cp etc/keystone.conf.sample etc/keystone.conf
$ sed -i "s|database]|database]\nconnection = mysql://keystone:keystone@localhost/keystone|g" etc/keystone.conf
$ sed -i 's/#admin_token = ADMIN/admin_token = $TOKEN/g' etc/keystone.conf

$ keystone-manage db_sync

Run Keystone

$ sudo -u keystone /usr/local/bin/keystone-all --config-file=/etc/keystone/keystone.conf  --log-file=/var/log/keystone/keystone.log

Create service entry

$ keystone --os-token $TOKEN --os-endpoint http://localhost:35357/v2.0/ service-create --name=keystone --type=identity --description="Keystone Identity Service"
$ keystone --os-token $TOKEN --os-endpoint http://localhost:35357/v2.0/ endpoint-create --service keystone --publicurl 'http://localhost:5000/v2.0' --adminurl 'http://localhost:35357/v2.0' --internalurl 'http://localhost:5000/v2.0'

Create admin user

$ keystone --os-token $TOKEN --os-endpoint http://localhost:35357/v2.0/ user-create --name admin --pass password
$ keystone --os-token $TOKEN --os-endpoint http://localhost:35357/v2.0/ role-create --name admin
$ keystone --os-token $TOKEN --os-endpoint http://localhost:35357/v2.0/ tenant-create --name admin
$ keystone --os-token $TOKEN --os-endpoint http://localhost:35357/v2.0/ user-role-add --user admin --role admin --tenant admin

Setup keystone for use

Same as above until ‘Getting the source’

Create directories

$ useradd --home-dir "/var/lib/keystone" --create-home --system --shell /bin/false keystone
$ mkdir -p /var/log/keystone
$ mkdir -p /etc/keystone
$ chown -R keystone:keystone /var/log/keystone
$ chown -R keystone:keystone /var/lib/keystone
$ chown keystone:keystone /etc/keystone

Configuration

$ cd keystone

$ cp -R etc/* /etc/keystone/
$ mv  /etc/keystone/keystone.conf.sample /etc/keystone/keystone.conf
$ sed -i "s|database]|database]\nconnection = mysql://keystone:keystone@localhost/keystone|g" /etc/keystone/keystone.conf
$ sed -i 's/#admin_token = ADMIN/admin_token = SuperSecreteKeystoneToken/g' /etc/keystone/keystone.conf

Install

$ python setup.py install
$ keystone-manage db_sync

$ cat >> /etc/logrotate.d/keystone << EOF
/var/log/keystone/*.log {
    daily
    missingok
    rotate 7
    compress
    notifempty
    nocreate
}
EOF

Run Keystone

$ sudo -u keystone /usr/local/bin/keystone-all --config-file=/etc/keystone/keystone.conf  --log-file=/var/log/keystone/keystone.log

Verify Keystone using cURL

• http://docs.openstack.org/developer/keystone/api_curl_examples.html

export PRETTYJSON="python -mjson.tool" # or use "jq ."
$ curl -s -d '{"auth":{"passwordCredentials":{"username": "admin", "password": "password"},"tenantName": "admin"}}' -H "Content-Type: application/json" http://localhost:5000/v2.0/tokens | $PRETTYJSON

{
  "access": {
    "metadata": {
      "roles": [
        "1180e62cfaff4d288f26730d0c613b78"
      ],
      "is_admin": 0
    },
    "user": {
      "name": "admin",
      "roles": [
        {
          "name": "admin"
        }
      ],
      "id": "497e17c64d6b4d10a07dc3d61e75170b",
      "roles_links": [],
      "username": "admin"
    },
    "serviceCatalog": [
      {
        "name": "keystone",
        "type": "identity",
        "endpoints_links": [],
        "endpoints": [
          {
            "publicURL": "http://localhost:5000/v2.0",
            "id": "184cfc47a8f84e4e9a49ef621f67a18e",
            "internalURL": "http://localhost:5000/v2.0",
            "region": "regionOne",
            "adminURL": "http://localhost:35357/v2.0"
          }
        ]
      }
    ],
    "token": {
      "audit_ids": [
        "6EKt-yI7Rmu1glQKA9GhpA"
      ],
      "tenant": {
        "name": "admin",
        "id": "fd45db445d1a47e38d41d38c12cdbf43",
        "enabled": true,
        "description": null
      },
      "id": "627a40ea7cbb4a93bc3f0e9d2df607b0",
      "expires": "2016-09-19T10:14:40Z",
      "issued_at": "2016-09-19T09:14:40.345707Z"
    }
  }
}

Get token using cURL

To continue this:

$ curl -i -H "Content-Type: application/json" -d '
{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "admin",
          "domain": { "name": "Default" },
          "password": "password"
        }
      }
    },
    "scope": {
      "project": {
        "name": "admin",
        "domain": { "name": "Default" }
      }
    }
  }
}' http://172.17.0.6:5000/v3/auth/tokens

A response will follow. You need to set the value of X-Subject-Token to OS_TOKEN.

$ OS_TOKEN=8d4079e745064b578470d87c816119e2

To list users:

$ curl -s -H "X-Auth-Token: $OS_TOKEN" http://localhost:5000/v3/users | PRETTYJSON

Use python-openstackclient

export OS_TOKEN=secrete
$ openstack --os-token $OS_TOKEN --os-url http://localhost:35357/v2.0/ user list

Puppet modules

All-in-one

$ git clone git://git.openstack.org/openstack/puppet-openstack-integration
$ cd puppet-openstack-integration
$ ./all-in-one.sh

or

$ curl -sL http://git.openstack.org/cgit/openstack/puppet-openstack-integration/plain/all-in-one.sh | bash

Note: scenario description

RDO

PackStack scenario001 test using WeIRDO

#!/bin/bash
# How to run WeIRDO on localhost from a fresh CentOS installation
yum -y install git epel-release python-devel "@Development Tools"
yum -y install python-pip
pip install tox
git clone https://github.com/redhat-openstack/weirdo.git
cd weirdo
cat <<EOF >hosts
localhost ansible_connection=local

[openstack_nodes]
localhost log_destination=/var/log/weirdo
EOF
tox -e ansible-playbook -- -vv -i hosts playbooks/packstack-scenario001.yml --extra-vars openstack_release=mitaka

WeIRDO localhost

Repositories

• liberty
• mitaka

OpenStack PackStack

• PackStack Answerfiles

Installation

Step by step

/etc/environment

LANG=en_US.utf-8
LC_ALL=en_US.utf-8

$ sudo yum install -y centos-release-openstack-mitaka
$ sudo yum update -y
$ sudo yum install -y openstack-packstack
$ packstack --allinone

Note: consider disabling NetworkManager

For more info see: Quickstart Packstack

Oneliners

• OpenStack PackStack all-in-one
  curl -sSL https://raw.githubusercontent.com/gbraad/oneliners/master/install_packstack.sh | bash
• WeIRDO Packstack scenario
  curl -sSL https://raw.githubusercontent.com/gbraad/oneliners/master/run_weirdo_packstack_scenario001.sh | bash

Multinode configuration

Generate the default answerfile on the controller node

$ packstack --gen-answer-file=~/multinode.cfg

Update the following in the answerfile

CONFIG_CONTROLLER_HOSTS=192.168.1.205
CONFIG_COMPUTE_HOSTS=192.168.1.206
CONFIG_NETWORK_HOSTS=192.168.1.207
CONFIG_STORAGE_HOST=192.168.1.205

Install OpenStack using the answerfile

$ packstack --answer-file=~/multinode.cfg

In case you do not want to run the installation on already configured servers, add the following to the answerfile:

EXCLUDE_SERVERS=<serverIP>,<serverIP>,...

Options to consider

CONFIG_HORIZON_SSL=y
CONFIG_PROVISION_DEMO=n

Linux

Notes on Linux distributions, hardware platforms, and system tools.

• Linux - general Linux tips and commands
• ARM - ARM architecture, toolchains, bare-metal, Logue SDK
• Raspberry Pi - Raspberry Pi setup and usage
• RISC-V - RISC-V architecture, toolchains, emulator workbench
• Photon OS - VMware Photon OS
• ostree - OSTree image-based updates
• Flatpak - application sandboxing and distribution
• Libvirt/virsh - virtual machine management
• Software installation - package managers and install methods
• Storage - disk, LVM, and filesystem management
• Cockpit - web-based Linux administration
• Android - Android development and ADB
• GlusterFS - distributed filesystem
• WSL - Windows Subsystem for Linux, mounts, metadata, wsl.conf

Linux

Kernel

Workspace

• Cloud9

Build wrappers (at GitLab)

• mainline, kernel-mainline.repo
• stable, kernel-stable.repo
• centos

Distributions

• Fedora
• Ubuntu
• OpenSUSE
• Debian
• …

Swapfile

dd if=/dev/zero of=/.swapfile bs=1M count=1024
chmod 0600 /.swapfile
mkswap -f /.swapfile
sudo swapon /.swapfile

/etc/fstab

/.swapfile	swap	swap	defaults	0	0

ARM

ARM is a family of Reduced Instruction Set Computer (RISC) architectures developed and licensed by Arm Holdings. Rather than manufacturing chips, Arm licenses its ISA and core designs to partners (Apple, Qualcomm, Samsung, etc.) who produce the actual silicon. This licensing model has made ARM the dominant architecture for mobile, embedded, and increasingly server and desktop workloads.

Architecture versions

Profiles:

• A (Application) - high-performance, runs full OS (Linux, Android, macOS)
• R (Real-time) - deterministic latency, automotive, industrial
• M (Microcontroller) - low power, bare-metal embedded (Arduino, STM32)

ABI and naming

The naming of ARM ABIs can be confusing:

Most modern Linux distributions target aarch64. Use armhf only for older 32-bit boards.

Registers (AArch64)

Key instructions (AArch64)

// Data movement
mov x0, #42          // x0 = 42
mov x1, x0           // x1 = x0
ldr x0, [x1]         // x0 = memory[x1]
str x0, [x1]         // memory[x1] = x0
ldr x0, [x1, #8]     // x0 = memory[x1 + 8]

// Arithmetic
add x0, x1, x2       // x0 = x1 + x2
sub x0, x1, x2       // x0 = x1 - x2
mul x0, x1, x2       // x0 = x1 * x2
lsl x0, x1, #2       // x0 = x1 << 2 (multiply by 4)
lsr x0, x1, #1       // x0 = x1 >> 1 (unsigned divide by 2)

// Logical
and x0, x1, x2
orr x0, x1, x2
eor x0, x1, x2       // XOR
bic x0, x1, x2       // AND NOT

// Branches
b   label             // unconditional branch
bl  label             // branch with link (call)
ret                   // return (branch to x30)
cbz x0, label         // branch if x0 == 0
cbnz x0, label        // branch if x0 != 0
b.eq / b.ne / b.lt / b.gt  // conditional branches after cmp

// Compare
cmp x0, x1           // sets flags, discards result

Hello world (AArch64 Linux)

.section .data
msg:    .ascii "Hello, ARM!\n"
len = . - msg

.section .text
.global _start
_start:
    mov x8, #64        // syscall: write
    mov x0, #1         // fd: stdout
    adr x1, msg        // buffer
    mov x2, #len       // length
    svc #0

    mov x8, #93        // syscall: exit
    mov x0, #0         // status
    svc #0

as -o hello.o hello.s
ld -o hello hello.o
./hello

Toolchain

There are two distinct toolchain families for ARM. Choosing the wrong one is a common source of confusion:

Linux cross-compilation (from x86_64)

# Install cross toolchains
sudo apt install gcc-aarch64-linux-gnu binutils-aarch64-linux-gnu   # 64-bit
sudo apt install gcc-arm-linux-gnueabihf binutils-arm-linux-gnueabihf  # 32-bit

# Cross-compile a C program
aarch64-linux-gnu-gcc -o hello hello.c
arm-linux-gnueabihf-gcc -o hello hello.c

# Cross-compile with Rust
rustup target add aarch64-unknown-linux-gnu
cargo build --target aarch64-unknown-linux-gnu

Bare-metal (arm-none-eabi)

The arm-none-eabi toolchain targets ARM with no OS and no standard C library (or a minimal one like newlib). This is the standard for Cortex-M microcontrollers, custom firmware, and synthesizer plugin development.

sudo apt install gcc-arm-none-eabi binutils-arm-none-eabi

Typical compiler flags depend on the exact core:

# Cortex-M4 with FPU (e.g. STM32F4, KORG logue hardware)
arm-none-eabi-gcc \
  -mcpu=cortex-m4 \
  -mthumb \
  -mfpu=fpv4-sp-d16 \
  -mfloat-abi=hard \
  -Os -fno-exceptions \
  -o firmware.elf main.c

# Cortex-M0+ (no FPU, smaller core)
arm-none-eabi-gcc \
  -mcpu=cortex-m0plus \
  -mthumb \
  -mfloat-abi=soft \
  -Os \
  -o firmware.elf main.c

Key differences from a Linux cross-compile:

• No libc by default - printf, malloc etc. require linking [[development/newlib|newlib]] and implementing syscall stubs
• No OS syscalls - everything goes through hardware registers or a vendor HAL
• Needs a linker script to place .text, .data, .bss at correct flash/RAM addresses
• Needs a startup file (startup.s or startup.c) to set up the stack and call main

Minimal bare-metal linker script:

MEMORY {
    FLASH (rx)  : ORIGIN = 0x08000000, LENGTH = 256K
    SRAM  (rwx) : ORIGIN = 0x20000000, LENGTH = 64K
}

SECTIONS {
    .text : { *(.text*) *(.rodata*) } > FLASH
    .data : { *(.data*) } > SRAM AT > FLASH
    .bss  : { *(.bss*) *(COMMON) } > SRAM
}

QEMU user-mode (run ARM binaries on x86)

sudo apt install qemu-user-static binfmt-support
qemu-aarch64-static ./hello

QEMU system emulation

qemu-system-aarch64 \
  -M virt \
  -cpu cortex-a57 \
  -m 2G \
  -kernel Image \
  -append "console=ttyAMA0 root=/dev/vda" \
  -drive if=virtio,file=rootfs.img \
  -nographic

Linux distributions

Common hardware

Containers

Multi-arch images are the standard approach - no need to distinguish ARM images manually if the registry has a manifest list.

# Pull natively on ARM host (automatic)
docker pull ubuntu:24.04

# Build multi-arch image from x86 host
docker buildx create --use
docker buildx build --platform linux/amd64,linux/arm64 -t myimage:latest --push .

# Inspect available platforms
docker buildx imagetools inspect ubuntu:24.04

Run x86 containers on ARM with emulation (slower):

docker run --platform linux/amd64 ubuntu:24.04 uname -m

Logue SDK (synthesizer unit development)

KORG’s logue-sdk allows developing custom oscillators and effects for logue-series synthesizers. The hardware is ARM Cortex-M4F based; units are compiled with arm-none-eabi-gcc and uploaded via KORG Librarian or the NTS-1 mk2 web app.

Supported platforms

Unit types

Toolchain setup

# Install arm-none-eabi toolchain
sudo apt install gcc-arm-none-eabi binutils-arm-none-eabi

# Clone the SDK
git clone https://github.com/korginc/logue-sdk.git
cd logue-sdk

# Initialise submodules and toolchain
git submodule update --init --recursive

Build flags (from the SDK Makefile)

MCU     = cortex-m4
MCFLAGS = -mcpu=$(MCU) -mthumb -mfpu=fpv4-sp-d16 -mfloat-abi=hard
CFLAGS  = $(MCFLAGS) -Os -fno-exceptions -fno-non-call-exceptions \
          -ffast-math -fsingle-precision-constant

Unit structure

A minimal oscillator unit exposes three functions the firmware calls into:

#include "userosc.h"

void OSC_INIT(uint32_t platform, uint32_t api) {
    // Called once at load time
}

void OSC_CYCLE(const user_osc_param_t *const params,
               int32_t *yn, const uint32_t frames) {
    // Called every audio block (frames = 64 samples typically)
    // Fill yn[] with Q31 audio samples
}

void OSC_NOTEON(const user_osc_param_t *const params) { }
void OSC_NOTEOFF(const user_osc_param_t *const params) { }

void OSC_PARAM(uint16_t index, uint16_t value) {
    // Handle parameter knob changes (value 0–1023)
}

Building and deploying

# Build a unit (from its directory)
cd logue-sdk/platform/minilogue-xd/osc/waves
make

# Output: build/waves.mnlgxdunit
# Upload via KORG Librarian (desktop) or drag-and-drop (NTS-1 mk2)

Move / Schwung effects

Move and Schwung are custom logue units. See [[development/audio/schwung-move-effects]] for implementation notes.

NEON (SIMD)

Not yet documented. NEON is ARM’s SIMD extension for Cortex-A (not Cortex-M). Used for DSP, ML inference, and multimedia workloads on application processors.

Resources

• Arm Architecture Reference Manual (Arm ARM)
• Arm Learning Paths
• Arm Developer
• Raspberry Pi documentation
• KORG logue-sdk
• logue-sdk unit development guide

RaspberryPi

… [wip] …

RISC-V

RISC-V (pronounced “risk five”) is an open, free ISA (Instruction Set Architecture) developed at UC Berkeley starting in 2010. Unlike ARM or x86, nobody owns RISC-V - it is governed by RISC-V International, a non-profit with hundreds of member companies. Anyone can implement it without paying royalties or signing NDAs.

This openness has made RISC-V attractive for research, custom silicon, embedded systems, and increasingly for general-purpose computing.

ISA structure

RISC-V is modular: a small mandatory base plus optional standard extensions.

Base ISAs

Standard extensions

G = IMAFD + Zicsr + Zifencei - the general-purpose combination used for Linux systems.

A target is described by combining letters: rv64gc means 64-bit + G + Compressed.

Registers

RISC-V has 32 integer registers (+ 32 float registers with F/D extension):

Key instructions (RV64I)

# Data movement
li   a0, 42          # load immediate (pseudo, expands to lui+addi)
mv   a0, a1          # move register (pseudo: addi a0, a1, 0)
la   a0, label       # load address (pseudo)
ld   a0, 0(sp)       # load doubleword from memory
sd   a0, 0(sp)       # store doubleword to memory
lw   a0, 0(sp)       # load word (32-bit, sign-extended)
sw   a0, 0(sp)       # store word

# Arithmetic
add  a0, a1, a2      # a0 = a1 + a2
addi a0, a1, 10      # a0 = a1 + 10 (immediate)
sub  a0, a1, a2      # a0 = a1 - a2
mul  a0, a1, a2      # a0 = a1 * a2 (M extension)
div  a0, a1, a2      # a0 = a1 / a2 (M extension)

# Logical
and  a0, a1, a2
or   a0, a1, a2
xor  a0, a1, a2
sll  a0, a1, a2      # shift left logical
srl  a0, a1, a2      # shift right logical
sra  a0, a1, a2      # shift right arithmetic

# Branches (compare-and-branch, no flags register)
beq  a0, a1, label   # branch if a0 == a1
bne  a0, a1, label   # branch if a0 != a1
blt  a0, a1, label   # branch if a0 < a1 (signed)
bltu a0, a1, label   # branch if a0 < a1 (unsigned)
bge  a0, a1, label   # branch if a0 >= a1 (signed)
bgeu a0, a1, label   # branch if a0 >= a1 (unsigned)

# Jumps
jal  ra, label       # jump and link (call)
jalr ra, 0(ra)       # jump and link register (used for ret)
ret                  # return (pseudo: jalr zero, 0(ra))

Hello world (RV64 Linux)

.section .data
msg:    .string "Hello, RISC-V!\n"
len = . - msg

.section .text
.global _start
_start:
    li   a7, 64         # syscall: write
    li   a0, 1          # fd: stdout
    la   a1, msg        # buffer address
    li   a2, len        # length
    ecall

    li   a7, 93         # syscall: exit
    li   a0, 0          # status
    ecall

riscv64-linux-gnu-as -o hello.o hello.s
riscv64-linux-gnu-ld -o hello hello.o
./hello   # on RISC-V hardware or QEMU

Key differences from ARM/x86

Notable: RISC-V has no flags register. All branches explicitly compare two registers, making pipelines simpler.

Toolchain

As with ARM, there are two toolchain families:

Linux cross-compilation (RV64)

# Debian/Ubuntu
sudo apt install gcc-riscv64-linux-gnu binutils-riscv64-linux-gnu

# Cross-compile C
riscv64-linux-gnu-gcc -march=rv64gc -mabi=lp64d -o hello hello.c

# Rust
rustup target add riscv64gc-unknown-linux-gnu
cargo build --target riscv64gc-unknown-linux-gnu

ABI naming

lp64d is standard for Linux on capable hardware.

Bare-metal (riscv32-unknown-elf)

For bare-metal RV32 targets (no OS, no libc) use the ELF toolchain:

sudo apt install gcc-riscv64-linux-gnu   # includes riscv32 multilib on some distros
# Or build from source / use a pre-built release:
# https://github.com/riscv-collab/riscv-gnu-toolchain

# Compile bare-metal RV32IM
riscv32-unknown-elf-gcc \
  -march=rv32im \
  -mabi=ilp32 \
  -nostdlib \
  -T linker.ld \
  -o program.elf startup.s main.c

# Extract raw binary (no ELF headers)
riscv32-unknown-elf-objcopy -O binary program.elf program.bin

Key flags:

• -march=rv32im - RV32 base + M extension (no float, no C)
• -mabi=ilp32 - 32-bit integers, software float, 32-bit pointers
• -nostdlib - no libc, no startup files; drop this and add --specs=nano.specs to use [[development/newlib|newlib]]
• -T linker.ld - provide your own memory layout

QEMU emulation

sudo apt install qemu-system-riscv64 qemu-user-static

# User-mode (run a single binary)
qemu-riscv64-static ./hello

# System emulation with OpenSBI + U-Boot
qemu-system-riscv64 \
  -M virt \
  -m 2G \
  -smp 4 \
  -bios /usr/lib/riscv64-linux-gnu/opensbi/generic/fw_jump.bin \
  -kernel u-boot.bin \
  -drive file=rootfs.img,format=raw,id=hd0 \
  -device virtio-blk-device,drive=hd0 \
  -nographic

Linux distributions

Hardware

Development boards

Microcontrollers

Privileged architecture

RISC-V defines three privilege levels:

OpenSBI (Open Source Supervisor Binary Interface) is the standard M-mode firmware, equivalent to UEFI/BIOS on x86 or ATF on ARM.

Boot sequence: OpenSBI → U-Boot → Linux kernel

emu - RV32IM bare-metal workbench

~/Projects/riscv/emu/ is a self-contained RV32IM emulator and workbench. It is not a QEMU wrapper - it is a full software implementation of a RISC-V CPU plus a set of memory-mapped peripherals, with both a native CLI and a WebAssembly build for in-browser use.

What it implements

• ISA: RV32IM - 32-bit integer base + M extension (multiply/divide)
• Privilege: Machine mode (M-mode) only - no OS, no MMU
• Memory: Configurable 64 KB – 16 MB, little-endian
• Peripherals (memory-mapped):

Memory layout

0x00000000  RAM (default 256 KB, configurable)
0x10000000  GPIO
0x10001000  UART
0x10003000  I2C / LCD
0x10005000  Interrupt controller

Build

cd ~/Projects/riscv/emu

# Native CLI
mkdir build && cd build
cmake .. && cmake --build .
# Produces: build/riscv-emu  build/riscv-as

# WebAssembly (requires Emscripten)
./build-wasm.sh
# Produces: web/emu.js  web/emu.wasm  web/assembler.js

Running programs

# Assemble with the included Python assembler
./tools/riscv-asm.py examples/fibonacci.s examples/fibonacci.bin

# Run the binary (max 100 instructions)
./build/riscv-emu examples/fibonacci.bin 100

# Run with more memory (1 MB)
./build/riscv-emu examples/fibonacci.bin 0 1024

Output shows final register state and the LCD display buffer if used.

Bare-metal program structure

Every program needs a startup file and a linker script:

start.s - sets up the stack and clears BSS:

.section .text._start
.global _start

_start:
    la   sp, _stack_top     # load stack pointer from linker symbol

    # clear BSS
    la   t0, _bss_start
    la   t1, _bss_end
1:  beq  t0, t1, 2f
    sw   zero, 0(t0)
    addi t0, t0, 4
    j    1b
2:
    call main
    ecall                   # signal emulator to stop

linker.ld - memory layout for the emulator:

MEMORY {
    RAM (rwx) : ORIGIN = 0x00000000, LENGTH = 256K
}

SECTIONS {
    . = 0x00000000;
    .text   : { KEEP(*(.text._start)) *(.text*) } > RAM
    .rodata : { *(.rodata*) }                     > RAM
    .data   : { *(.data*) }                       > RAM
    .bss    : {
        _bss_start = .;
        *(.bss*) *(COMMON)
        _bss_end = .;
    } > RAM
    .stack (NOLOAD) : {
        . = ALIGN(16);
        _stack_top = . + 16K;
    } > RAM
}

Build and run a C program:

riscv32-unknown-elf-gcc \
  -march=rv32im -mabi=ilp32 -nostdlib \
  -T linker.ld \
  -o program.elf start.s main.c

riscv32-unknown-elf-objcopy -O binary program.elf program.bin
./build/riscv-emu program.bin

UART output

#define UART_BASE  0x10001000
#define UART_CTRL  (*(volatile uint32_t*)(UART_BASE + 0x08))
#define UART_DATA  (*(volatile uint32_t*)(UART_BASE + 0x00))

void putchar(char c) {
    UART_CTRL = 1;   // enable
    UART_DATA = c;
}

Interrupt handling

# Install trap vector
la   t0, trap_handler
csrw mtvec, t0          # set machine trap vector

# Enable machine external interrupt + global interrupts
li   t0, 0x800          # MIE.MEIE
csrw mie, t0
li   t0, 0x8            # MSTATUS.MIE
csrw mstatus, t0

trap_handler:
    csrr t0, mcause     # read cause
    # handle ...
    mret                # return from trap (restores MSTATUS.MIE)

WebAssembly / web workbench

The emulator compiles to WASM (~18 KB) and is callable from JavaScript:

createRISCVEmu().then(Module => {
    const emu = Module._emu_create(65536);       // 64 KB memory
    Module._emu_load_program(emu, ptr, size);    // load .bin
    Module._emu_run(emu, 1000);                  // run ≤1000 instructions
    const a0 = Module._emu_get_register(emu, 10);
    const lcd = Module.UTF8ToString(Module._emu_get_lcd_line1(emu));
    Module._emu_destroy(emu);
});

Open web/index.html locally to use the full workbench (editor + assembler + emulator + LCD display).

Resources

• RISC-V Specifications
• RISC-V Unprivileged ISA
• RISC-V International
• riscv-software-list
• SiFive docs
• riscv/emu workbench

Photon OS

• GitLab buildwrapper
• Unofficial mirror on GitLab, my fork

Install on OpenStack

$ wget https://bintray.com/artifact/download/vmware/photon/photon-ami-1.0-13c08b6.tar.gz
$ tar -zxvf photon-ami-1.0-13c08b6.tar.gz
$ openstack image create photon --file photon-ami-1.0-13c08b6.raw --container-format bare --disk-format raw
$ openstack server create photon --flavor m1.tiny --image photon --key-name c9
$ ssh root@[ipaddress]
root@photon [ ~ ]#

Disable DHCP

root [ ~ ]# mv /etc/systemd/network/10-dhcp-eth0.network  /etc/systemd/network/10-static-eth0.network
root [ ~ ]# vi /etc/systemd/network/10-static-eth0.network
root [ ~ ]# cat /etc/systemd/network/10-static-eth0.network
[Match]
Name=eth0

[Network]
Address=192.168.79.134/24
Gateway=192.168.79.2
DNS=8.8.8.8
Domains=photon.local
root [ ~ ]# systemctl restart systemd-networkd.service

Start Docker Container

$ docker run -d -p 80:80 vmwarecna/nginx

• -d - Run the container in the background
• -p 80:80 - Publish the container’s port to the host

Display the public-facing port that is NAT-ed to the container

$ docker port 5f6b0e03c6de

Automatically start Docker containers at boot time

$ docker run --restart=always -d -p 80:80 vmwarecna/nginx

ostree

• See also Atomic
• See also Flatpak

OSTree tests

• OSTree tests
• OSTree tests wrapper; GitLab, GitHub

Links

• ostree documentation
• Build Docker container from OSTree

Flatpak

• Project Flatpak
• Getting Started
• Examples

Installation

Runtime

$ dnf install -y flatpak
$ wget https://sdk.gnome.org/keys/gnome-sdk.gpg -o /tmp/gnome-sdk.gpg
$ flatpak remote-add --gpg-import=/tmp/gnome-sdk.gpg gnome https://sdk.gnome.org/repo/
$ flatpak remote-add --gpg-import=/tmp/gnome-sdk.gpg gnome-apps https://sdk.gnome.org/repo-apps/
$ flatpak install gnome org.gnome.Platform 3.20

Application

$ flatpak install gnome-apps org.gnome.gedit stable
$ flatpak run org.gnome.gedit

SDK

$ flatpak install gnome org.gnome.Sdk 3.20

Builders

• GNOME Runtime and SDK: GitLab, GitHub
  docker run -it --rm -v $PWD:/workspace registry.gitlab.com/gbraad/flatpak-builder-gnome bash
• freedesktop Runtime and SDK: GitLab, GitHub
  docker run -it --rm -v $PWD:/workspace registry.gitlab.com/gbraad/flatpak-builder-freedesktop bash

Runtimes

• Official runtimes
• Automated runtime build; GitLab
• CentOS: https://github.com/matthiasclasen/flatpak-runtime, build
• openSUSE: https://github.com/fcrozat/opensuse-flatpak-runtime
• Mono: (WIP!): https://gitlab.com/gbraad/flatpak-mono-runtime, https://github.com/gbraad/flatpak-mono-runtime
• Fedora (WIP!): https://gitlab.com/gbraad/flatpak-fedora-runtime, https://github.com/gbraad/flatpak-fedora-runtime

Also see Examples at the top of the page

Applications

• Spotify build wrapper: GitLab
• Hello app: GitLab
• OpenStack (WIP!) https://github.com/gbraad/flatpak-openstack-client

Test

• Test container: GitHub, GitLab

Libvirt/virsh

Create snapshot

$ virsh snapshot-create-as [vmname] snapshot1 "Before installation" --disk-only --atomic

Start and stop

Start

$ virsh start [vmname]

Stop (forcibly)

$ virsh destroy [vmname]

ACPI events

• Reboot

$ virsh reboot [vmname]

$ sudo su - stack -c "virsh reboot undercloud"

• Shutdown

$ virsh shutdown [vmname]

Configure libvirt pool

Create a libvirt persistent pool and start it:

$ virsh pool-define-as --type=dir --name=default --target=/var/lib/libvirt/images
$ virsh pool-autostart default
$ virsh pool-start default

/var/lib/libvirt/images is where QEMU QCOW images will be stored, so make sure this directory is attached to a file system with sufficient storage.

Install software

Install PackStack (bash)

curl -sSL https://raw.githubusercontent.com/gbraad/oneliners/master/install_packstack.sh | bash

Install C9 (bash)

curl -sSL https://raw.githubusercontent.com/gbraad/oneliners/master/install_c9.sh | bash

Install C9 (ansible)

curl -sSL https://github.com/gbraad/ansible-playbooks/raw/master/playbooks/install-c9sdk.yml -o install-c9sdk.yml
ansible-playbook install-c9sdk.yml

Storage

• Ceph
• GlusterFS
• xfs
• zfs

Cockpit

• Cockpit
• Homepage

Install

On Fedora 24

$ dnf update
$ dnf install docker
$ systemctl enable docker
$ systemctl start docker
$ dnf install cockpit cockpit-docker
$ systemctl enable cockpit.socket
$ systemctl start cockpit
$ firewall-cmd --add-service=cockpit
$ setenforce 0  # disable selinux :-(

Open http://[hostname/ip]:9090

Android

• GitLab

Install SDK APIs

$ android list sdk
$ android update sdk --no-ui --force --filter 1,2,3,4,5,6,7,8,9,10

GlusterFS

Use Gluster client on Atomic

Source

Installation

$ dnf install -y glusterfs-server xfsprogs

Read blog post about setting HA using GlusterFS

WSL - Windows Subsystem for Linux

WSL2 runs a real Linux kernel in a lightweight VM. Windows drives are automounted under /mnt/ (e.g. D: -> /mnt/d) but without Linux metadata support by default, which means file permissions and ownership are not preserved on Windows filesystem paths.

Reference: Microsoft WSL configuration docs

Metadata: fixing file permissions on Windows mounts

Without metadata, chmod, chown, and sed -i (which uses a temp file) behave unexpectedly or fail with permission errors.

Remount on the fly

sudo umount /mnt/d
sudo mount -t drvfs D: /mnt/d -o metadata

Make it permanent

Add to /etc/wsl.conf:

[automount]
options = "metadata"

Note: The official Microsoft docs use options = "metadata,..." — spaces around = and quotes around the value. The quotes are part of the expected format when passing a comma-separated list of sub-options.

Then restart WSL from PowerShell or CMD:

wsl --shutdown

wsl.conf reference

Located at /etc/wsl.conf inside the distro. Changes take effect after wsl --shutdown.

[automount]

DrvFs mount sub-options for options=:

[automount]
enabled=true
root=/mnt/
options = "metadata,umask=22,fmask=11"
mountFsTab=true

[boot]

[boot]
systemd=true

[network]

[interop]

[user]

[gpu] / [time]

Full example

[user]
default=gbraad

[boot]
systemd=true

[network]
generateResolvConf=false

[automount]
options = "metadata"

.wslconfig reference

Located at %UserProfile%\.wslconfig on the Windows side. Controls the WSL2 VM globally across all distros.

[wsl2]

[wsl2]
memory=8GB
processors=4
swap=2GB
networkingMode=mirrored

Path values require escaped backslashes: C:\\Temp\\kernel

Common WSL commands

wsl --shutdown                          # stop all running distros
wsl --list --verbose                    # list distros and status
wsl --set-default Ubuntu                # set default distro
wsl --set-version Ubuntu 2             # upgrade distro to WSL2
wsl --export Ubuntu ubuntu.tar         # back up distro
wsl --import Ubuntu C:\WSL ubuntu.tar  # restore distro

Development

Notes on programming languages, tools, and development workflows.

• Git - version control, history rewriting, GitHub, GitLab
• Python - syntax, Django
• Rust - ownership, traits, error handling
• Go - Go language basics
• newlib - C library for bare-metal targets
• Ansible - infrastructure automation
• C9 IDE - Cloud9 development environment
• sed - stream editor
• Shell scripting - bash scripting
• Audio - Pure Data, Csound, Faust, Logue SDK, Schwung

Git

• GitLab
• GitHub

Delete commits from a branch

Careful: git reset –hard WILL DELETE YOUR WORKING DIRECTORY CHANGES

  git reset --hard HEAD~1

Set vim as default editor

Needed on Ubuntu and C9 containers

git config --global core.editor "vim"

export GIT_EDITOR=vim

Update all subdirectories

Used for a project directory with no git submodule.

git-pullall

#!/bin/bash
for i in $(find . -maxdepth 1 -type d)
do
    pushd $i
    git checkout master
    git pull --reb
    popd
done

Git submodules

git submodule add [repo] [location/name]

git submodule init

git submodule update

Initial commit

git newclone name [gitlab|github]

#!/bin/sh
git clone git@${2:-"gitlab"}.com:gbraad/${1}.git
cd ${1}
touch README.md
git add README.md
git commit -m "Initial commit"
git push -u origin master

git newrepo name [gitlab|github]

#!/bin/sh
mkdir -p ${1}
cd ${1}
git init
git remote add origin git@${2:-"gitlab"}.com:gbraad/${1}.git
touch README.md
git add README.md
git commit -m "Initial commit"
git push -u origin master

GitHub

• Homepage

Pull PR

$ git config --add remote.origin.fetch '+refs/pull//head:refs/remotes/origin/pr/'

$ git config --global --add remote.origin.fetch "+refs/pull/*/head:refs/remotes/origin/pr/*"

GitLab

• Homepage

Deployment

• Ansible playbook
• Docker images

On OpenShift

• OpenShift commons article
• Gitlab blog
• OpenShift blog
• OpenShift template

Access API using pyapi-gitlab

• GitHub

Usage

import gitlab
import requests.packages.urllib3
requests.packages.urllib3.disable_warnings()
lab = gitlab.Gitlab(host="gitlab.com",token="fdsf")
project = lab.getproject("gbraad/gauth")
issues = lab.getprojectissues(project["id"])

table = PrettyTable(["Title", "Labels", "State"])
table.align["Title"] = "l"

issues = lab.getprojectissues(project["id"])
for issue in issues:
    title = issue['title']
    labels = ",".join(issue['labels'])
    state = issue['state']
    table.add_row([title, labels, state])

print(table)

Rewriting Git history

Git history is immutable by design, but there are legitimate reasons to rewrite it: removing accidentally committed secrets, purging large binary files, or eliminating a directory that was never meant to be tracked.

Any history rewrite changes commit SHAs. Everyone who has cloned the repo must re-clone or hard-reset after a force-push.

git filter-repo (recommended)

git filter-repo is the modern replacement for git filter-branch. It is faster, safer, and ships as a single Python script.

pip install git-filter-repo

Remove a folder from all history

git filter-repo --path some/folder --invert-paths

--invert-paths means “keep everything except this path”. Run from the repo root.

Remove a single file from all history

git filter-repo --path secrets.env --invert-paths

Remove by glob pattern

git filter-repo --path-glob '*.zip' --invert-paths

Recommended workflow (fresh clone)

git filter-repo refuses to run on a repo that still has its original remote configured, unless you pass --force. Working on a mirror clone avoids that:

git clone --mirror git@github.com:you/repo.git repo-mirror
cd repo-mirror
git filter-repo --path some/folder --invert-paths
git push --force

After pushing, all collaborators must re-clone:

git clone git@github.com:you/repo.git

Or reset an existing clone:

git fetch origin
git reset --hard origin/main

Stop tracking a folder (without rewriting history)

If keeping the old history is acceptable and you only want to stop tracking a folder going forward:

echo "some/folder/" >> .gitignore
git rm -r --cached some/folder
git commit -m "stop tracking some/folder"

The folder remains on disk and in old commits, but Git will no longer track changes to it.

BFG Repo Cleaner (alternative)

BFG is a Java-based alternative, simpler for the specific case of removing large files or secrets:

# Remove all files named id_rsa anywhere in history
java -jar bfg.jar --delete-files id_rsa repo-mirror.git

# Remove files larger than 50MB
java -jar bfg.jar --strip-blobs-bigger-than 50M repo-mirror.git

# After BFG, expire and clean
cd repo-mirror.git
git reflog expire --expire=now --all
git gc --prune=now --aggressive
git push --force

BFG is faster than git filter-repo for large repos but less flexible - it cannot filter by path prefix or apply arbitrary transformations.

Notes on GitHub / GitLab caching

Even after a force-push, the hosting platform may cache old objects for a while. If you removed sensitive data:

• GitHub: contact support to run a git garbage collection on their end
• GitLab: Repository → Housekeeping in project settings, or contact support for object pruning

Resources

• git-filter-repo on GitHub
• BFG Repo Cleaner
• GitHub: removing sensitive data

Python

• Syntax
• Django

Books

• Learning Python The Hard Way
• Hacker’s Guide to Python

Setup development

$ python -m ensurepip --user
$ python -m pip install --user --upgrade pip
$ python -m pip install --user --upgrade virtualenv

$ python -m virtualenv lets-go
$ . ./lets-go/bin/activate
(lets-go) $ _

Source

$ python3 -m venv lets-go
$ source ./lets-go/bin/activate
(lets-go) $ pip install -U pip setuptools

Install pip

$ yum -y install epel-release
$ yum -y install python-pip

$ apt-get -y install python-pip

Upgrade pip

$ pip install -U pip

Install requirements using pip

$ pip install -U -r requirements.txt

Install in userhome using pip

$ pip install --user [name]

Install specific version using pip

$ pip install ansible==2.0

Install virtualenv

$ dnf install python-virtualenv

$ dnf install python-pip
$ pip install virtualenv

Create (and activate) virtualenv

$ mkdir .venv
$ virtualenv .venv
$ source .venv/bin/activate

Debugging: trigger trace/breakpoint

Insert the following somewhere in your code:

$ import pdb; pdb.set_trace()

More information: Tools training

Debugging using ‘q’

$ pip install q

• Output is sent to $TMPDIR/q
• replace print with q
• use the @q decorator

Source

Troubleshooting

Needed on Ubuntu 14.04 when dealing with oudated SSL and InsecurePlatformWarning

$ pip install -U pyopenssl ndg-httpsclient pyasn1

Syntax

Comments

>>> # This is a comment
>>> # print("Hello World!")
... 
>>>

Keywords and Identifiers

False      class      finally    is         return
None       continue   for        lambda     try
True       def        from       nonlocal   while
and        del        global     not        with
as         elif       if         or         yield
assert     else       import     pass
break      except     in         raise

Variables and Datatypes

>>> a = 12
>>> type(a)
<class 'int'>

>>> a = 1.0
>>> type(a)
<class 'float'>

>>> a = "Hello World"
>>> type(a)
<class 'str'>

Strings

>>> 'World'
'World'
>>> "World"
'World'
>>> 'World\'s Best'
"World's Best"
>>> "World's Best"
"World's Best"

Read User Input

>>> number = int(input("Enter a number: "))
Enter a number: 5
>>> print(number)
5

name = str(input("Enter your name: "))
Enter your name: root
>>> print(name)

Multiple Assignments

>>> a, b = 4, 5
>>> a
4
>>> b
5

>>> a, b = b, a
>>> a
5
>>> b
4

Operators

• Mathematical
  +, -, /, %, *
• Logical
  and, or
• Relational
  <, <=, >, >=, ==, !=

Expressions

a = 9
b = 12
c = 3
x = a - b / 3 + c * 2 - 1
y = a - b / (3 + c) * (2 - 1)
z = a - (b / (3 + c) * 2) - 1
print("X = ", x)
print("Y = ", y)
print("Z = ", z)

Type Conversion

float(string) -> float value
int(string) -> integer value
str(integer) -> string representation
str(float) -> string representation

>>> a = 8.126768
>>> str(a)
'8.126768'

Print and string formatting

>>> name = 'Yongyuan'
>>> country = 'China'
>>> print("%s is from %s" % (name, country))
Yongyuan is from China
>>> print("{0} is from {1}".format(name, country))
Yongyuan is from China
>>> print("{} is from {}".format(name, country))
Yongyuan is from China
>>> print("{} is from {}".format(name))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: tuple index out of range

Conditions and Control Flow

if expression:
    do this
elif expression:
    do this
else: 
    do this

Do’s and dont’s

if x == True:
    do not this  
 
if x == False:
    do not this

if x:
    do this
 
if not x:
    do this

Looping

While Loop

while condition:
    statement1
    statement2

>>> # Print 0-10
>>> n = 0 
>>> while n < 11:
...     print(n)
...     n += 1

For Loop

for iterating_var in sequence:
    statement
>>> for i in range(0, 11):  # Print 0-10
...     print(i)

>>> sum = 0
>>> n = 10
>>> for i in range(1, n):
...     sum += i
...
>>> print(sum)
45

Break

>>> word = "Python2 Programming"
>>> for letter in word:
...     if letter == "2":
...         break
...     print(letter)
...
P
y
t
h
o
n

Continue

>>> word = "Python3 Programming"
>>> for letter in word:
...     if letter == "3":
...         continue
...     print(letter)
...
P
y
t
h
o
n
 
P
r
.

Data structures

• List
• Tuple
• Dictionary
• Set
• String

Lists

Supports multiple datatypes

>>> list = [1, 2, "hello", 1.0]
>>> list
[1, 2, 'hello', 1.0]

List Operations

>>> a = [23, 45, 1, -3434, 43624356, 234]
>>> a.append(45)
>>> a
[23, 45, 1, -3434, 43624356, 234, 45]

>>> a.insert(0, 111) # Inserts 111 at 0th Position
>>> a
[111, 1, 23, 45, 1, -3434, 43624356, 234, 45]
>>> a.count(45)
2
>>> a.remove(234)
>>> a
[111, 1, 23, 45, 1, -3434, 43624356, 45]

>>> b = [45, 56, 90]
>>> a.append(b)
>>> a [45, 43624356, -3434, 1, 45, 23, 1, 111, [45, 56, 90]]
>>> a[-1]
>>> [45, 56, 90]
>>> a.extend(b) #To add the values of b not the b itself
>>> a
[45, 43624356, -3434, 1, 45, 23, 1, 111, [45, 56, 90], 45, 56, 90]
>>> a[-1]
90
>>> a.remove(b)
>>> a
[45, 43624356, -3434, 1, 45, 23, 1, 111, 45, 56, 90]
>>> a.sort()
>>> a
[-3434, 1, 1, 23, 45, 45, 45, 56, 90, 111, 43624356]

Lists Operations (Stack and Queue)

>>> a = [1, 2, 3, 4, 5, 6]
>>> a
[1, 2, 3, 4, 5, 6]
>>> a.pop() # Pop Operation
6
>>> a.pop()
5
>>> a.pop()
4
>>> a
[1, 2, 3]
>>> a.append(34) # Push Operation
>>> a
[1, 2, 3, 34]
>>> a.append(1)
>>> a
[1, 2, 3, 34, 1]

Lists Comprehensions

>>> a = [1, 2, 3]
>>> [x ** 2 for x in a]
[1, 4, 9]

>>> z = [x + 1 for x in [x ** 2 for x in a]]
>>> z
[2, 5, 10]

Tuple

>>> a = 'Fedora', 'Debian', 10, 12
>>> a
('Fedora', 'Debian', 10, 12)
>>> a[1]
'Debian'
>>> for x in a:
...     print(x, end=' ')
...
Fedora Debian 10 12
>>> type(a)
<class 'tuple'>
>>> len(a)
4

Immutable (Cannot modify any value)

>>> a = (1, 2, 3, 4)
>>> del a[0]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object doesn't support item deletion

Tuple Packing & Unpacking

>>> a = (1, 2, 3) # Packing
>>> (first, second, third) = a # Unpacking
>>> first
1
>>> second
2
>>> third
3
>>> (first, second, third) = (1, 2, 3) # Another way

Dictionary

>>> data = {'quentin':'Fedora', 'diwen':'Debian', 'steve':'Mac'}
>>> data
{'quentin': 'Fedora', 'steve': 'Mac', 'diwen': 'Debian'}
>>> data['steve']
'Mac'
>>> data['gerard'] = 'Fedora'
>>> data
{'gerard': 'Fedora', 'quentin': 'Fedora', 'steve': 'Mac', 'diwen': 'Debian'}
>>> del data['steve']
>>> data
{'gerard': 'Fedora', 'quentin': 'Fedora', 'diwen': 'Debian'}

Loop through Dict

>>> for x, y in data.items():
...     print("%s uses %s" % (x, y))
...
gerard uses Fedora
quentin uses Fedora
diwen uses Debian

Sets

>>> a = set('abcthabcjwethddda')
>>> a
{'a', 'c', 'b', 'e', 'd', 'h', 'j', 't', 'w'}
>>> a.add('p')
>>> a
{'a', 'c', 'b', 'e', 'd', 'h', 'j', 'q', 'p', 't', 'w'}

Slicing

List

>>> a = [1, 2, 3, 4, 5, 6, 7]
>>> a[1:4]
[2, 3, 4]
>>> a[:-1]
[1, 2, 3, 4, 5, 6]
>>> a[::-1]
[7, 6, 5, 4, 3, 2, 1] # Reverse

Tuple

>>> a = (1, 2, 3, 4, 5, 6, 7)
>>> a[1:4]
(2, 3, 4)
>>> a[::-1]
(7, 6, 5, 4, 3, 2, 1)

String

>>> a = 'Hello World'
>>> a[0]
'H'
>>> a[5]
' '
>>> a[1:]
'ello World'
>>> a[::-1] # Reverse
'dlroW olleH'

>>> s = "I am Dutch"
>>> s
I am Dutch'
>>> s = "Here is a line \
... split in two lines"
>>> s
'Here is a line split in two lines

>>> s = """ This is a
... multiline string, so you can
... write many lines"""
>>> print(s)
This is a
multiline string, so you can
write many lines

Operations

>>> s = "gerard braad"
>>> s.title()
'Gerard Braad'
>>> z = s.upper()
>>> z
'GERARD BRAAD'
>>> z.lower()
'gerard braad'
​>>> s = "123"
>>> s.isdigit()
True
>>> s = "Fedora24"
>>> s.isdigit()
False

Split Method

>>> s = "Let's learn Python3"
>>> s.split(" ")
["Let's", 'learn', 'Python3']

>>> s.split()
["Let's", 'learn', 'Python3']

>>> s = "This is: ME"
>>> s.split(":")
['This is', ' ME']

Join Method

>>> s = "We all love Python."
>>> s.split(".")
['We all love Python', '']

>>> "3".join(s.split("."))
'We all love Python3'

Strip Method

>>> s = "    abc  "
>>> s.strip()
'abc'

>>> # Strip from left hand and right hand size
>>> s = "www.fedoraproject.org"
>>> s.lstrip("www.")
'fedoraproject.org'

>>> s.rstrip(".org")
'www.fedoraproject'

Enumerate

>>> for i, j in enumerate(['a', 'b', 'c']):
...     print(i, j)
...
0 a
1 b
2 c

Functions

def functionname(params):
    statement1
    statement2

>>> def sum(a, b):
...     return a + b
 
>>> res = sum(2, 3)
>>> res
5

Check Palindrome

>>> def check_palindrome(word):
...     rev_word = word[::-1]
...     if word == rev_word:
...         print("Palindrome")
...     else:
...         print("Not Palindrome")
...
>>> check_palindrome("madam")
Palindrome
>>> check_palindrome("python3")
Not Palindrome

Default Arguments

>>> def test(a , b=-99):
...     if a > b:
...         return True
...     else:
...         return False

>>> test(12, 23)
False
>>> test(12)
True

Keyword Arguments

>>> def func(a, b=5, c=10):
...     print('a is', a, 'and b is', b, 'and c is', c)
...
>>> func(12)
a is 12 and b is 5 and c is 10
>>> func(12, 24)
a is 12 and b is 24 and c is 10
>>> func(12, c = 25)
a is 12 and b is 5 and c is 25
>>> func(a=12)
a is 12 and b is 5 and c is 10

Note: Can not have non-keyword argument after a keyword based argument

File Handling

r - Open with read only mode
w - Open with write mode
a - Open with append mode

Reading File

>>> f = open('sample.txt', 'r')
>>> f.read()
"Hello World.\nLet's learn Python3.\n"
>>> f.close()
>>> f = open('sample.txt', 'r')
>>> f.readline()
'Hello World.\n'
>>> f.readline()
"Let's learn Python3.\n"
>>> f.close()
>>> f = open('sample.txt', 'r')
>>> f.readlines()
['Hello World.\n', "Let's learn Python3.\n"]
>>> f.close()

Loop through Lines

>>> f = open('sample.txt', 'r')
>>> for i in f:
...     if 'Python' in i:
...         print(i)
...    f.close()

Let's learn Python3.

Write and Append

>>> f = open('test.txt', 'w') # Write
>>> f.write("Hello world")
>>> f.close()
>>> f = open('test.txt', 'r')
>>> f.read()
'Hello world'
>>> f.close()
>>> f = open('test.txt', 'a') # Append
>>> f.write(" Python')
>>>f.close()
>>> f = open('test.txt', 'r')
>>> f.read()
"Hello world Python"
>>>f.close()

Using with Statement

>>> with open('sample.txt', 'r') as f:
...     f.read()
...
"Hello World.\nLet's learn Python3.\n"

>>> with open('sample.txt', 'r') as f:
...     for i in f:
...         if 'Python' in i:
...             print(i)
...
Let's learn Python3.

Modules

• import module
• from module import something
• from module import *

math.py

def square(x):
    return x * x

result.py

from math import square
print(square(5))

__name__ and __main__

math.py

def square(x):
    return x * x
if __name__ == '__main__':
    print("Result: for square(5) == %d " % square(5))

result.py

import math
print(math.square(5))

Django

Installation

$ mkdir the-d-is-silent
$ cd the-d-is-silent
$ python -m virtualenv .
$ . ./bin/activate
$ pip install Django

Rust

An introductory course covering Rust’s core concepts. Rust is a systems language that guarantees memory safety without a garbage collector through its ownership model.

Course outline

• Basics - installation, hello world, variables, types, functions
• Ownership & Borrowing - ownership, borrowing, references, slices
• Structs, Enums & Patterns - structs, enums, pattern matching
• Error Handling - Result, Option, the ? operator
• Traits & Generics - traits, generics, lifetimes

Resources

• The Rust Book
• Rust by Example
• Rustlings exercises
• Standard library docs

Install

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup update

cargo new hello_world
cargo build
cargo run
cargo test

Rust - Basics

Hello world

fn main() {
    println!("Hello, world!");
}

Variables

Variables are immutable by default. Use mut to make them mutable.

#![allow(unused)]
fn main() {
let x = 5;          // immutable
let mut y = 5;      // mutable
y = 6;

const MAX: u32 = 100_000;  // constant, must have type annotation
}

Shadowing - redeclare with let to change type or value:

#![allow(unused)]
fn main() {
let x = 5;
let x = x + 1;      // shadows the previous x
let x = "hello";    // can even change type
}

Data types

Scalar types

#![allow(unused)]
fn main() {
let n: i32 = -42;
let f: f64 = 3.14;
let b: bool = true;
let c: char = '🦀';
}

Compound types

#![allow(unused)]
fn main() {
// Tuple - fixed length, mixed types
let tup: (i32, f64, char) = (42, 3.14, 'z');
let (x, y, z) = tup;    // destructure
let first = tup.0;       // index access

// Array - fixed length, same type
let arr: [i32; 5] = [1, 2, 3, 4, 5];
let zeros = [0; 5];      // [0, 0, 0, 0, 0]
let elem = arr[0];
}

Functions

#![allow(unused)]
fn main() {
fn add(x: i32, y: i32) -> i32 {
    x + y  // no semicolon = expression, this is the return value
}

fn nothing() {  // returns () (unit type) implicitly
    println!("side effect only");
}
}

Expressions vs statements:

• Statement - performs an action, no value (let x = 5;)
• Expression - evaluates to a value (blocks, if, match, function calls)

#![allow(unused)]
fn main() {
let y = {
    let x = 3;
    x + 1   // expression - block evaluates to 4
};
}

Control flow

#![allow(unused)]
fn main() {
// if is an expression
let number = 7;
let msg = if number < 10 { "small" } else { "large" };

// loop
let mut count = 0;
let result = loop {
    count += 1;
    if count == 10 {
        break count * 2;  // break with value
    }
};

// while
while count > 0 {
    count -= 1;
}

// for - preferred over manual indexing
let a = [10, 20, 30];
for elem in a {
    println!("{elem}");
}

for i in 0..5 {       // 0, 1, 2, 3, 4
    println!("{i}");
}
for i in 0..=5 {      // 0, 1, 2, 3, 4, 5
    println!("{i}");
}
}

Strings

Rust has two string types:

#![allow(unused)]
fn main() {
let s1: &str = "hello";           // string literal, 'static lifetime
let s2: String = String::from("hello");
let s3 = "world".to_string();

// Concatenation
let s4 = s2 + " world";           // moves s2, appends
let s5 = format!("{s1} world");   // doesn't move anything

// Slices
let hello = &s5[0..5];
let len = s5.len();
let is_empty = s5.is_empty();

// Iteration
for c in "hello".chars() {
    println!("{c}");
}
}

Printing

#![allow(unused)]
fn main() {
println!("{}", value);        // Display
println!("{:?}", value);      // Debug
println!("{:#?}", value);     // Pretty debug
println!("{value}");          // Inline (Rust 1.58+)
eprintln!("to stderr: {}", value);
}

Rust - Ownership & Borrowing

Rust’s ownership system is what makes memory safety without a GC possible. It is checked entirely at compile time.

Ownership rules

1. Each value has exactly one owner.
2. When the owner goes out of scope, the value is dropped (memory freed).
3. There can only be one owner at a time - assigning moves it.

#![allow(unused)]
fn main() {
{
    let s = String::from("hello");  // s owns the String
    // use s
}   // s goes out of scope, String is dropped here
}

Move

Assigning a heap-allocated value moves ownership - the original variable is no longer valid:

#![allow(unused)]
fn main() {
let s1 = String::from("hello");
let s2 = s1;          // s1 is moved into s2

// println!("{s1}");  // ERROR: s1 was moved
println!("{s2}");     // OK
}

Stack-only types (integers, bools, chars, tuples of these) implement Copy and are copied instead of moved:

#![allow(unused)]
fn main() {
let x = 5;
let y = x;   // x is copied
println!("{x} and {y}");  // both valid
}

Clone

To deep-copy heap data explicitly:

#![allow(unused)]
fn main() {
let s1 = String::from("hello");
let s2 = s1.clone();
println!("{s1} and {s2}");  // both valid
}

References & borrowing

A reference lets you use a value without taking ownership - this is called borrowing.

#![allow(unused)]
fn main() {
fn calculate_length(s: &String) -> usize {
    s.len()
}   // s goes out of scope but does NOT drop the String (it doesn't own it)

let s = String::from("hello");
let len = calculate_length(&s);  // pass reference
println!("{s} has length {len}");  // s still valid
}

Mutable references

#![allow(unused)]
fn main() {
fn change(s: &mut String) {
    s.push_str(", world");
}

let mut s = String::from("hello");
change(&mut s);
}

Rules:

• Any number of immutable references at the same time - OR —
• Exactly one mutable reference
• Not both at the same time (prevents data races at compile time)

#![allow(unused)]
fn main() {
let mut s = String::from("hello");

let r1 = &s;
let r2 = &s;
println!("{r1} and {r2}");  // r1, r2 last used here

let r3 = &mut s;            // OK - r1 and r2 no longer in use
println!("{r3}");
}

Slices

A slice is a reference to a contiguous sequence - it does not own data.

#![allow(unused)]
fn main() {
let s = String::from("hello world");

let hello: &str = &s[0..5];
let world: &str = &s[6..11];
let all:   &str = &s[..];

// String literals are slices
let literal: &str = "hello";  // &'static str
}

Array slices:

#![allow(unused)]
fn main() {
let a = [1, 2, 3, 4, 5];
let slice: &[i32] = &a[1..3];  // [2, 3]
}

Lifetimes

Lifetimes ensure references don’t outlive the data they point to. The compiler infers them in most cases; explicit annotations are needed when returning references from functions with multiple inputs.

#![allow(unused)]
fn main() {
// 'a is a lifetime annotation - both inputs and output share the same lifetime
fn longest<'a>(x: &'a str, y: &'a str) -> &'a str {
    if x.len() > y.len() { x } else { y }
}
}

The 'static lifetime means the reference is valid for the entire program:

#![allow(unused)]
fn main() {
let s: &'static str = "I live forever";
}

Rust - Structs, Enums & Pattern Matching

Structs

#![allow(unused)]
fn main() {
struct User {
    username: String,
    email: String,
    age: u32,
    active: bool,
}

let user = User {
    username: String::from("gbraad"),
    email: String::from("me@gbraad.nl"),
    age: 30,
    active: true,
};

println!("{}", user.username);

// Update syntax - copy remaining fields from another instance
let user2 = User {
    email: String::from("other@example.com"),
    ..user   // moves remaining fields from user
};
}

Tuple structs

#![allow(unused)]
fn main() {
struct Color(u8, u8, u8);
struct Point(f64, f64);

let black = Color(0, 0, 0);
let origin = Point(0.0, 0.0);
println!("{}", black.0);
}

Methods

#![allow(unused)]
fn main() {
#[derive(Debug)]
struct Rectangle {
    width: f64,
    height: f64,
}

impl Rectangle {
    // Associated function (constructor, no self)
    fn new(width: f64, height: f64) -> Self {
        Rectangle { width, height }
    }

    // Method (takes &self)
    fn area(&self) -> f64 {
        self.width * self.height
    }

    fn can_hold(&self, other: &Rectangle) -> bool {
        self.width > other.width && self.height > other.height
    }
}

let rect = Rectangle::new(10.0, 5.0);
println!("Area: {}", rect.area());
println!("{rect:#?}");   // needs #[derive(Debug)]
}

Enums

#![allow(unused)]
fn main() {
enum Direction {
    North,
    South,
    East,
    West,
}

let dir = Direction::North;
}

Enum variants can carry data:

#![allow(unused)]
fn main() {
enum Message {
    Quit,
    Move { x: i32, y: i32 },
    Write(String),
    Color(u8, u8, u8),
}

impl Message {
    fn call(&self) {
        match self {
            Message::Quit => println!("quit"),
            Message::Move { x, y } => println!("move to {x},{y}"),
            Message::Write(text) => println!("write: {text}"),
            Message::Color(r, g, b) => println!("color: {r},{g},{b}"),
        }
    }
}
}

Option

Option<T> replaces null - a value is either Some(T) or None:

#![allow(unused)]
fn main() {
let some_number: Option<i32> = Some(5);
let no_number: Option<i32> = None;

// Must handle both cases before using the value
let value = some_number.unwrap_or(0);
let doubled = some_number.map(|n| n * 2);
}

Pattern matching

match

#![allow(unused)]
fn main() {
let coin = Coin::Quarter;
let value = match coin {
    Coin::Penny   => 1,
    Coin::Nickel  => 5,
    Coin::Dime    => 10,
    Coin::Quarter => 25,
};

// Catch-all
match number {
    1 => println!("one"),
    2 | 3 => println!("two or three"),
    4..=9 => println!("four to nine"),
    other => println!("got {other}"),  // binds the value
    // _ => ()                         // discard catch-all
}

// Destructuring in match
match msg {
    Message::Move { x, y } => println!("{x},{y}"),
    Message::Write(text)   => println!("{text}"),
    _ => (),
}
}

if let - single-arm match shorthand

#![allow(unused)]
fn main() {
let config_max = Some(3u8);
if let Some(max) = config_max {
    println!("max is {max}");
}

if let Some(n) = some_value {
    println!("{n}");
} else {
    println!("nothing");
}
}

while let

#![allow(unused)]
fn main() {
let mut stack = vec![1, 2, 3];
while let Some(top) = stack.pop() {
    println!("{top}");
}
}

Collections

Vec

#![allow(unused)]
fn main() {
let mut v: Vec<i32> = Vec::new();
let v2 = vec![1, 2, 3];     // macro shorthand

v.push(4);
v.push(5);

let third = &v[2];           // panics if out of bounds
let third = v.get(2);        // returns Option<&i32>

for n in &v {
    println!("{n}");
}
for n in &mut v {
    *n += 10;                // dereference to modify
}

v.pop();                     // removes and returns last element
v.len();
v.is_empty();
}

HashMap

#![allow(unused)]
fn main() {
use std::collections::HashMap;

let mut scores: HashMap<String, i32> = HashMap::new();
scores.insert(String::from("Alice"), 100);
scores.insert(String::from("Bob"), 85);

// Access
let alice = scores.get("Alice");     // Option<&i32>
let alice = scores["Alice"];         // panics if missing

// Insert only if key absent
scores.entry(String::from("Charlie")).or_insert(70);

// Iterate
for (name, score) in &scores {
    println!("{name}: {score}");
}
}

Rust - Error Handling

Rust has no exceptions. Errors are values, split into two categories:

panic!

#![allow(unused)]
fn main() {
panic!("something went terribly wrong");

// Automatic panics
let v = vec![1, 2, 3];
v[99];            // index out of bounds - panics

let n: i32 = "abc".parse().unwrap();  // unwrap panics on Err
}

Set RUST_BACKTRACE=1 to see a full stack trace.

Result

#![allow(unused)]
fn main() {
enum Result<T, E> {
    Ok(T),
    Err(E),
}
}

#![allow(unused)]
fn main() {
use std::fs::File;

let f = File::open("hello.txt");

let file = match f {
    Ok(file) => file,
    Err(e)   => panic!("failed to open: {e}"),
};
}

Matching on error kind

#![allow(unused)]
fn main() {
use std::io::ErrorKind;

let file = match File::open("hello.txt") {
    Ok(f) => f,
    Err(e) => match e.kind() {
        ErrorKind::NotFound => match File::create("hello.txt") {
            Ok(fc) => fc,
            Err(ce) => panic!("could not create: {ce}"),
        },
        other => panic!("other error: {other:?}"),
    },
};
}

Shortcuts

#![allow(unused)]
fn main() {
// unwrap - returns T or panics
let f = File::open("hello.txt").unwrap();

// expect - like unwrap but with a message
let f = File::open("hello.txt").expect("hello.txt should exist");

// unwrap_or - return default on Err
let f = File::open("hello.txt").unwrap_or_else(|_| File::create("hello.txt").unwrap());
}

The ? operator

Propagates errors up to the caller - equivalent to a match that returns Err early:

#![allow(unused)]
fn main() {
use std::io::{self, Read};
use std::fs::File;

fn read_username() -> Result<String, io::Error> {
    let mut f = File::open("username.txt")?;  // returns Err early if fails
    let mut s = String::new();
    f.read_to_string(&mut s)?;
    Ok(s)
}

// Chained
fn read_username() -> Result<String, io::Error> {
    let mut s = String::new();
    File::open("username.txt")?.read_to_string(&mut s)?;
    Ok(s)
}

// Even shorter with std::fs
fn read_username() -> Result<String, io::Error> {
    std::fs::read_to_string("username.txt")
}
}

? can also be used with Option<T> to return None early.

Custom error types

#![allow(unused)]
fn main() {
use std::fmt;

#[derive(Debug)]
enum AppError {
    Io(std::io::Error),
    Parse(std::num::ParseIntError),
    NotFound(String),
}

impl fmt::Display for AppError {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        match self {
            AppError::Io(e)        => write!(f, "IO error: {e}"),
            AppError::Parse(e)     => write!(f, "parse error: {e}"),
            AppError::NotFound(s)  => write!(f, "not found: {s}"),
        }
    }
}

impl From<std::io::Error> for AppError {
    fn from(e: std::io::Error) -> Self {
        AppError::Io(e)
    }
}
}

With From implemented, ? automatically converts the error type.

Returning errors from main

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let n: i32 = "42".parse()?;
    println!("{n}");
    Ok(())
}

Box<dyn std::error::Error> accepts any error type - useful for quick scripts. Use a typed error for libraries.

Rust - Traits & Generics

Traits

A trait defines shared behaviour - similar to interfaces in other languages.

#![allow(unused)]
fn main() {
trait Summary {
    fn summarize(&self) -> String;

    // Default implementation (can be overridden)
    fn preview(&self) -> String {
        format!("{}...", &self.summarize()[..20])
    }
}

struct Article {
    title: String,
    content: String,
}

impl Summary for Article {
    fn summarize(&self) -> String {
        format!("{}: {}", self.title, self.content)
    }
}

struct Tweet {
    username: String,
    content: String,
}

impl Summary for Tweet {
    fn summarize(&self) -> String {
        format!("@{}: {}", self.username, self.content)
    }
}
}

Traits as parameters

#![allow(unused)]
fn main() {
// impl Trait syntax (shorthand)
fn notify(item: &impl Summary) {
    println!("{}", item.summarize());
}

// Trait bound syntax (equivalent, more flexible)
fn notify<T: Summary>(item: &T) {
    println!("{}", item.summarize());
}

// Multiple bounds
fn notify(item: &(impl Summary + std::fmt::Display)) { }
fn notify<T: Summary + std::fmt::Display>(item: &T) { }

// where clause - cleaner with many bounds
fn notify<T, U>(t: &T, u: &U)
where
    T: Summary + Clone,
    U: std::fmt::Debug,
{ }
}

Returning traits

#![allow(unused)]
fn main() {
// Return something that implements Summary (concrete type hidden)
fn make_summary() -> impl Summary {
    Tweet {
        username: String::from("gbraad"),
        content: String::from("Hello!"),
    }
}
}

Common standard library traits

Generics

#![allow(unused)]
fn main() {
// Generic function
fn largest<T: PartialOrd>(list: &[T]) -> &T {
    let mut largest = &list[0];
    for item in list {
        if item > largest {
            largest = item;
        }
    }
    largest
}

// Generic struct
struct Pair<T> {
    first: T,
    second: T,
}

impl<T> Pair<T> {
    fn new(first: T, second: T) -> Self {
        Pair { first, second }
    }
}

// Conditional implementation (only for T that implements Display + PartialOrd)
impl<T: std::fmt::Display + PartialOrd> Pair<T> {
    fn cmp_display(&self) {
        if self.first >= self.second {
            println!("largest: {}", self.first);
        } else {
            println!("largest: {}", self.second);
        }
    }
}
}

Iterators

Any type implementing Iterator gets all adaptor methods for free:

#![allow(unused)]
fn main() {
trait Iterator {
    type Item;
    fn next(&mut self) -> Option<Self::Item>;
}
}

#![allow(unused)]
fn main() {
let v = vec![1, 2, 3, 4, 5];

// Adaptors (lazy - don't execute until consumed)
let doubled: Vec<i32> = v.iter().map(|x| x * 2).collect();
let evens: Vec<&i32>  = v.iter().filter(|x| *x % 2 == 0).collect();
let sum: i32           = v.iter().sum();
let total: i32         = v.iter().fold(0, |acc, x| acc + x);

// any / all
let has_even = v.iter().any(|x| x % 2 == 0);
let all_pos  = v.iter().all(|x| *x > 0);

// Chaining
let result: Vec<i32> = v.iter()
    .filter(|&&x| x > 2)
    .map(|&x| x * 10)
    .collect();
}

Closures

#![allow(unused)]
fn main() {
let add = |x, y| x + y;
let double = |x: i32| -> i32 { x * 2 };

// Closures capture their environment
let multiplier = 3;
let triple = |x| x * multiplier;   // captures multiplier by reference

// move closure - takes ownership of captured variables
let s = String::from("hello");
let greet = move || println!("{s}");
}

Closure trait bounds:

• Fn - borrows immutably (can call multiple times)
• FnMut - borrows mutably (can call multiple times)
• FnOnce - takes ownership (can only call once)

#![allow(unused)]
fn main() {
fn apply<F: Fn(i32) -> i32>(f: F, value: i32) -> i32 {
    f(value)
}

let result = apply(|x| x * 2, 5);   // 10
}

Newlib

Newlib is a C standard library implementation designed for embedded and bare-metal targets - systems with no operating system. It is the default libc bundled with the arm-none-eabi and riscv32-unknown-elf (and similar ELF) toolchains, replacing glibc which assumes a full POSIX OS beneath it.

Why newlib exists

A standard C library needs OS services for things like:

• Writing output (printf → write syscall)
• Allocating heap memory (malloc → sbrk/brk syscall)
• Exiting (exit → _exit syscall)
• Opening files (fopen → open syscall)

On bare-metal there are no syscalls. Newlib solves this by splitting the library into two layers:

• High-level portable code - printf, malloc, memcpy, strlen, etc.
• Low-level syscall stubs - thin functions you must implement to bridge newlib to your hardware

This bridging is called retargeting.

Two variants

newlib-nano replaces the standard printf/scanf with smaller versions that drop support for floating-point format specifiers by default (add --specs=nano.specs -u _printf_float to restore them).

Syscall stubs you must implement

Newlib calls these functions; you provide the implementation for your hardware:

Minimal retarget for a UART-based target:

#include <sys/stat.h>
#include <errno.h>

// UART transmit - implement for your hardware
extern void uart_putchar(char c);

int _write(int fd, char *buf, int len) {
    for (int i = 0; i < len; i++) {
        uart_putchar(buf[i]);
    }
    return len;
}

// Heap: grow from end of .bss toward stack
extern char _end;           // linker symbol: end of .bss
static char *heap_ptr = &_end;

void *_sbrk(int incr) {
    char *prev = heap_ptr;
    heap_ptr += incr;
    return (void *)prev;
}

// Minimal stubs - just enough to link
int  _close(int fd)                        { return -1; }
int  _fstat(int fd, struct stat *st)       { st->st_mode = S_IFCHR; return 0; }
int  _isatty(int fd)                       { return 1; }
int  _lseek(int fd, int offset, int whence){ return 0; }
int  _read(int fd, char *buf, int len)     { return 0; }
void _exit(int status)                     { while (1); }

Linking with newlib

arm-none-eabi

# newlib-nano + stub syscalls provided by -lnosys (all return -1/ENOSYS)
arm-none-eabi-gcc \
  -mcpu=cortex-m4 -mthumb -mfpu=fpv4-sp-d16 -mfloat-abi=hard \
  --specs=nano.specs --specs=nosys.specs \
  -T linker.ld -o firmware.elf startup.s retarget.c main.c

# newlib-nano + your own retarget.c (omit --specs=nosys.specs)
arm-none-eabi-gcc \
  -mcpu=cortex-m4 -mthumb -mfpu=fpv4-sp-d16 -mfloat-abi=hard \
  --specs=nano.specs \
  -T linker.ld -o firmware.elf startup.s retarget.c main.c -lc -lm

riscv32-unknown-elf

# newlib-nano + stub syscalls
riscv32-unknown-elf-gcc \
  -march=rv32im -mabi=ilp32 \
  --specs=nano.specs --specs=nosys.specs \
  -T linker.ld -o program.elf startup.s retarget.c main.c

# Your own retarget
riscv32-unknown-elf-gcc \
  -march=rv32im -mabi=ilp32 \
  --specs=nano.specs \
  -T linker.ld -o program.elf startup.s retarget.c main.c -lc -lm

--specs flags reference

Semihosting

Semihosting is an alternative to retargeting: instead of routing printf to a UART, it tunnels the output back through the debugger (JTAG/SWD) to the host terminal. Useful during development - no UART required.

arm-none-eabi-gcc ... --specs=rdimon.specs -lrdimon

In QEMU:

qemu-system-arm -M mps2-an385 -semihosting -kernel firmware.elf

Do not ship firmware built with rdimon.specs - it will hang without a debugger attached.

What newlib provides (without retargeting)

Even without implementing any syscalls, you get the full non-OS-dependent portion of libc:

• string.h - memcpy, memset, strlen, strcmp, strcpy, …
• stdlib.h - atoi, strtol, abs, qsort, bsearch
• math.h - sin, cos, sqrt, fabs, … (link with -lm)
• stdint.h, stdbool.h, stddef.h - type definitions
• printf / sprintf - works without _write if you only use sprintf to format into a buffer

malloc/free require _sbrk. printf to stdout requires _write.

Linker script considerations

Newlib’s _sbrk needs a _end symbol marking the end of statically allocated data (the start of the heap). Add it to your linker script:

.bss : {
    _bss_start = .;
    *(.bss*) *(COMMON)
    _bss_end = .;
    _end = .;        /* newlib heap starts here */
} > RAM

.heap (NOLOAD) : {
    . = ALIGN(8);
    . += 4K;         /* reserve heap space */
} > RAM

.stack (NOLOAD) : {
    . = ALIGN(8);
    _stack_top = . + 8K;
} > RAM

Resources

• Newlib source
• newlib-nano announcement
• Retargeting guide (Arm)

Go (programming language)

Installation

$ VERSION=1.7.3
$ OS=linux
$ ARCH=amd64
$ wget https://storage.googleapis.com/golang/go$VERSION.$OS-$ARCH.tar.gz
$ tar -C /usr/local -xzf go$VERSION.$OS-$ARCH.tar.gz
$ export PATH=$PATH:/usr/local/go/bin

• Download page
• Install instructions

Ansible

Passing variables from the command line

---

- hosts: '{{ hosts }}'
  remote_user: '{{ user }}'

  tasks:
     - ...

ansible-playbook release.yml --extra-vars "hosts=vipers user=starbuck"

Ref: source

Execute command on remote host

$ ansible -i hosts all -u centos -s -m shell -a "rm -rf /"

Executable playbooks

#!/usr/bin/env ansible-playbook -i ../hosts -K
---
- hosts: ...

Debug modules

print will not work, so instead use the following at the top of the module:

import logging
logging.basicConfig(filename="/tmp/ansible-debug.log', level=logging.DEBUG)

Somewhere else in the code do:

logging.debug('your message')

And then tail the log file:

$ tail -f /tmp/ansible-debug.log

Alternatively you can use ‘q’

Debug messages

- debug: msg="System {{ inventory_hostname }}"

Source

keep remote files

$ ANSIBLE_KEEP_REMOTE_FILES=1 ansible-playbook ...

$ ls .ansible/tmp/ansible-tmp*

Cloud 9 IDE

This scratchpad will detail how to setup a powerful development environment in the cloud, under your control!

For more information about Cloud 9, have a look at:

• Homepage
• Source, Team with all the plugins

Installation

To install, choose your preferred method.

Oneliner

This will install C9 core (SDK and plugins) and needed dependencies on any machine or virtual machine, by using a single command.

$ curl -sSL https://raw.githubusercontent.com/gbraad/oneliners/master/install_c9.sh | bash

Install C9 on CentOS as SSH remote workspace

To setup a SSH remote workspace for C9 on CentOS do the following on the host:

curl -sSL https://raw.githubusercontent.com/gbraad/oneliners/master/install_c9.sh | bash

Go to http://c9.io/new and select SSH workspace. Add the key to ~/.ssh/authorized_keys for the user you allow access. As node executable refer to ~/.c9/node/bin/node

Note: I previously used the RPM packaged versions of node and npm, but it was not always succeeding. This might have been related to the network connection. This following these instructions the environment is immediately usable without having to do the install from the C9 workspace.

Ansible

This will install C9 core (SDK and plugins) and needed dependenciesusing, using an Ansible playbook. It is assumed that you target your localhost with this.

$ curl -sSL https://github.com/gbraad/ansible-playbooks/raw/master/playbooks/install-c9sdk.yml -o install-c9sdk.yml`
$ ansible-playbook install-c9sdk.yml

Also available available as a role for use in Ansible playbooks: GitHub, Galaxy. This can be helpful if you want to install to several machines. You first need to install the role:

$ ansible-galaxy install gbraad.c9sdk

After which you can define the playbook deploy-c9sdk-workstations.yml:

- name: Install C9 SDK
  hosts: workstations
  roles:
    - gbraad.c9sdk

And substitute your targetes in the hosts file:

[workstations]
192.168.1.[2:10]

you can deploy to multiple machines with:

$ ansible-playbook -i hosts deploy-c9sdk-workstations.yml

Running

After a susccesul install, the intrface can be started using the following command:

$ /home/user/.c9/node/bin/node /opt/c9sdk/server.js \
    --listen 0.0.0.0 \
    --port 8181 \
    -a $USERNAME:$PASSWORD \
    -w /workspace

Docker

The Docker container is based on standard images, provided by CentOS, Fedora and Ubuntu, and install curl, ansible and python to allow the Docker build process to deploy using the Ansible playbook. This means that the containers are minimal in setup, but allow you to fully customize the environment.

Using the following alias, you can easily develop on code in the current directory from your web browser.

$ alias c9ide='docker run -it --rm -v `pwd`:/workspace gbraad/c9ide:c7'
$ cd ~/Projects/[something]
$ c9ide

Different flavours exist, such as ‘c7’ (CentOS 7), ‘f24’ (Fedora 24) and ‘u16.04’ (Ubuntu Xenial).

Source: GitLab, GitHub, Docker hub

Docker cloud

Create a new account, or re-use your Docker hub acount, at Docker cloud. Now open the Node list and Bring your own node. Setting up the first node is free of charge. It explaisn you how to deploy the docker cloud engine on a machine with a simple one-liner.

Now open the Service list and create a new service. Search Docker hub for gbraad/c9ide, and confirm with clicking Select. Below are the settings to can/need to be changed to allow you to start the C9 service on your own Docker cloud node.

General settings

Select the preferred flavor from the image selecter.

• c7 is based on CentOS 7
• f24 is based on Fedora 24
• u1604 is based on Ubuntu 16.04 (xenial)

If you want to know what is in these images, please see the source under the Docker header on this page.

Note: Images with the -devenv suffix are based on my devenv environment. Unless you know what you are doing, it is best to not use these.

Ports

Container port    Protocol   Published   Node port
8181              tcp        X           80

Environment variables

Name              Value
PASSWORD          pass
USERNAME          user

After clicking Create & Deploy you will be shown the endpoints Open the Service endpoint for tcp/80 in your browser (without the tcp:// prefix) and you will be shown a authentication prompt. User the parameters you set from the Environment variables section and happy coding! Be aware that the first load of the page might take some time, as about 8.5M of data (JavaScript and CSS) need to be trasnferred.

Note: Be sure to push your code out to a git repository or attach a volume to /workspace. Also, the container runs on port 80, which is unprotected (no TLS), which means that you should not work on super-secret code or use ultra-secure user/pass combination, as these will be littered all over the internet.

Self-hosted

I wrote an article which improves on the previous trial with Docker cloud, by hosting the environment yourself, supported by a service container for Nginx and certificate provided by Let’s Encrypt.

• Published on my blog
• Source at GitLab, GitHub
• Published on GitBook

sed; stream editor

Change value

$ sed 's/SELINUX=targeted/SELINUX=permisssive/g' /etc/selinux/config

Shell scripting

Add current folder to path

$ export PATH=$PATH:$PWD

Audio

Notes on audio programming tools, environments, and projects.

• Pure Data - visual dataflow programming for audio and multimedia
• Csound - orchestra/score language for sound synthesis and processing
• Faust - functional DSP language that compiles to standalone C; targets Logue SDK, WebAssembly, and more
• Logue - Logue SDK effect units for the KORG NTS-1 / minilogue xd
• Schwung/Move effects - DSP effects for the Ableton Move (using Schwung)

Pure Data

Pure Data (Pd) is an open-source visual dataflow programming environment for audio, video, and interactive media. It was created by Miller Puckette - the same person who wrote Max/MSP - and is the free, open-source counterpart to that commercial tool.

Flavours

For most purposes start with Pd-vanilla and add externals as needed.

# Install
sudo apt install puredata          # Debian/Ubuntu
brew install pd                    # macOS

Core concepts

A Pd program is called a patch (.pd file). Everything in a patch is one of:

Patch cords connect an outlet of one object to an inlet of another. Data flows downward.

Control vs signal domain

Pd has two parallel domains:

Signal objects are suffixed with ~. You cannot connect a signal outlet to a control inlet directly - use snapshot~ or sig~ to cross between them.

Essential objects

Audio generation

[osc~ 440]      - sine oscillator at 440 Hz
[phasor~ 440]   - sawtooth wave (0..1 ramp)
[noise~]        - white noise
[sig~ 0.5]      - convert control value to signal (DC)

Audio processing

[*~ 0.5]        - multiply signal (amplitude scaling)
[+~ ]           - add two signals
[bp~ 1000 10]   - bandpass filter (freq, Q)
[lop~ 500]      - low-pass filter (one-pole)
[hip~ 100]      - high-pass filter (one-pole)
[vcf~ ]         - voltage-controlled filter
[delwrite~ buf 1000]  - write to delay buffer (name, max ms)
[delread~ buf 100]    - read from delay buffer (name, delay ms)
[rev3~]         - reverb

Envelopes

[line~]         - ramp generator: send [target time( message
[vline~]        - high-precision line~
[env~]          - amplitude follower (RMS envelope)
[adsr~ 10 100 0.7 200]  - ADSR envelope (attack/decay/sustain/release ms)

Control flow

[metro 500]     - send bang every 500 ms
[delay 200]     - delay a bang by 200 ms
[timer]         - measure time between bangs
[counter 0 7]   - count bangs from 0 to 7, wrap
[select 1 2 3]  - route bang to outlet matching value
[route float symbol]  - route by type
[trigger b f]   - fan out in right-to-left order (alias: [t b f])
[pack f f]      - combine values into a list
[unpack f f]    - split a list into values

I/O

[dac~]          - audio output (left, right inlets)
[adc~]          - audio input (left, right outlets)
[print]         - print to console
[receive name]  / [r name]  - receive messages by name (like a bus)
[send name]     / [s name]  - send messages by name

A minimal synthesizer patch

[keyboard]              ← MIDI note input (requires extra)

[notein]                ← MIDI note in (note, velocity, channel)
|         |
[mtof]    [/ 127]       ← MIDI to Hz, velocity 0..1
|         |
[osc~ ]   |
|         |
[*~      ]              ← scale amplitude by velocity
|
[dac~]                  ← to speakers

In text form (how the .pd file looks):

#X obj 100 100 notein;
#X obj 100 150 mtof;
#X obj 200 150 / 127;
#X obj 100 200 osc~;
#X obj 100 250 *~;
#X obj 100 300 dac~;
#X connect 0 0 1 0;
#X connect 0 1 2 0;
#X connect 1 0 3 0;
#X connect 3 0 4 0;
#X connect 2 0 4 1;
#X connect 4 0 5 0;
#X connect 4 0 5 1;

Running from the command line

# Open a patch with GUI
pd mypatch.pd

# Run headless (no GUI) - useful for servers or embedded
pd -nogui -open mypatch.pd

# Specify audio device and sample rate
pd -audiodev 2 -r 48000 mypatch.pd

# List available audio devices
pd -listdev

Subpatches and abstractions

[pd subpatch-name]  - inline subpatch (double-click to open)
[myabstraction]     - load myabstraction.pd as a reusable object

Abstractions use [inlet]/[outlet] (control) or [inlet~]/[outlet~] (signal) to define their ports. Store them in the same directory as the main patch or in Pd’s search path.

Externals

The ecosystem of third-party objects extends Pd significantly:

Install via Pd’s built-in package manager: Help → Find externals.

Resources

• Pd documentation
• The Programming Language PURE DATA (Kreidler)
• Pd vanilla releases
• ELSE library

Csound

Csound is a sound and music computing system with a long lineage: it descends from Music III/IV/V written by Max Mathews at Bell Labs in the 1950s–60s, and was developed by Barry Vercoe at MIT in 1985. It is one of the most powerful and flexible tools for sound synthesis and signal processing, used in music composition, sound design, research, and live performance.

Unlike Pure Data’s visual approach, Csound is a text-based language with a rich library of over 1500 opcodes.

Installation

sudo apt install csound csoundqt        # Debian/Ubuntu
brew install csound                     # macOS

Or download from csound.com.

The unified file format (.csd)

A Csound program is typically a single .csd file combining all three sections:

<CsoundSynthesizer>

<CsOptions>
-odac          ; output to DAC (real-time audio)
-d             ; suppress displays
</CsOptions>

<CsInstruments>
; Orchestra - defines instruments and signal flow
sr     = 44100   ; sample rate
ksmps  = 64      ; samples per control block
nchnls = 2       ; number of output channels
0dbfs  = 1       ; full-scale amplitude = 1.0

instr 1
  ifreq = p4     ; pitch from score (4th p-field)
  iamp  = p5     ; amplitude from score (5th p-field)

  aenv  linen iamp, 0.01, p3, 0.1    ; linear envelope
  asig  poscil  aenv, ifreq           ; sine oscillator
        outs    asig, asig            ; stereo output
endin
</CsInstruments>

<CsScore>
; Score - schedules instrument events
;  i  start  dur  freq   amp
   i1  0      1    440    0.5
   i1  1      1    550    0.5
   i1  2      2    660    0.3
e                             ; end of score
</CsScore>

</CsoundSynthesizer>

csound mypatch.csd

Rate system

Csound processes audio at three rates, and opcodes are rate-specific:

A variable’s rate is determined by its prefix letter. An opcode that outputs at a-rate has the output variable prefixed a.

p-fields

Each score event passes parameters to the instrument via numbered p-fields:

Inside the instrument, p4, p5, etc. read these values at i-rate.

Essential opcodes

Oscillators

asig  poscil  iamp, ifreq           ; precise sine (table interpolation)
asig  vco2    iamp, ifreq           ; anti-aliased oscillator (saw, square, etc.)
asig  oscil   iamp, ifreq, ifn      ; sine from a function table
asig  noise   iamp, ibandwidth      ; band-limited noise

Envelopes

aenv  linen   iamp, iatt, idur, idec      ; linear attack/sustain/decay
aenv  adsr    iatt, idec, isus, irel      ; ADSR (0..1, multiply by amp yourself)
aenv  madsr   iatt, idec, isus, irel      ; ADSR with note-off handling
kenv  linseg  0, 0.01, 1, 0.1, 0.7, p3-0.2, 0  ; arbitrary line segments

Filters

afilt  moogladder  asig, kcutoff, kresonance   ; Moog ladder filter
afilt  butterlp    asig, kcutoff               ; Butterworth low-pass
afilt  butterhp    asig, kcutoff               ; Butterworth high-pass
afilt  tone        asig, kcutoff               ; one-pole low-pass
afilt  bqrez       asig, kcutoff, kQ           ; biquad resonant filter

Effects

; Delay
adel  vdelay  asig, kdelaytime_ms, imaxdelay_ms

; Reverb
aL, aR  reverbsc  aL, aR, kfeedback, kcutoff

; Chorus / flanger
achorus  chorus  asig, krate, kdepth

; Distortion / waveshaping
adist  distort1  asig, kpregain, kpostgain, kshape1, kshape2

Synthesis techniques

; FM synthesis
amod  poscil  imodamp, imodfreq
acar  poscil  iamp, icarfreq + amod

; Subtractive (noise through filter)
anoise  noise 1, 0
afilt   moogladder anoise, 800, 0.5

; Additive (sum of partials)
a1    poscil  0.3, ifreq
a2    poscil  0.2, ifreq * 2
a3    poscil  0.1, ifreq * 3
aout  =  a1 + a2 + a3

MIDI input

; Read MIDI note and velocity
  imidi_note  notnum               ; MIDI note number (0–127)
  imidi_vel   veloc                ; velocity (0–127)
  ifreq       cpsmidi              ; convert note to Hz
  iamp        ampmidi 0dbfs * 0.5  ; scale velocity to amplitude

Real-time control

; Read a MIDI control change (CC)
kval  ctrl7  1, 7, 0, 1       ; channel 1, CC 7 (volume), range 0..1

; Read from a named software channel (set from host or API)
kval  chnget  "cutoff"

; Output to a channel
       chnset  kval, "output_level"

Running Csound

# Real-time audio output
csound -odac mypatch.csd

# Render to file
csound -o output.wav mypatch.csd

# Specify sample rate and bit depth
csound -odac -r 48000 --format=float mypatch.csd

# Real-time MIDI input
csound -odac -M0 mypatch.csd          # device 0
csound -odac -Ma mypatch.csd          # all MIDI devices

# Verbose output (useful for debugging)
csound -v mypatch.csd

CsoundQt (IDE)

CsoundQt provides a GUI with a code editor, widgets (knobs, sliders), and a real-time scope. Useful for interactive patches.

csoundqt

Cabbage

Cabbage wraps Csound instruments as VST/AU plugins, enabling use inside a DAW. Add a <Cabbage> section to the .csd for widget layout.

Csound in the browser (WebAssembly)

<script src="https://unpkg.com/@csound/browser/dist/csound.js"></script>
<script>
  const { Csound } = window;
  async function run() {
    const cs = await Csound();
    await cs.setOption("-odac");
    await cs.compileCsdText(`
      <CsoundSynthesizer>
      <CsInstruments>
        sr=44100
        ksmps=128
        nchnls=2
        0dbfs=1
        instr 1
          asig poscil 0.3, 440
          outs asig, asig
        endin
      </CsInstruments>
      <CsScore>
        i1 0 2
      </CsScore>
      </CsoundSynthesizer>
    `);
    await cs.start();
  }
  run();
</script>

Csound vs Pure Data

Resources

• Csound documentation
• The Canonical Csound Reference Manual
• Csound FLOSS Manual
• Csound on GitHub
• CsoundQt
• Cabbage

Faust

Faust (Functional Audio Stream) is a domain-specific language for audio signal processing developed at GRAME (Centre National de Création Musicale) in Lyon, starting around 2002. It occupies a unique position in the audio programming landscape: like Csound it is text-based and mathematically expressive, but like hvcc/Pure Data it compiles to self-contained C/C++ with no runtime - making it deployable on everything from web browsers to ARM Cortex-M4 microcontrollers.

The central idea is block diagram algebra: you describe a signal processor as a composition of simpler processors using a small set of operators, and the compiler works out the efficient C implementation automatically.

Installation

sudo apt install faust                  # Debian/Ubuntu
brew install faust                      # macOS

# Or get the latest with all tools
git clone https://github.com/grame-cncm/faust.git
cd faust && make && sudo make install

The online IDE requires no installation: faustide.grame.fr

Core concept: block diagram algebra

In Faust, everything is a block - a function that takes N input signals and produces M output signals. Blocks are composed with five operators:

The entire language is built from these five operators plus a set of primitives. There are no loops, no mutable state - feedback is expressed explicitly with ~.

Minimal example

process = _;   // identity: one input, one output (wire)
process = !;   // cut: discard input, produce nothing
process = 0;   // constant: produce the value 0

A sine oscillator at 440 Hz:

import("stdfaust.lib");

process = os.osc(440);

Stereo output (duplicate the mono signal):

import("stdfaust.lib");

process = os.osc(440) <: _, _;   // split to left and right

Primitives

Math

+, -, *, /          // arithmetic (infix: two inputs → one output)
%                   // modulo
^                   // power
&, |, xor          // bitwise
<<, >>              // bit shift
<, <=, >, >=, ==, !=  // comparison (output: 0 or 1)
abs, floor, ceil, round
sin, cos, tan, asin, acos, atan, atan2
exp, log, log10, sqrt, pow
min, max

Signal primitives

_           // identity (wire)
!           // cut (terminate a signal)
mem         // one-sample delay
@           // variable delay: x @ d  reads x delayed by d samples
rdtable     // read a table
rwtable     // read/write table
select2     // if/else: select2(cond, a, b)
select3     // three-way selector

Conversions

ba.samp2sec(n)      // samples to seconds
ba.sec2samp(s)      // seconds to samples
ma.SR               // current sample rate
ma.T                // sample period (1/SR)

UI elements

UI elements define parameters that the host (DAW, hardware, web app) can control. They produce a signal - a stream of values reflecting the current setting.

hslider("name", default, min, max, step)   // horizontal slider
vslider("name", default, min, max, step)   // vertical slider
nentry("name", default, min, max, step)    // numeric entry
button("name")                             // momentary (0 or 1)
checkbox("name")                           // toggle (0 or 1)

Grouping (for UI layout):

hgroup("Group Name", ...)    // horizontal group
vgroup("Group Name", ...)    // vertical group
tgroup("Tab Name", ...)      // tabbed group

Example - a tunable oscillator with volume:

import("stdfaust.lib");

freq = hslider("freq [unit:Hz]", 440, 20, 20000, 0.1);
gain = hslider("gain", 0.5, 0, 1, 0.01);

process = os.osc(freq) * gain <: _, _;

The standard library

stdfaust.lib imports all standard libraries. Key namespaces:

Common patterns

Envelope

import("stdfaust.lib");

gate   = button("gate");
attack = hslider("attack",  0.01, 0.001, 2, 0.001);
decay  = hslider("decay",   0.1,  0.001, 2, 0.001);
sustain= hslider("sustain", 0.8,  0,     1, 0.01);
release= hslider("release", 0.2,  0.001, 4, 0.001);

env    = en.adsr(attack, decay, sustain, release, gate);
process = os.osc(440) * env <: _, _;

Filter

import("stdfaust.lib");

cutoff = hslider("cutoff [unit:Hz]", 1000, 20, 20000, 1);
res    = hslider("resonance", 0.5, 0, 1, 0.01);

// Noise through a resonant low-pass filter
process = no.noise : fi.resonlp(cutoff, res, 1) <: _, _;

Feedback delay (recursive)

import("stdfaust.lib");

// feedback: output is delayed and added back to input
// ~ is the recursive operator
delay_samples = int(hslider("delay [unit:ms]", 250, 1, 2000, 1) * ma.SR / 1000);
feedback      = hslider("feedback", 0.5, 0, 0.99, 0.01);

echo = _ <: _, (@ (delay_samples) * feedback) :> _;
process = echo <: _, _;

FM synthesis

import("stdfaust.lib");

freq  = hslider("freq",  440, 20, 8000, 0.1);
ratio = hslider("ratio", 2,   0.1, 10,  0.01);
depth = hslider("depth", 200, 0,   5000, 1);

modulator = os.osc(freq * ratio) * depth;
carrier   = os.osc(freq + modulator);

process = carrier * 0.5 <: _, _;

Polyphony

Faust supports built-in polyphony via special parameter names. Declare freq, gain, and gate and compile with -nvoices N:

import("stdfaust.lib");

freq = hslider("freq",   440, 20, 20000, 0.1);
gain = hslider("gain",   0.5, 0,  1,     0.01);
gate = button("gate");

env  = en.adsr(0.01, 0.1, 0.8, 0.2, gate);
process = os.sawtooth(freq) * env * gain <: _, _;

faust2juce -nvoices 8 mysynth.dsp   # 8-voice polyphonic VST/AU

Compilation targets

Faust compiles via architecture files - wrappers that connect the generated DSP code to a specific host API. The faust2* scripts bundle generation + compilation:

Logue SDK (NTS-1 / minilogue xd / prologue)

Faust compiles directly to Logue oscillator units - no intermediate step required (unlike the Pd→hvcc→Logue pipeline):

# Install dependencies
sudo apt install faust gcc-arm-none-eabi

# Clone logue-sdk alongside your Faust file
git clone https://github.com/korginc/logue-sdk.git

# Compile for NTS-1 (nutekt-digital)
faust2logue -nvoices 1 myosc.dsp

# Output: myosc.ntkdigunit  (or .prlgunit / .mnlgxdunit)

The faust2logue script handles:

• ARM Cortex-M4 compilation (-mcpu=cortex-m4 -mthumb -mfpu=fpv4-sp-d16 -mfloat-abi=hard)
• Memory layout within the 32 KB SRAM constraint
• Mapping Faust UI sliders → Logue knob parameters
• Wrapping the DSP code in OSC_INIT, OSC_CYCLE, OSC_PARAM

Parameter mapping: Faust sliders named freq, gain, gate are automatically bound to the corresponding Logue signals; other sliders become assignable parameters.

import("stdfaust.lib");

// These names are special - bound automatically by faust2logue
freq = nentry("freq", 440, 20, 20000, 1);
gain = nentry("gain", 0.5, 0, 1, 0.01);

// Custom parameters appear as knobs in the Logue menu
shape = hslider("shape", 0, 0, 1, 0.01);

osc = os.sawtooth(freq) * (1 - shape) + os.osc(freq) * shape;

process = osc * gain;

Comparing Faust, Csound, and Pure Data

Faust and Csound share a mathematical, declarative approach - if you are comfortable with Csound’s signal flow thinking, Faust will feel familiar. The key difference is that Faust’s block diagram model is more restrictive (everything must be statically composable) but that restriction is what allows the compiler to produce efficient, allocation-free code.

Resources

• Faust website
• Faust online IDE
• Faust documentation
• Faust standard libraries
• Faust on GitHub
• faust2logue
• Faust examples

Logue SDK

The Korg logue SDK allows developing custom oscillators and effects for Korg’s logue-compatible hardware. Units compile to ARM Cortex-M4/M7 bare-metal code - see ARM for toolchain setup.

Hardware targets

Unit types

genericfx is NTS-3 Kaoss exclusive. Unlike other effect types, it can process audio continuously in the background regardless of touch pad state.

GenericFX - NTS-3 background effects

Normal effect units (modfx, delfx, revfx) receive audio via the in parameter in unit_render, which is routed through the effect on/off state. When the touch pad is not held (and HOLD/XY Freeze is off), the input is bypassed - the effect stops processing.

genericfx units can sidestep this by calling get_raw_input() through the runtime context, which returns the raw audio input unaffected by touch pad state. The effect runs continuously in the background.

Storing the runtime descriptor

Save the descriptor passed to unit_init - it carries the hooks needed later in unit_render:

#include "unit_genericfx.h"

static unit_runtime_desc_t s_runtime_desc;

__unit_callback int8_t unit_init(const unit_runtime_desc_t *desc) {
    if (!desc) return k_unit_err_undef;
    if (desc->target != unit_header.common.target) return k_unit_err_target;
    if (!UNIT_API_IS_COMPAT(desc->api)) return k_unit_err_api_version;

    s_runtime_desc = *desc;   // store for get_raw_input() access

    // optional: SDRAM allocation for delay buffers etc.
    if (s_effect_instance.getBufferSize() > 0) {
        float *buf = (float *)desc->hooks.sdram_alloc(
            s_effect_instance.getBufferSize() * sizeof(float));
        if (!buf) return k_unit_err_memory;
        s_effect_instance.init(buf);
    }
    return k_unit_err_none;
}

Background render

__unit_callback void unit_render(const float *in, float *out, uint32_t frames) {
    // Cast runtime context to genericfx type to access get_raw_input()
    const unit_runtime_genericfx_context_t *ctxt =
        static_cast<const unit_runtime_genericfx_context_t *>(
            s_runtime_desc.hooks.runtime_context);

    // get_raw_input() bypasses effect on/off routing - always live audio
    const float *raw_input = ctxt->get_raw_input();
    s_effect_instance.process(raw_input, out, frames);
}

Using raw_input instead of in means the effect processes audio continuously without requiring the HOLD button.

NTS-3 extras

genericfx also receives XY pad events via unit_touch_event, not available on other platforms:

__unit_callback void unit_touch_event(uint8_t id, uint8_t phase, uint32_t x, uint32_t y) {
    s_effect_instance.touchEvent(id, phase, x, y);
}

Memory available: 32 KB SRAM + 3 MB SDRAM (allocated via desc->hooks.sdram_alloc).

Template: logue-sdk/platform/nts-3_kaoss/dummy-genericfx/

Callback architecture

The SDK calls MIDI callbacks (unit_note_on, unit_note_off) from a different context than unit_render. Doing DSP state changes inside MIDI callbacks races against the audio thread.

Rule: MIDI callbacks only enqueue events. All DSP work happens in unit_render.

unit_note_on  → push to ring buffer only
unit_note_off → push to ring buffer only
unit_render   → drain ring buffer → process events → render audio

SPSC ring buffer pattern

enum NoteEventType : uint8_t { EVT_NOTE_ON = 0, EVT_NOTE_OFF = 1, EVT_ALL_NOTE_OFF = 2 };
struct NoteEvent { NoteEventType type; uint8_t note; uint8_t velocity; };

static constexpr int NOTE_QUEUE_SIZE = 16;
NoteEvent note_queue_[NOTE_QUEUE_SIZE];
std::atomic<uint8_t> queue_write_;  // producer: MIDI callbacks
std::atomic<uint8_t> queue_read_;   // consumer: Render()

// Producer (MIDI callback context)
inline void enqueue(NoteEventType type, uint8_t note, uint8_t vel) {
    uint8_t w    = queue_write_.load(std::memory_order_relaxed);
    uint8_t next = (w + 1) % NOTE_QUEUE_SIZE;
    if (next != queue_read_.load(std::memory_order_acquire)) {
        note_queue_[w] = {type, note, vel};
        queue_write_.store(next, std::memory_order_release);
    }
}

// Consumer (render context)
inline void drainNoteQueue() {
    uint8_t w = queue_write_.load(std::memory_order_acquire);
    uint8_t r = queue_read_.load(std::memory_order_relaxed);
    while (r != w) {
        NoteEvent evt = note_queue_[r];
        queue_read_.store((r + 1) % NOTE_QUEUE_SIZE, std::memory_order_release);
        switch (evt.type) {
            case EVT_NOTE_ON:      processNoteOn(evt.note, evt.velocity); break;
            case EVT_NOTE_OFF:     processNoteOff(evt.note);              break;
            case EVT_ALL_NOTE_OFF: processAllNoteOff();                   break;
        }
        r = queue_read_.load(std::memory_order_relaxed);
        w = queue_write_.load(std::memory_order_acquire);
    }
}

fast_inline void Render(float* out, size_t frames) {
    drainNoteQueue();   // always first
    for (size_t i = 0; i < frames; i++) {
        // DSP processing
    }
}

VLA pitfall

unit_render receives frames as a runtime value. Stack-allocating with a runtime size is a C99 VLA - not valid C++ and dangerous on Cortex-M with limited stack:

// WRONG: VLA, not valid C++
float buf[frames * 2];

// CORRECT: fixed max (NTS-1 mkII block size is always ≤ 256 frames)
float buf[256 * 2];

Minimal unit structure

#include "unit.h"   // from logue-sdk platform headers

static UnitHeader unit_header = {
    .header_size  = sizeof(UnitHeader),
    .target       = UNIT_TARGET_PLATFORM | k_unit_module_osc,
    .api          = UNIT_API_VERSION,
    .dev_id       = 0x00000000,
    .unit_id      = 0x00000000,
    .version      = 0x00010000,
    .name         = "MyUnit",
    .num_params   = 2,
    .params       = {{ "Param1", 0, 0, 100, k_unit_param_type_none },
                     { "Param2", 0, 0, 100, k_unit_param_type_none }},
};

__unit_callback void unit_init(const UnitRuntimeDesc* desc) { }
__unit_callback void unit_teardown() { }
__unit_callback void unit_reset() { }
__unit_callback void unit_resume() { }
__unit_callback void unit_suspend() { }
__unit_callback void unit_note_on(uint8_t note, uint8_t vel) { /* enqueue only */ }
__unit_callback void unit_note_off(uint8_t note) { /* enqueue only */ }
__unit_callback void unit_all_note_off() { /* enqueue only */ }
__unit_callback void unit_set_param_value(uint8_t id, int32_t val) { }
__unit_callback void unit_render(float* in, float* out, uint32_t frames) {
    drainNoteQueue();
    // process audio
}

Building

# From an SDK platform directory, e.g. nts-3_kaoss/
cd ~/Projects/logue-sdk/platform/nts-3_kaoss/myunit
make

# Output: myunit.nts3unit
# Install via logue-tool or USB drag-and-drop

Toolchain: arm-none-eabi- for Cortex-M targets, arm-linux-gnueabihf- for drumlogue. See ARM for installation.

Deployment paths

The same DSP core can target multiple runtimes beyond bare-metal hardware:

The Regroovelizer loads .nts3unit ELF files on ARM Linux by parsing relocations and resolving logue SDK symbols at runtime - allowing the same unit binary to run on desktop for development before flashing to hardware.

Resources

• Korg logue SDK
• logue-tool - CLI for flashing units
• Faust → Logue - faust2logue
• See ARM for cross-compilation toolchain setup
• See Faust for generating Logue units from Faust DSP code

Drumlogue Units

The Korg drumlogue runs a full ARM Cortex-A7 Linux system, which makes it fundamentally different from the Cortex-M targets (NTS-1, minilogue xd). Custom synth units are shared libraries (.drmlgunit) loaded at runtime, with access to the full C standard library, heap allocation, and libm. The toolchain is arm-linux-gnueabihf- rather than arm-none-eabi-.

See Logue SDK for a general overview of the SDK and other platforms.

Unit type

Drumlogue units use PROJECT_TYPE := synth and target k_unit_module_synth. Unlike the Cortex-M platforms, which separate oscillators (osc), delay effects, and reverb effects into distinct module types, the drumlogue has a single synth slot type that handles everything: sound generation, internal state, and MIDI response.

# config.mk
PROJECT := my_drum
PROJECT_TYPE := synth

// header.c
const __unit_header unit_header_t unit_header = {
    .target  = UNIT_TARGET_PLATFORM | k_unit_module_synth,
    .api     = UNIT_API_VERSION,
    .dev_id  = 0x4D593130U,   /* 'MY10' — unique per unit */
    .unit_id = 0x01U,
    .version = 0x00010000U,
    .name    = "MyDrum",
    .num_presets = 0,
    .num_params  = 8,
    .params = { ... }
};

File layout

A minimal drumlogue unit project contains five files:

my_drum/
├── Makefile      — copy from platform/drumlogue/dummy-synth/; references common/ and config.mk
├── config.mk     — PROJECT, PROJECT_TYPE, CSRC, CXXSRC, UINCDIR, ULIBS
├── header.c      — unit_header_t: name, params, dev_id
├── synth.h       — Synth class: Init, Render, NoteOn, GateOn, setParameter, …
└── unit.cc       — SDK callbacks delegating to Synth class instance

unit.cc is boilerplate that barely changes between projects. Copy it from the SDK reference unit (platform/drumlogue/dummy-synth/unit.cc) and it delegates every SDK callback to the Synth class defined in synth.h.

Parameters

The drumlogue supports up to 24 parameters in 6 pages of 4. Only num_params of those are active; the rest must be listed as k_unit_param_type_none placeholders to fill the array.

.num_params = 8,
.params = {
    /* Page 1 */
    {0, 100, 0, 80, k_unit_param_type_percent, 0, 0, 0, {"LEVEL"}},
    {0, 5,   0, 0,  k_unit_param_type_none,    0, 0, 0, {"PRESET"}},
    {0, 0,   0, 0,  k_unit_param_type_none,    0, 0, 0, {""}},
    {0, 0,   0, 0,  k_unit_param_type_none,    0, 0, 0, {""}},
    /* Page 2 */
    ...
    /* Pages 3–6: all none placeholders */
}

getParameterStrValue can return label strings for enum-style parameters (preset names, drum names, mode names). Return nullptr for percent-type parameters where the runtime displays the integer directly.

Gate vs Note

This is the key architectural distinction for the drumlogue.

The drumlogue’s front panel pads and step sequencer drive the internal analog drum parts (BD, SD, LT, HT, CY, HH). When a custom synth unit occupies one of the four SYN slots, the sequencer fires that slot with a gate signal — no note number, just velocity. External MIDI can send full note-on messages to the synth slot via MIDI channel assignment.

Consequences for drum machine units:

• GateOn(velocity) should trigger a single, configurable drum sound
• NoteOn(note, velocity) can implement full GM note mapping for external MIDI
• A GATE parameter (0–N) lets the user select which drum the sequencer fires

// In synth.h
inline void GateOn(uint8_t velocity) {
    // trigger whatever drum PARAM_GATE selects
    Voice* voices[] = { &kick_, &snare_, &ch_, &oh_, &clap_, &tom_ };
    voice_note_on(voices[gate_drum_], 60, velocity);
}

inline void NoteOn(uint8_t note, uint8_t velocity) {
    // GM drum map for external MIDI
    switch (note) {
    case 36: voice_note_on(&kick_,  60, velocity); break;
    case 38: voice_note_on(&snare_, 60, velocity); break;
    case 39: voice_note_on(&clap_,  60, velocity); break;
    case 42: voice_note_on(&ch_,    60, velocity); break;
    case 46: voice_note_on(&oh_,    60, velocity); break;
    case 50: voice_note_on(&tom_,   60, velocity); break;
    }
}

Memory model

Because drumlogue runs Linux, malloc/free and C++ new/delete are fully available. Heap-allocated engines per voice are normal:

MyEngine* engine_ = nullptr;

inline int8_t Init(const unit_runtime_desc_t* desc) {
    engine_ = my_engine_create((float)desc->samplerate);
    if (!engine_) return k_unit_err_memory;
    ...
}

inline void Teardown() {
    my_engine_destroy(engine_);
    engine_ = nullptr;
}

VLAs inside Render() are available (GCC extension) and used for temporary audio buffers, but a fixed-size array is safer if the block size is known:

fast_inline void Render(float* out, size_t frames) {
    float mix[frames], l[frames], r[frames];   // VLA, GCC only — fine on Linux
    ...
}

The Cortex-M SPSC ring buffer pattern for MIDI/render thread safety (see Logue SDK) is not required on drumlogue — MIDI callbacks and unit_render run in the same Linux process context. Direct state updates in NoteOn are safe.

Render output format

unit_render receives a stereo interleaved float* buffer:

__unit_callback void unit_render(const float* in, float* out, uint32_t frames) {
    for (uint32_t i = 0; i < frames; i++) {
        float s = generate_sample();
        out[i * 2]     = s;  // left
        out[i * 2 + 1] = s;  // right
    }
}

Engines that output split left/right buffers (not interleaved) need to be mixed voices into a mono accumulator, then write to both channels:

float mix[frames], l[frames], r[frames];
engine_process(voice_, l, r, (int)frames);
for (size_t i = 0; i < frames; i++) mix[i] += l[i] * level_;
// ... accumulate other voices ...
for (size_t i = 0; i < frames; i++) {
    out[i * 2] = out[i * 2 + 1] = mix[i];
}

Building

Units are cross-compiled inside a container using the logue SDK development environment. Place the unit directory inside platform/drumlogue/, then use the SDK’s run_cmd.sh script:

# From the logue-sdk root
./docker/run_cmd.sh build drumlogue/my_drum

This mounts platform/ as /workspace inside the container, sources the drumlogue environment, and runs make install. The resulting my_drum.drmlgunit is written back into platform/drumlogue/my_drum/.

For units that live outside the SDK tree (e.g. a separate project repo), mount the unit directory and any additional source paths manually:

podman run --rm \
    -v "$PLATFORM_PATH:/workspace:rw" \
    -v "$PROJECT_DIR:/workspace/drumlogue/my_drum:rw" \
    ghcr.io/gbraad-logue/logue-sdk/dev-env:latest \
    /app/cmd_entry bash -c "
        source /app/drumlogue/environment &&
        cd /workspace/drumlogue/my_drum &&
        make clean &&
        make install
    "

Any extra source paths (shared libraries, external engines) are passed as make variable overrides and mounted as additional read-only volumes:

-v "$ENGINE_SRC:/engine:ro"
# then in make:
make install ENGINE_PATH=/engine

Loading onto the device

Power on the drumlogue while holding the WRITE button to enter USB mass storage mode. The device mounts as a drive with a Units/Synths/ directory. Copy .drmlgunit files there, then restart the device. Units load in alphabetical order and appear in the SYN1–4 slot selection.

Filesystem access

Because the drumlogue runs Linux, units have full POSIX file I/O via the standard C library (fopen, fread, fwrite, opendir, etc.). Three distinct mechanisms are available depending on what you need to access.

User filesystem (userfs)

The drumlogue daemon exposes a persistent writable area at:

/var/lib/drumlogued/userfs/

This is where units read and write user data: sample files, presets, configuration. It survives reboots and is accessible from USB mass storage as a subdirectory of the mounted drive. On MicroKorg2 the equivalent base is /var/lib/microkorgd/userfs/.

Namespace your unit’s files under a subdirectory to avoid collisions with other units:

char path[256];
snprintf(path, sizeof(path), "/var/lib/drumlogued/userfs/MyUnit/preset_%d.dat", idx);
FILE* f = fopen(path, "rb");
if (f) {
    /* read data */
    fclose(f);
}

For units that target both drumlogue and MicroKorg2, UNIT_TARGET_PLATFORM is a C enum value, not a preprocessor macro, so it cannot be used in #if expressions. Pass a -D flag via UDEFS in config.mk and test with #if defined(...):

# config.mk (drumlogue)
UDEFS = -DUNIT_TARGET_PLATFORM_DRUMLOGUE

# config.mk (MicroKorg2)
UDEFS = -DUNIT_TARGET_PLATFORM_MICROKORG2

/* paths.h */
#if defined(UNIT_TARGET_PLATFORM_DRUMLOGUE)
#  define MYUNIT_USERFS "/var/lib/drumlogued/userfs/MyUnit"
#elif defined(UNIT_TARGET_PLATFORM_MICROKORG2)
#  define MYUNIT_USERFS "/var/lib/microkorgd/userfs/MyUnit"
#endif

SDK sample bank API

unit_runtime_desc_t (passed to unit_init) carries three function pointers for accessing the drumlogue’s built-in sample library without touching the filesystem directly:

typedef struct unit_runtime_desc {
    // ...
    unit_runtime_get_num_sample_banks_ptr     get_num_sample_banks;
    unit_runtime_get_num_samples_for_bank_ptr get_num_samples_for_bank;
    unit_runtime_get_sample_ptr               get_sample;
} unit_runtime_desc_t;

get_sample(bank, index) returns a sample_wrapper_t*:

typedef struct sample_wrapper {
    uint8_t      bank, index, channels, _padding;
    char         name[32];
    size_t       frames;
    const float* sample_ptr;   /* PCM already in memory, ready to use */
} sample_wrapper_t;

The sample data is pre-loaded into memory by the runtime; sample_ptr points directly to 32-bit float PCM. Store desc at Init time and call get_sample later as needed:

const unit_runtime_desc_t* runtime_desc_ = nullptr;

inline int8_t Init(const unit_runtime_desc_t* desc) {
    runtime_desc_ = desc;
    return k_unit_err_none;
}

bool loadSample(uint8_t bank, uint8_t index, const float** data, size_t* frames) {
    const sample_wrapper_t* s = runtime_desc_->get_sample(bank, index);
    if (!s || !s->sample_ptr) return false;
    *data   = s->sample_ptr;
    *frames = s->frames;
    return true;
}

Embedded data (cart injection)

For data that ships with the unit itself (cartridges, wavetables, sample banks) the cart injection pattern embeds the data in a named ELF section inside the .drmlgunit binary. The runtime loads the binary as a shared library, so the section is mapped directly into the process address space with no file I/O.

/* declare a replaceable block inside the unit binary */
__attribute__((section(".my_cart")))
const uint8_t my_cart_data[4096] = { /* placeholder bytes */ };

A host-side patcher reads the ELF, locates the section by name, and overwrites its contents without recompiling. The unit accesses the array as a normal C pointer at runtime.

Unit patterns

Single-voice drum

One synth engine per unit. The unit produces a single instrument sound. GateOn triggers it; NoteOn can optionally vary pitch. Typical parameter set: pitch, decay, level, tone.

inline void GateOn(uint8_t velocity) {
    engine_trigger(&voice_, 60, velocity);
}

inline void NoteOn(uint8_t note, uint8_t velocity) {
    engine_trigger(&voice_, note, velocity);
}

Multi-voice drum machine

Six independent voices: kick, snare, closed hat, open hat, clap, tom. Each voice has a level parameter. NoteOn uses GM note mapping; GateOn fires whichever voice PARAM_GATE selects.

PARAM_KICK_LEVEL  PARAM_SNARE_LEVEL  PARAM_CH_LEVEL  PARAM_OH_LEVEL
PARAM_CLAP_LEVEL  PARAM_TOM_LEVEL    PARAM_MASTER    PARAM_GATE (0-5)

inline void NoteOn(uint8_t note, uint8_t velocity) {
    switch (note) {
    case 36: voice_trigger(&kick_,  velocity); break;
    case 38: voice_trigger(&snare_, velocity); break;
    case 39: voice_trigger(&clap_,  velocity); break;
    case 42: voice_trigger(&ch_,    velocity); break;
    case 46: voice_trigger(&oh_,    velocity); break;
    case 50: voice_trigger(&tom_,   velocity); break;
    }
}

inline void GateOn(uint8_t velocity) {
    void* voices[] = { &kick_, &snare_, &ch_, &oh_, &clap_, &tom_ };
    voice_trigger(voices[gate_drum_], velocity);
}

Polyphonic synth with file-loaded presets

One polyphonic engine (4-8 voices internally). NoteOn/NoteOff handle melody; GateOn triggers at a fixed pitch. PARAM_PRESET selects from N preset slots.

Each slot loads a preset file at Init; if the file is absent a built-in preset is used as fallback. The preset name shown on the display comes from a name field embedded in the file, falling back to the factory name.

/var/lib/drumlogued/userfs/MyUnit/preset_0.dat  ->  slot 0
/var/lib/drumlogued/userfs/MyUnit/preset_1.dat  ->  slot 1
...

void loadPresets() {
    for (int i = 0; i < NUM_PRESETS; i++) {
        char path[256];
        snprintf(path, sizeof(path),
                 "/var/lib/drumlogued/userfs/MyUnit/preset_%d.dat", i);
        bool loaded = false;
        FILE* f = fopen(path, "rb");
        if (f) {
            uint8_t buf[PRESET_SIZE];
            if (fread(buf, 1, PRESET_SIZE, f) == PRESET_SIZE)
                loaded = preset_deserialize(&presets_[i], buf);
            fclose(f);
        }
        if (!loaded)
            presets_[i] = factory_preset(i);
    }
}

getParameterStrValue returns presets_[value].name for the preset selector so the display shows the embedded name rather than a raw number.

Ecosystem examples

Several third-party ports demonstrate what the drumlogue runtime can host:

These show that the Linux runtime allows porting substantial audio codebases without the severe size and performance constraints of bare-metal Cortex-M.

Resources

• Korg logue SDK — drumlogue platform
• Logue SDK overview
• ARM cross-compilation toolchain

Schwung - Custom Modules for Ableton Move

Schwung (formerly Move Everything) is an unofficial framework for running custom instruments, effects, and controllers on the Ableton Move hardware. It adds a Shadow UI that runs alongside stock Move firmware, allowing additional synths, FX, and tools to run in parallel with the normal Move interface.

Disclaimer: Not endorsed or supported by Ableton. Modifying Move firmware carries risk - back up your sets and know how to use DFU restore mode before installing.

How it works

Schwung injects itself via LD_PRELOAD. The schwung-shim.so library intercepts the Move process and launches the Schwung host, which manages the Shadow UI and loads modules. Move itself continues to run normally.

Access the Shadow UI with Shift+Vol+Track (and +Menu). Overtake modules use Shift+Vol+Jog click.

A web interface is available at schwung.local for managing modules, files, and settings.

Installation

# Via curl (Move must be on same WiFi network)
curl -L https://raw.githubusercontent.com/charlesvestal/schwung/main/scripts/install.sh | sh

# Or use the desktop installer (macOS/Windows/Linux GUI)
# https://github.com/charlesvestal/schwung-installer/releases/latest

Uninstall:

curl -L https://raw.githubusercontent.com/charlesvestal/schwung/main/scripts/uninstall.sh | sh

Modes

Module types

Hardware target

Ableton Move runs on ARM64 (aarch64) Linux with glibc. All modules must be cross-compiled for this target.

• Audio: 44.1 kHz, 128-sample blocks (~3ms latency)
• Modules loaded via dlopen() as shared .so files

Plugin API v2

All new modules use API v2, which supports multiple instances and is required for Signal Chain integration.

#include "host/plugin_api_v2.h"

typedef struct plugin_api_v2 {
    uint32_t api_version;    // must be 2
    void* (*create_instance)(const char *module_dir, const char *json_defaults);
    void  (*destroy_instance)(void *instance);
    void  (*on_midi)(void *instance, const uint8_t *msg, int len, int source);
    void  (*set_param)(void *instance, const char *key, const char *val);
    int   (*get_param)(void *instance, const char *key, char *buf, int buf_len);
    void  (*render_block)(void *instance, int16_t *out_lr, int frames);
} plugin_api_v2_t;

// Export this from your shared library
extern "C" plugin_api_v2_t* move_plugin_init_v2(const host_api_v1_t *host);

render_block is called every ~3ms with 128 stereo int16 frames. set_param / get_param pass string key-value pairs for parameter communication with the JS UI.

Building a module

Cross-compilation for ARM64 is done via Docker. Each module has a scripts/build.sh:

./scripts/build.sh       # Docker cross-compile (macOS/Windows/Linux)
./scripts/install.sh     # Deploy to Move over SSH

Manual cross-compile (Ubuntu/Debian):

# Install toolchain
sudo apt install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu

# Compile shared library
aarch64-linux-gnu-g++ -O3 -shared -fPIC \
    src/dsp/my_plugin.cpp \
    -o build/modules/my_module/dsp.so \
    -lm

# Deploy
scp build/modules/my_module/dsp.so ableton@move.local:~/schwung/modules/audio_fx/my_module/

module.json

Every module has a src/module.json that defines its identity and parameters:

{
  "id": "my_effect",
  "name": "My Effect",
  "version": "0.1.0",
  "component_type": "audio_fx",
  "parameters": [
    { "key": "mix",    "name": "Mix",    "min": 0, "max": 100, "default": 50 },
    { "key": "gain",   "name": "Gain",   "min": 0, "max": 200, "default": 100 }
  ]
}

component_type determines the install path. Parameters defined here are exposed automatically to the Shadow UI knob pages.

Standalone modules

Standalone modules take full control of Move hardware - Move firmware is suspended while the module runs, then restarted on exit. They talk directly to the SPI interface:

// Open the SPI device
int fd = open("/dev/ablspi0.0", O_RDWR);
void *buf = mmap(NULL, 0x1000, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);

// Main loop: ~2.9ms per tick at 44.1kHz/128 frames
while (running) {
    // Write MIDI out, display slice, audio out to buf[0..767]
    // Read MIDI in, audio in from buf[2048..2815]
    ioctl(fd, 0xa, 0x300);   // blocking SPI transfer
}

SPI buffer layout:

Display: 128×64 pixels, 1-bit, packed as 8 vertical bands. Each byte encodes 8 vertical pixels (bit 0 = top). Sent in 6 slices of 172 bytes.

See schwung-standalone-example for a working template.

Releasing a module

# 1. Bump version in src/module.json
# 2. Commit + tag
git tag v0.2.0
git push origin main --tags
# GitHub Actions builds, creates release, updates release.json automatically

The Module Store reads release.json from each module’s repo to discover new versions. The download_url points to the release tarball on GitHub.

Notable modules

Related projects

• schwung - Main framework
• schwung-installer - Desktop installer (macOS/Windows/Linux)
• schwung-standalone-example - Standalone module template
• move-anything-control - Base techniques Schwung builds on
• Discord: https://discord.gg/GHWaZCC9bQ

Networking

Notes on network tools, proxies, and protocols.

• nginx - web server and reverse proxy
• traefik - modern reverse proxy and load balancer
• tcpdump - packet capture and analysis
• SSH - secure shell, tunneling, key management
• rsync - file synchronization
• curl/wget - HTTP clients

NGINX

Usage

Standard container

$ docker run --name some-nginx -v /some/content:/usr/share/nginx/html:ro -d nginx

nginx-proxy

$ docker run -d -p 80:80 -p 443:443 \
   --name nginx-proxy \
   -v /path/to/certs:/etc/nginx/certs:ro \
   -v /etc/nginx/vhost.d \
   -v /usr/share/nginx/html \
   -v /var/run/docker.sock:/tmp/docker.sock:ro \
   jwilder/nginx-proxy

Source

Let’s Encrypt companion for nginx-proxy

$ docker run -d \
   --name nginx-proxy-letsencrypt \
   -v /path/to/certs:/etc/nginx/certs:rw \
   --volumes-from nginx-proxy \
   -v /var/run/docker.sock:/var/run/docker.sock:ro \
   jrcs/letsencrypt-nginx-proxy-companion

Source

Example of deployment

Setting up a powerful self-hosted IDE in the cloud

Traefik

• https://github.com/containous/traefik

Usage

$ ./traefik --configFile=traefik.toml

or using Docker

$ docker run -d -p 8080:8080 -p 80:80 -v $PWD/traefik.toml:/etc/traefik/traefik.toml traefik

Configuration sample

tcpdump

Filter for DHCP/BOOTP

tcpdump -i any -vvv -s 1500 '(port 67 or port 68)'

tcpdump -i any -vvv -s 1500 '((port 67 or port 68) and (udp[8:1] = 0x1))'

Secure shell

• Hardening practices by Mozilla

rsync

Sync remote to local

$ rsync --progress [username]@[hostname]:[path]/[file(s)] [local path]

Note: --progress or -P show progress bar

Resume partial download

If a transfer failed using scp, you can resume using rsync

rsync --partial --progress --rsh=ssh [username]@[hostname]:[path][/file(s)] [local path] 

Note: using zsh you might need to use " around the path specifiers (due to globbing)

Curl and wget

Download files from website path

$ wget \
     --recursive \
     --no-clobber \
     --page-requisites \
     --html-extension \
     --convert-links \
     --restrict-file-names=windows \
     --domains website.org \
     --no-parent \
         www.website.org/files/

Silent mode

curl

$ curl --silent 

wget

$ wget --quiet

Security

Notes on security tools and practices.

• Security - general security tools and hardening

Security

• Secure shell

Modes of persuasion

• Logos
  what you are saying
• Pathos how excited you are
• Ethos
  who (they think) you are

Wiki

Quotes

“What doesn’t go upstream is a liability!”

“Do good things and talk about it”

“Every user is a potential contributor”

“Putting the pieces together”

“When you’re dying, no one looks back and says, ‘My God, I wish I’d spent more time at the office.’”

“Software is the hidden writing that whispers the stories of possibility to our hardware”, and we are the Storytellers. - Grady Booch

“Fail faster to succeed sooner”

“Being a kid has nothing to do with age”

“Ecosystem is more important than the technology”

“I’m going to get exactly what I want. And I’m not going to stop until I get it.”

“In its essence the attraction of hacking is the joy of coming to understand a complex, dynamic, semi-autonomous system, and then bending it to your will.”

“Failure is always an option” - Adam Savage

“Making a mistake is human. Failing is God-like!” - I often use this to acquint users to contribute ;-)

“If someone offers you an amazing opportunity and you’re not sure you can do it, say yes – then learn how to do it later.” - Richard Branson

“Ceaseless change is the only constant thing in Nature.” - John Candee Dean

“To succeed in bringing a person to a certain place, you need to start by finding out where is in the first place.”

“If your heart is not right, no one cares about your skills”

“Writing is like driving at night in the fog. You can only see as far as your headlights, but you can make the whole trip that way.” - E.L. Doctorow

“It’s a crazy world out there. Be curious.” - Stephen Hawking

“Set goals, not limits”

“The best way to predict the future is to create it”

“If you don’t have any shadows, you’re not standing in the light” - Marissa Mayer

“Do not change to fit how the tool works, but change the tool to fit how you work.”

Todo

Add cheatsheet and instructions for:

• tmux
• screen
• irssi

IRSSI

Commands

To Connect to Server:

/connect irc.freenode.net

To Connect to Channel:

/join #fedora

To leave Channel:

/part #fedora

To Set nick:

/set nick username

To Identify nick:

/msg nickserv identify password

To Change nick:

/nick username1

Open edX

Deployment

• https://github.com/gbraad/deploy-openedx
• https://openedx.atlassian.net/wiki/display/OpenOPS/Open+edX+Installation+Options

In a few steps

1. Update and upgrade Ubuntu 14.04

sudo apt-get update
sudo apt-get upgrade

2. Install dependencies

sudo apt-get install -y build-essential software-properties-common python-software-properties curl git-core libxml2-dev libxslt1-dev python-pip python-apt python-dev  libxmlsec1-dev swig
sudo pip install --upgrade pip
sudo pip install --upgrade virtualenv

3. Clone the configuration repository

$ cd /var/tmp
$ git clone -b release https://github.com/edx/configuration

4. Allow password-based SSH authentication

configuration/playbooks/roles/common/defaults/main.yml

COMMON_SSH_PASSWORD_AUTH="yes"

5. Install requirements

$ cd /var/tmp/configuration
$ sudo pip install -r requirements.txt

6. Run the edx_sandbox.yml playbook in the configuration/playbooks directory

$ cd /var/tmp/configuration/playbooks && sudo ansible-playbook -c local ./edx_sandbox.yml -i "localhost,"

7. Open edX

http://localhost

Troubleshooting

no alternatives for libblas.so.3gf

TASK: [edxapp | code sandbox | Use libblas for 3gf] *************************** 
failed: [localhost] => {"changed": true, "cmd": ["update-alternatives", "--set", "libblas.so.3gf", "/usr/lib/libblas/libblas.so.3gf"], "delta": "0:00:00.002302", "end": "2016-09-18 09:26:37.399477", "rc": 2, "start": "2016-09-18 09:26:37.397175", "warnings": []}
stderr: update-alternatives: error: no alternatives for libblas.so.3gf

FATAL: all hosts have already failed -- aborting

cd /edx/app/edx_ansible/edx_ansible/

diff --git a/playbooks/roles/edxapp/tasks/python_sandbox_env.yml b/playbooks/roles/edxapp/tasks/python_sandbox_env.yml
index 1793cb5..08070b3 100644
--- a/playbooks/roles/edxapp/tasks/python_sandbox_env.yml
+++ b/playbooks/roles/edxapp/tasks/python_sandbox_env.yml
@@ -2,11 +2,11 @@
 # MITx 6.341x course.
 # TODO: Switch to using alternatives module in 1.6
 - name: code sandbox | Use libblas for 3gf
-  command: update-alternatives --set libblas.so.3gf /usr/lib/libblas/libblas.so.3gf
+  command: update-alternatives --set libblas.so.3 /usr/lib/libblas/libblas.so.3
 
 # TODO: Switch to using alternatives module in 1.6
 - name: code sandbox | Use liblapac for 3gf
-  command: update-alternatives --set liblapack.so.3gf /usr/lib/lapack/liblapack.so.3gf
+  command: update-alternatives --set liblapack.so.3 /usr/lib/lapack/liblapack.so.3
 
 - name: code sandbox | Create edxapp sandbox user
   user: name={{ edxapp_sandbox_user }} shell=/bin/false home={{ edxapp_sandbox_venv_dir }}

$ vi /edx/app/edx_ansible/edx_ansible/playbooks/roles/edxapp/tasks/python_sandbox_env.yml
# change .3gf to .3
$ ansible-playbook -i localhost, -c local vagrant-fullstack.yml -e@$ANSIBLE_ROOT/server-vars.yml -e@$ANSIBLE_ROOT/extra-vars.yml --skip-tags install:code

Windows

Essentials

• Putty
• Cygwin
• Conemu
• Process Explorer

Books

• Reviews I post on my blog can be found here

Technology

• The Architecture of Open Source Applications
  aosabook.org

Management and Leadership

• The Open Organization
• The Secret: What Great Leaders Know - And Do
• I Moved Your Cheese
• Chess Not Checkers
• Let’s stop meeting like this
• Remote: Office Not Required
• Singletasking: Get More Done-One Thing at a Time
• Personal Kanban: Mapping Work | Navigating Life

Software

• Clean code
• Object Design
• The Pragmatic Programmer: From Journeyman to Master

Chess Not Checkers

Title: Chess Not Checkers Author: Mark Miller ISBN: 978-1-62656-396-4

Chess Not Checkers

Notes

1. Bet on Leadership
   Growing leaders grow organizations
2. Act as One
   Alignment multiplies impact
3. Win the Heart
   Engagement energizes effort
4. Excel at Execution
   Greatness hinges on execution

Let's stop meeting like this

Title: Let’s stop meeting like this Author: Dick and Emily Axelrod ISBN: 978-1-62656-085-7

Title

Meeting Canoe

• Welcome
• Connect opeople to each other and the task
• Discover the way things are
• Elicit people’s dreams
• Decide
• Attend to the end

Chinese: Technology

English

Introduction

Essential reads to learn (and teach) English are:

• Oxford Guide to English Grammar, by John Eastwood
  Goodreads
• Practical English Usage, by Michael Swan
  Goodreads