Cloning a Raspberry Pi SD card

The goal is to clone the SD card of a Raspberry Pi after setup and installation, perhaps for backup or to make more copies of the same setup. Copying the SD card as an image file (.img) onto a regular machine is simple - just install Win32DiskImager (Windows) or Etcher (Windows/Mac) and click through the prompt. The tricky part is writing the .img file back into other SD cards.

All SD cards are not made equal. Not for cards that claim to have the same capacity. Not even for cards of the same stated capacity that come from the same Amazon order. Off by one byte and Win32DiskImager / Etcher would complain about the card being too small to hold the image. The simple solution is to use a bigger card. Card sizes grow by a factor of two, so you'd need a 32GB card to hold the image of a 16GB card (pretty much the smallest size you can find in Q4 2017). It works, but how'd you backup your 32GB card?

Another way to get around this is to resize the .img file. Usually very little space is actually used on a newly prepared Pi (a 2017 Raspbian Lite image with RabbitMQ, pika, numpy, etc. all installed for both Python 2 and 3 takes only 1.6GB of space), so most of the card is actually empty and can be trimmed from the image file. The process goes roughly as follow: boot a machine using a GParted / Ubuntu Live image, mount the storage device (e.g. USB stick) holding the .img file, mount the .img file as a loopback device, resize the file system using GParted, unmount the loopback device, then truncate the .img file.

First boot into the live image (if you are using the Ubuntu Live image, make sure you don't accidentally install it). You will also need to mess with the boot order of your machine at least temporarily. There's extensive resource elsewhere for this. Once in, mount the USB drive holding the .img file (if it isn't automatically mounted; use lsblk to find out what the device name is). The partition on my USB drive shows up as /dev/sdb1. My .img file is named node-080.img):

sudo mkdir /media/usb0
sudo mount /dev/sdb1 /media/usb0

Now mount the .img file as loopback device:

sudo losetup -f (take note of its output; mine is /dev/loop1)
sudo losetup -P /dev/loop1 /media/usb0/node-080.img

Launch GParted:

sudo gparted /dev/loop1

Resize the second partition (the first one is the /boot partition, which is tiny) to the content, leaving ~200MB of empty space at the end. Commit and exit.

Unmount the loopback device:

sudo -d /dev/loop1

Now truncate it, but first find out where the last sector of the partition is:

sudo fdisk -lu /media/usb0/node-080.img

Take note the sector size (512 bytes for me) and End sector. Then truncate:

truncate --size=$[(END_SECTOR+1)*512] /media/usb0/node-080.img

Verify:

ls -al

Missing stack trace in Python Flask on CentOS

To redirect Flask app traceback (exception, logging.error() etc.) to Apache's error log, add

import app
app.logger.addHandler(logging.StreamHandler(sys.stderr))

Otherwise all I get was a "500 Internal Server Error" with no debug information anywhere in the server.

For some strange reason, with Apache 2.4 on CentOS 7 using mod_wsgi and Python Flask, traceback actually goes to /var/log/httpd/ssl_error_log instead of /var/log/apache2/error.log on Ubuntu Server (even with SSL).

There are some other strange things with CentOS too, like the site would be accessible via http, but when https is used it would throw "Internal Server Error" unless the permissions of the folder that hold the Flask code are changed to allow R and X for user apache. It makes sense that Apache requires R and X, but that doesn't explain why it would work for http but not https.

Python3 matplotlib on RPi3 Raspbian

After sudo pip3 install matplotlib, my Pi complains about the missing cairo package and some other stuff. This fixed it:

sudo apt install -y python3-gi-cairo python3-cairocffi

Using the Beaglebone Serial Ports

The Beaglebone Black has four (five if you count the incomplete port 3; six if you count the debug header) serial ports. It used to be possible to enable them by adding the line

cape_enable=bone_capemgr.enable_partno=BB-UART1,BB-UART2,BB-UART4,BB-UART5

in /boot/uEnv.txt.

In the newer console images (> 2017-02) however this no longer works (echo to /dev/ttySX would proceed but nothing comes out of the actual pins). After some experiment, adding these lines instead of the line above solved the issue:

uboot_overlay_addr4=/lib/firmware/BB-UART1-00A0.dtbo
uboot_overlay_addr5=/lib/firmware/BB-UART2-00A0.dtbo
uboot_overlay_addr6=/lib/firmware/BB-UART4-00A0.dtbo
uboot_overlay_addr7=/lib/firmware/BB-UART5-00A0.dtbo

They are also renamed from /dev/ttyOX to /dev/ttySX, though there are symlinks in place for backward compatibility.


Relevant links:
http://elinux.org/Beagleboard:BeagleBoneBlack_Debian#U-Boot_.2Fboot.2FuEnv.txt_configuration

https://github.com/beagleboard/bb.org-overlays

A Checklist For New Base Stations

Hardware: Beaglebone Black (BBB) with v0.2 custom cape
SD card with console image (NO graphical interface. Edit /boot/uEnv.txt to enable flasher)

Important note: The watchdog will reboot the Beaglebone if it is not fed more frequently than once every five minutes. Keep the cape off the bone until software to feed the dog is configured.

Features that require verification:

• Real-Time Clock (RTC)
• Verify that the RTC has the correct UTC time
• Verify that the BBB is configured to use the WDT
• sudo i2cdetect -y -r 2 should show "UU" at address 0x68
• BBB has the correct time after boot (remove power for > 1 minute, then restore power, without Internet connection)
• Watchdog Timer (WDT)
• Verify that the WDT is fed
• Linux uptime should be > 5 minutes
• Verify that the WDT is active
• BBB is rebooted if WDT is not fed
• When in shutdown state, BBB is powered up by WDT within 5 minutes
• Four hardware serial ports
• Tx/Rx test, loopback test
• Both 3V3 TTL and RS232 levels
• The latest application software
• git pull ...
• sampling.py is running upon power up 
• System software all up-to-date
• sudo apt update && sudo apt upgrade -y
• Apache/MySQL are disabled (if installed)
• RabbitMQ link active and using TLS
• User does not have administrator tag
• Queue to_uhm is empty or near empty
• Daily backup scheduled in crontab
• Reverse SSH link active
• "Heartbeat" active (check heartbeat.py)
• Remaining disk space >1GB

During initial setup:

• Use 5V 2A power adapter and Ethernet cable. No SD card required.
• sudo apt install git
• cd /opt/scripts/tools..., git pull, expand...
• RTC must be tested without internet (SSH through the emulated Ethernet connection over USB)
• RTC must have the correct time (sudo hwclock -r -f /dev/rtc1)
• On boot, BBB's own clock must be correct (date and sudo hwclock -r -f /dev/rtc)
• Update system clock; apt update, upgrade etc.; tools and library installation
• cd ~ && git clone https://github.com/stanleylio/fishie.git node
• Check node/setup/setup.sh to see required libraries
  
• Setup Real-time Clock (RTC)
• Install NTP
• Set internal and external clock
• Setup service to set system clock on boot
• Verify (shutdown, completely remove power (mandatory), then power on without internet and check that the clock is correct)
• Set hostname
• Edit /etc/hostname and /etc/hosts
• Base stations are named base-NNN
• Generate new RSA key pair: ssh-keygen
• Setup Watchdog Timer (WDT)
• Verify (bone should restart within 15 minutes if dog is not fed)
• Setup sampling_core.py and sampling.py (which feed the dog in a loop), or
• Put "remove before flight section" in crontab: python ~/node/drivers/watchdog.py
• Setup automated backup and "phone home" etc.
• Database dump, configs etc.
• rsync to glazerlab-e5 (uhcmbackup)
• python node/send2server.py crontab every hour
• (optional) Reverse SSH to glazerlab-i7
• Setup supervisord 
• Enable I2C 1 and 2, UART 1, 2, 4, and 5; disable HDMI
• /boot/uEnv.txt
• Verify the files in /dev (i2c*, rtc*, ttyOX)
• Create MySQL database uhcm, create tables
• (optional) Enable remote access
• Setup RabbitMQ
• Create local user
• Setup Shovel to i7
• (optional) Create remote user
• (optional) Enable management plugin
• (optional) Enable Web STOMP
• Full acceptance test
• Receive message from XBee, retrieve latest reading in database, verify that messages appear on the real-time rtcomm.html page
• Check uptime periodically over several days

Date and Time - ntpdate Replacement

The ntpdate utility is deprecated (though still available via apt get install in Debian Jessie). To sync local clock without ntpdate:

sudo systemctl stop ntp
sudo ntpd -gq
sudo systemctl start ntp

The list of time servers to use is usually in /etc/ntp.conf (for the Beaglebone Jessie console image and Ubuntu server 16.04.2 LTS).

- - - - -

UHCM-specific:

NTP using public time servers doesn't work when using the UHM/SOEST network. Add server ntp.soest.hawaii.edu to /etc/ntp.conf to fix this.

MySQL Cheatsheet

Under Ubuntu Server 16.04:

To login to MySQL from terminal as root:

mysql -u root -p

Once inside the MySQL prompt:

To view the list of databases:

SHOW SCHEMAS;

To use a particular database:

USE databasename;

To view the list of tables inside the database:

SHOW TABLES;

To create a new user:

CREATE USER 'username'@'localhost' IDENTIFIED BY 'password';

To enable remote access to the MySQL database (for example, using MySQL Workbench):

GRANT ALL PRIVILEGES ON *.* TO 'username'@'%' IDENTIFIED BY 'password' WITH GRANT OPTION;

(optional) To grant only specific rights instead of all (using SELECT as example):

GRANT SELECT ON *.* TO 'username'@'localhost';

 Back to terminal, comment out the "bind_address" line in the MySQL config file. It's in /etc/mysql/mysql.conf.d/mysqld.cnf on my Ubuntu server 16.04.

Finally, restart the MySQL process:

sudo service mysql restart

- - - - -

Recently migrated from SQLite to MySQL. You can't beat SQLite's simplicity, but it's not so good for simultaneous access (single writer, multiple readers). Try to rsync the db to another machine while it's open and it quickly gets out of sync. On top of that there are the Linux permission issues (read-only users need write access to the db and its containing folder because SQLite needs to write temporary journal files).