# Evaluation of different hardware set ups
In the following, it shall be evaluated if a cluster of Raspberry Pi 5 single board computers (SBC) can be used to run a proxmox server sufficiently fluently.

The goal is hence to first install proxmox on an arm based board and then to configure a cluster of SBC configure it such that it can replicate virtual machines and redundantly continue tasks if one node breaks down(high availability, HA).

# Installation of Proxmox on Raspberry Pi 5
Proxmox can effectively be installed on a Raspberry Pi 5 model B when following the step by step guide described at pi my life up (https://pimylifeup.com/raspberry-pi-proxmox/). Flash the latest Raspberry pi OS onto the microSD card using RaspberryPi Imager. In the mentined tutorial, Raspberry pi OS bookworm was used. Further, for external storage, connect an external flash drive or SSD via USB to the board. This setup of the Raspberry Pi with the latest OS and an external storage memory was also used for setting up next cloud through freedombox described later in this document.

Important to note is, that for each board, a different hostname should be used (Step5: use different name than raspberrypi). This can be achieved by running

	sudo hostnamectl set-hostname new-hostname
 
 and then by setting the new-hostname in step5 instead of raspberrypi.

The Web UI of proxmox can then be accessed when running `IP_ADDRESS:8006` of the device where proxmox has been installed. aparld's adventures https://www.youtube.com/watch?v=TXFYTQKYlno describes then how to set up and configure the proxmox servers to form a cluster. A cluster of raspberry pis running proxmox could be configured but setting up shared storage (ceph) always caused problems, apparently due to I/O errors. Overall, the raspberry pis hardware seems to be powerful enough to be able to run services, such as next cloud, in a virtualized environment.

# Change of Plan - Talos OS
Due to the complications we ran into during configuration and because the proxmox version used is not an official but a community driven one, setting up talos OS on different hardware was tested:

Talos OS was downloaded from https://github.com/siderolabs/talos/releases/tag/v1.10.0, specifically metal-arm64.raw.zst. and flashed onto the raspberry pis microSD card using BalenaEtcher.
Since the device didnt boot, we vound out according to https://github.com/siderolabs/talos/issues/7978, that Talos OS is not compatible with the hardware and the plan was hence not further pursued.

# Open source Hardware Olimex Lime2 board
Besides using Raspberry Pi 5 as chosen hardware, the Olimex Lime 2 board(A20-OLinuXino-LIME2-e16Gs16M), an open source single board computer was also assessed. The goal was to configure a docker swarm for high availability since neither Talos OS nor Porxmox was compatible with the board. Within Docker, next cloud services should be run to facilitate storage replication.

For the pysical setup, the board is connected to the 5V power adapters as well as to ethernet. A SATA III SSD is connected to the SATA connector as well as the corresponding power supply of the board using the original cable set by Olimex. The official operating system was flashed on a micro SD card, but could alternatively be copied into the internal eMMC flash memory of the board itself.

Download the operating system directly from Olimex: https://images.olimex.com/release/a20/ and flash it using Win32DiskImager, for example, to load it onto a micro SD card.
This card then has to be inserted into the Lime2 board and the board has to be powered on. The board will then boot from the micro SD card and the operating system will shortly be loaded.

After first boot, a software reboot is recommended according to the instructions given at https://github.com/OLIMEX/OLINUXINO/blob/master/DOCUMENTS/OLIMAGE/Olimage-guide.pdf. Note that the default password is olimex, you might wannt to change it.

# Set up a docker swarm on the lime 2 board
The goal is now to set up a docker swarm on lime 2 to allow for high availability of the next cloud services.

// the installation steps are based on the explanations in the video https://www.youtube.com/watch?v=nmas_zbcMeU
// 3 nodes ares set up with different static IP (here assigned by router), master node on 192.168.0.31, backup node 1 on 192.168.0.30, backup node 2 on 192.168.0.32 and keepalive virtual ip was set up on 192.168.0.29. if not stated otherwise, run the commands on all of the three nodes

	sudo apt update && sudo apt upgrade -y
	
	sudo apt install docker
	sudo apt install docker-compose

// on all nodes change the hostname to something different (unique definition for each node)

	sudo hostnamectl set-hostname NEW_HOSTNAME


// run on node 1

	sudo docker swarm init --advertise-addr 192.168.0.31

// copy response of previous command and run it on node 2 and 3

	sudo docker swarm join --token SWMTKN-1-0jsci5r7tygjc16rzi1yws4dmkisbua2f3ppethnkicgzt2fl5-65q7cpkjndk0autbd2tuw56ud 192.168.0.31:2377

// on the main node, node 1, run the following command to promote the other nodes to manager

	sudo docker node promote <hostname>
 
// install keepalived to create virtual ip for all of the nodes

	sudo apt install keepalived -y

// insert the following content for the three nodes, respectively

	sudo nano /etc/keepalived/keepalived.conf
 
// on master node add:

	vrrp_instance VI_1 {
		state MASTER
		interface eth0
		virtual_router_id 51
		priority 120
		advert_int 1
		authentication{
			auth_type PASS
			auth_pass Letmein1$	
		}
		unicast_peer {
		192.168.0.30
		192.168.0.32
		}
		virtual_ipaddress {
		192.168.0.29
		}
	}

// on backup node1 add:

	vrrp_instance VI_1 {
		state BACKUP
		interface eth0
		virtual_router_id 51
		priority 110
		advert_int 1
		authentication{
			auth_type PASS
			auth_pass Letmein1$	
		}
		unicast_peer {
		192.168.0.31
		192.168.0.32
		}
		virtual_ipaddress {
		192.168.0.29
		}
	}

// on backup node 2 add:

	vrrp_instance VI_1 {
		state BACKUP
		interface eth0
		virtual_router_id 51
		priority 100
		advert_int 1
		authentication{
			auth_type PASS
			auth_pass Letmein1$	
		}
		unicast_peer {
		192.168.0.30
		192.168.0.31
		}
		virtual_ipaddress {
		192.168.0.29
		}
	}


// on all of the nodes, start and enable keepalived and verify the connectivity of the virtual IP

	sudo systemctl start keepalived
	sudo systemctl enable keepalived
	sudo systemctl status keepalived
	ping 192.168.0.29

// how to mount ssd as file system on each node

	lsblk
	sudo mkfs.ext4 /dev/sda1
	sudo mkdir -p /mnt/nextcloud_data
	sudo mount /dev/sda1 /mnt/nextcloud_data

// copy the UUID for /dev/sdaX, retrieveable here

	sudo mlkid
 
// add a line to that file

	sudo nano /etc/fstab
 
 // add this to the file 
 
		UUID=xxxxx  /mnt/nextcloud_data  ext4  defaults  0  2
// then run

	sudo mount -a
 
// create a named volume:

	sudo docker volume create \
	  --driver local \
	  --opt type=none \
	  --opt device=/mnt/nextcloud_data \
	  --opt o=bind \
	  nextcloud_bind



// how to start docker with nextcloud on it

	sudo docker pull nextcloud
	sudo docker service create \
	  --name nextcloud_app \
	  --replicas 3 \
	  --publish 8080:80 \
	  --mount type=volume,source=/mnt/nextcloud_data,target=/var/www/html \
	  arm32v7/nextcloud


// useful commands when running docker as a swarm

	sudo docker node ls
	sudo docker service ls


//  To facilitate docker management, it is possible to install portainer to have a graphical user interface for container management. This setup was not done but could facilitate further container management...



///// in the previously stated video https://www.youtube.com/watch?v=nmas_zbcMeU, on each node a ceph storage is set up for redundant replication. The setup was not completed on all nodes due to the too low computational power of the lime2 boards for running next cloud efficiently but is yet further described hereafter. 

	sudo snap install microceph --channel=quincy/edge
	sudo snap refresh --hold microceph
	
	sudo microceph cluster bootstrap
	sudo microceph status

// on manager node run: (where a20-olixuinoX is the hostname of the two other nodes)

	sudo microceph cluster add a20-olinuxino1
	sudo microceph cluster add a20-olinuxino2

// then on the two other nodes that were just added accept the invite by copying the token returned by the previous command, respectively, and run

	sudo microceph cluster join <token>


// add the disk for the nodes, on each node run

	sudo microceph disk add /dev/sda --wipe

//on node one run:  (all disk should be listed:)

	sudo microceph status
	
	sudo ceph osd pool create cephfs_data 64
	sudo ceph osd pool create cephfs_metadata
	sudo ceph fs new cephfs cephfs_metadata cephfs_data
	
	sudo ceph fs ls

// 13 min in video: run on each node to mount the storage on all three

	sudo mkdir /mnt/cephfs

	sudo ceph auth get-key client.admin
 
// on each node add:

	sudo nano /etc/fstab
 
// add this to the end of the file

	ip_address1:6789, ipaddress2:6789, ipaddress3:6789, /mnt/cephfs ceph name=admin, secret=<secret token>, _netdev 0 0
 
// then run

	sudo systemctl daemon-reload
	df -h

// created persistent storage on /mnt/cephfs



# Due to the low computational power of the lime 2 boards, the next approach would be to install freedombox on a raspberry pi 5 with 16 GB and to have a Lime 2 board as additional ipfs server for file backup

# Setting up an IPFS server with SATA SSD storage
// Installing ipfs on a lime2 board. - First install go lang and then kubos as described hereafter
// The guide presented in the following screates the default ipfs directory in the attached SATA storage. Note that a special sata cable set is needed to power up the SATA drive with the Lime2 Boards. First, use a clean installation of the Linux debian based operating system provided by Olimex https://images.olimex.com/release/a20/. Flash it onto the microSD card using Win32DiskImager, for example.


	sudo apt update && sudo apt upgrade
	wget https://go.dev/dl/go1.22.3.linux-armv6l.tar.gz
	sudo rm -rf /usr/local/go
	sudo tar -C /usr/local -xzf go1.22.3.linux-armv6l.tar.gz

// add the following into the file .profile

	nano ~/.profile
 
 // add these lines
 
		export PATH=$PATH:/usr/local/go/bin
		export GOPATH=$HOME/go
		export GOBIN=$GOPATH/bin

//
	source ~/.profile
	go version


// format and initialize the external hard drive/ssd, choose the correct drive you want to use:

	lsblk
 
// unmount it

	sudo umount /dev/sdXY

	sudo wipefs -a /dev/sdX
	sudo parted /dev/sdX mklabel gpt
	sudo parted -a opt /dev/sdx mkpart primary ext4 0% 100%
	sudo mkfs.ext4 /dev/sdX1

//create mount point and mount it, here called sda1

	sudo mkdir -p /mnt/ipfs
	sudo mount /dev/sda1 /mnt/ipfs

//then make it persistent by finding the UUID and adding the indicated line to /etc/fstab

	sudo blkid /dev/sda1
	sudo nano /etc/fstab
 
 // add the following line
 
		UUID=your-uuid-here  /mnt/ipfs  ext4  defaults  0  2
	
// Then change permissions for writing to the drive
	sudo chown -R olimex:olimex /mnt/ipfs

// install ipfs kubos:

	wget https://dist.ipfs.tech/go-ipfs/v0.23.0/go-ipfs_v0.23.0_linux-arm.tar.gz
	tar -xvzf go-ipfs_v0.23.0_linux-arm.tar.gz
	cd go-ipfs
	sudo bash install.sh
	ipfs --version


// check if some known file is accessible via IPFS using the following commands:

	sudo IPFS_PATH=/mnt/ipfs ipfs init
	sudo IPFS_PATH=/mnt/ipfs ipfs cat QmbzhnPCWip1hk8nPRposWSk1BUCtr4B6jP7hDH74TTG1c


// configure now a service that, by default, uses /mnt/ipfs and runs at startup:

	sudo nano /etc/systemd/system/ipfs.service
 
 // edit/add the following
		[Unit]
		Description=IPFS Daemon
		After=network.target
		
		[Service]
		Environment=IPFS_PATH=/mnt/ipfs
		ExecStart=/usr/local/bin/ipfs daemon
		User=olimex
		Restart=always
		RestartSec=10
		
		[Install]
		WantedBy=multi-user.target


// then run the following commands:

	sudo systemctl daemon-reexec
	sudo systemctl daemon-reload
	sudo systemctl enable ipfs
	sudo systemctl start ipfs

//test IPFS
	echo "Hello Lime2 IPFS" > hello.txt
	ipfs add hello.txt

// a CID should now be returned and for each added file, the occupied size of the folder in /mnt/ipfs should increase.
// The IPFS server has now been successfully set up as a node and can be used to back up/ replicate data shared within IPFS.


# Using a Raspberry Pi as main hardware to run freedombox
Due to the limited computational power of a Lime2 board, a Raspberry Pi with 16Gb Ram was chosen as main SBC. 
Freedombox, an IPFS node, ufw (uncomplicated firewall) as well as an external disk/ssd was set up and configured.
For the external disk as well as the IPFS node, the same approach as described for the Olimex Lime2 board could be used. Instead of installing it on the official Olimex Lime 2 operating system, raspberry Pi OS was flashed on a microSD card using Raspberry Pi imager prior to the setup described. After correct installation of the IPFS node and the storage, the firewall as well as freedom box can be installed using apt package manager by running:

	sudo apt install ufw
	sudo apt install freedombox