GlusterFS on ARM
Category : Kubernetes Raspberry Pi
GlusterFS on ARM
ARM devices like the Raspberry Pi offer an inexpensive way to learn and play with new technologies. Combine a few Raspberry Pis with GlusterFS to create a distributed and fault tolerant storage system that can be used by Docker and Kubernetes.
What is GlusterFS
GlusterFS is a clustered file system. Without going into the mechanics, clustered file systems provide a reliable process for files to be replicated among all of the cluster file system nodes. GlusterFS is well documented and free, which provides an excellent way to learn how clustered file systems work and how they can fit into a private (non-cloud) storage solution.
Hardware setup
4x Raspberry Pi 3B
4x Micro SD cards
4x USB thumb drives
1x 4-layer stackable case
1x USB Power (4 x 2.0A)
1x Pack of micro USB cables
1x 5 port gigabit ethernet switch
1x Pack of flat/flexible network cables
Installing XFS and GlusterFS packages
The XFS and GlusterFS packages need to be installed on all devices. Log into each Raspberry Pi and run:
sudo apt-get install -y xfsprogs glusterfs-server
Preparing the thumb drives
Note: The following steps will wipe all the data from the thumb drives, ensure that any data that needs to be saved is copied from the thumb drive before proceeding.
Ensure the thumb drives are inserted into the Raspberry Pis before continuing. First we need to identify the device name for each thumb drive. This should be /dev/sda if only a single thumb drive is inserted, and the lsblk command can be used to confirm:
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 57.9G 0 disk
└─sda1 8:1 1 57.9G 0 part
mmcblk0 179:0 0 29.7G 0 disk
├─mmcblk0p1 179:1 0 43.2M 0 part /boot
└─mmcblk0p2 179:2 0 29.7G 0 part /
As shown in the output above, there are two drives; sda and mmcblk0. The mount point of / and /boot indicate that mmcblk0 is the Micro SD card that Raspbian is installed on, and sda is a 57.9GB disk with no mount point which is the USB thumb drive. Make note of the device name for the USB thumb drive and substitute sda in the commands below as necessary.
The USB thumb drive needs to be partitioned and formatted for GlusterFS to use. The first step is to create a new partition with the correct type using fdisk. On all Raspberry Pis execute the following commands, which are highlighted in bold in the output below.
- sudo fdisk -w auto /dev/sda
- g
- n
- (Just press the Enter key to accept the default)
- (Just press the Enter key to accept the default)
- (Just press the Enter key to accept the default)
- w
sudo fdisk -w auto /dev/sda
Welcome to fdisk (util-linux 2.29.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): g
Created a new GPT disklabel (GUID: 8BAE955A-00E2-4FAB-BF57-79B32194E5CD).
Command (m for help): n
Partition number (1-128, default 1):
First sector (2048-121307102, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-121307102, default 121307102):
Created a new partition 1 of type 'Linux filesystem' and of size 57.9 GiB.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
The new partition on each thumb drive must be formatted with an XFS file system for GlusterFS.
sudo mkfs.xfs -f -L myvol-brick1 /dev/sda1
Mounting the thumb drives
The thumb drive must be mounted before it can be used. Using the GlusterFS administrator guide as a reference, the naming convention is /data/glusterfs/[volume]/[brick]/brick. On all Raspberry Pis, create the directory to mount the USB drive at.
sudo mkdir -p /data/glusterfs/myvol1/brick1/
Using the raw device path, such as /dev/sda1, is not recommended for USB drives because the device path can change when more than one USB drive is plugged in. The recommended and modern approach is to use the partition UUID to identify the target partition, as Raspbian is already doing if /etc/fstab is inspected. The blkid command can be used to find the partition UUID, and the output is passed to tee to write to /etc/fstab.
printf $(sudo blkid -o export /dev/sda1|grep PARTUUID)" /data/glusterfs/myvol1/brick1 xfs defaults,noatime 1 2\n" | sudo tee -a /etc/fstab
Now the partition is ready to be mounted so it can be used by glusterd.
sudo mount /data/glusterfs/myvol1/brick1
Configuring GlusterFS
The gluster daemon must be configured to know what the other nodes in the cluster are. From one Raspberry Pi run the following command 3 times, replacing PEER_HOSTNAME with the hostname of the other 3 Raspberry Pis. Gluster will automatically share the
sudo gluster peer probe PEER_HOSTNAME
peer probe: success.
Now the status of the gluster network should be checked to ensure proper configuration on each Raspberry Pi.
sudo gluster peer status
Number of Peers: 3
Hostname: pi-node2
Uuid: 3abf3a5f-1d41-47f9-b4b4-0f1d92def46d
State: Peer in Cluster (Connected)
Hostname: pi-node3
Uuid: d635c767-1257-41f5-b598-0be84bbfa75d
State: Peer in Cluster (Connected)
Hostname: pi-node4
Uuid: d635c767-1257-41f5-b598-0be84bbfa75d
State: Peer in Cluster (Connected)
With the gluster network running, a new volume needs to be created. Be sure to replace HOSTNAMEX with your hostnames. Since I want each of my Raspberry Pis to have a copy of the data, I have specified replica 4 and provided all 4 hosts and paths that will replicate the data. Tailor these to how you want your clustered file system to be replicated.
Note: Notice at the end of the path an additional /brick was added to the thumb drive mount point of /data/glusterfs/myvol1/brick1. This is done intentionally so that if the thumb drive is not mounted, gluster will fail to start the volume. This prevents gluster from replicating the contents of the cluster volume to the root drive (in this case the Micro SD card where Raspbian is installed).
sudo gluster volume create myvol1 replica 4 HOSTNAME1:/data/glusterfs/myvol1/brick1/brick HOSTNAME2:/data/glusterfs/myvol1/brick1/brick HOSTNAME3:/data/glusterfs/myvol1/brick1/brick HOSTNAME4:/data/glusterfs/myvol1/brick1/brick
volume create: myvol1: success: please start the volume to access data
Then the volume needs to be started.
sudo gluster volume start myvol1
volume start: myvol1: success
Testing GlusterFS
With the gluster volume running, we should conduct a test to ensure the data replicates as expected. One one node only, create a directory to mount the clustered volume, mount the volume, create a test file, and unmount the clustered volume.
sudo mkdir -p /mnt/myvol1
sudo mount -t glusterfs localhost:/myvol1 /mnt/myvol1
sudo touch /mnt/myvol1/helloworld.txt
sudo umount /mnt/myvol1
Now we can check the thumb drive to see if helloworld.txt was replicated to all the nodes. On each Raspberry Pi, check the thumb drive for files.
ls -al /data/glusterfs/myvol1/brick1/brick
total 0
drwxr-xr-x 4 root root 63 Jul 13 09:04 .
drwxr-xr-x 3 root root 19 Jul 13 09:04 ..
drw------- 10 root root 194 Jul 13 09:04 .glusterfs
-rw-r--r-- 2 root root 0 Jul 13 09:04 helloworld.txt
drwxr-xr-x 3 root root 25 Jul 13 09:04 .trashcan
That is all it takes to get your very own GlusterFS clustered file system running! Next step is to incorporate it into your service architecture to put it to good use.
2 Comments
Alec Kubala
April 14, 2020 at 7:48 amI really appreciate this post. I have been looking everywhere for this! Thank goodness I found it on Bing. You have made my day! Thx again!
Alton Portman
April 20, 2020 at 3:03 amI was able to find good advice from your blog, thanks!