While LINBIT’s LINSTOR is on the way to becoming an industry standard as software defined block storage. MinIO stands out as the number one product in the object storage world.
LINBIT has been working on an integration with MinIO for a long time and has infrastructures that can be used safely in business solutions.
The biggest combination of LINSTOR and MinIO is reflected in INTEL’s RSD architecture.
INTEL, which wants to offer object storage on performance with 20 servers, 4 storage in a single rack, entrusted the management of disks to LINSTOR, while using MinIO for Object storage.
In this article, we wanted to give an example of how LINSTOR and MinIO can be combined.
If you have any questions about architecture and installation, please feel free to contact us on our slack channel.
Summary
This tutorial will show you a solution to de-couple MinIO application service and data on Kubernetes, by using LINSTOR as a distributed persistent volume instead of a local persistent volume.
1. Ubuntu virtual machine setup
Create a new and updated Ubuntu x86_64 virtual machine within 2 disks, one for Ubuntu and applications, the other one will be used for MinIO data storage.
In this case:
root@minikube:~# lsb_release -r Release: 18.04 root@minikube:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 30G 0 disk vda 252:0 0 60G 0 disk ├─vda1 252:1 0 59.9G 0 part / ├─vda14 252:14 0 4M 0 part └─vda15 252:15 0 106M 0 part /boot/efi
2. DRBD 9.0 setup
To install the latest DRBD 9.0, need to add PPA from LINBIT first (ref: https://launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack)
root@minikube:~# add-apt-repository ppa:linbit/linbit-drbd9-stack root@minikube:~# apt update root@minikube:~# apt install -y linux-headers-`uname -r` root@minikube:~# apt install -y drbd-utils drbd-dkms lvm2
Take a look at the details of DRBD 9.0
root@minikube:~# modinfo drbd filename: /lib/modules/4.15.0-106-generic/updates/dkms/drbd.ko alias: block-major-147-* license: GPL version: 9.0.23-1 description: drbd - Distributed Replicated Block Device v9.0.23-1 author: Philipp Reisner <phil@linbit.com>, Lars Ellenberg <lars@linbit.com> srcversion: 96D47E12A7A913F21D2507E depends: libcrc32c retpoline: Y name: drbd vermagic: 4.15.0-106-generic SMP mod_unload signat: PKCS#7 signer: sig_key: sig_hashalgo: md4 parm: enable_faults:int parm: fault_rate:int parm: fault_count:int parm: fault_devs:int parm: disable_sendpage:bool parm: allow_oos:DONT USE! (bool) parm: minor_count:Approximate number of drbd devices (1-255) (uint) parm: usermode_helper:string parm: protocol_version_min:drbd_protocol_version
3. minikube setup
According to the Kubernetes official documentation, install the latest minikube and kubectl both on Ubuntu.
root@minikube:~# curl -L https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 -o /usr/local/bin/minikube && chmod a+x /usr/local/bin/minikube root@minikube:~# curl -L https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl && chmod a+x /usr/local/bin/kubectl
Considering that minikube will be running in virtual machines directly, then set minikube driver to bare-metal.
There are two pre-requirements needed to be met first (ref: https://minikube.sigs.k8s.io/docs/drivers/none/)
3.1 Install Docker (ref: https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository)
root@minikube:~# apt install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common root@minikube:~# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - root@minikube:~# apt-key fingerprint 0EBFCD88 root@minikube:~# add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" root@minikube:~# apt update root@minikube:~# apt install -y docker-ce docker-ce-cli containerd.io root@minikube:~# docker run hello-world
3.2 Install conntrack
root@minikube:~# apt install -y conntrack
Now, it’s time to set minikube driver and start minikube.
root@minikube:~# minikube config set driver none root@minikube:~# minikube start
4. LINSTOR deployment with Helm v3 Chart
Deployment of LINSTOR CSI operator is recommended. We maintain Helm charts for this, and as such suggest the use of Helm v3.
Download Helm v3 from Github (https://github.com/helm/helm/releases/latest), and copy helm to /usr/local/bin/ .
And label the nodes that will be used for LINSTOR. <NODE_NAME> = hostname, in this case <NODE_NAME> = minikube .
root@minikube:~# kubectl label nodes minikube linstor.linbit.com/piraeus-node=true
In this tutorial, lvm-thin will be used for back-end storage.
root@minikube:~# git clone https://github.com/piraeusdatastore/piraeus-operator root@minikube:~# helm dependency update ./piraeus-operator/charts/piraeus root@minikube:~# helm install piraeus-op ./piraeus-operator/charts/piraeus --set operator.nodeSet.automaticStorageType=LVMTHIN
Now docker instances are being created, and DRBD v9.0 will be injected automatically, and all unused disks will be used for LINSTOR storage-pool automatically.
root@minikube:~# lsmod | grep -i drbd drbd_transport_tcp 24576 0 drbd 536576 3 drbd_transport_tcp libcrc32c 16384 6 nf_conntrack,nf_nat,dm_persistent_data,drbd,raid456,ip_vs root@minikube:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 30G 0 disk ├─linstor_sda-sda_tmeta 253:0 0 32M 0 lvm │ └─linstor_sda-sda-tpool 253:2 0 30G 0 lvm └─linstor_sda-sda_tdata 253:1 0 30G 0 lvm └─linstor_sda-sda-tpool 253:2 0 30G 0 lvm
Verify this deployment via LINSTOR client, to see what happened actually.
root@minikube:~# apt install -y linstor-client root@minikube:~# linstor node list +--------------------------------------------------------------------------------+ | Node | NodeType | Addresses | State | |================================================================================| | minikube | SATELLITE | 192.168.122.71:3366 (PLAIN) | Online | | piraeus-op-cs-controller-0 | CONTROLLER | 172.17.0.9:3366 (PLAIN) | Online | +--------------------------------------------------------------------------------+ root@minikube:~# linstor storage-pool list +--------------------------------------------------------------------------------------------------------------------+ | StoragePool | Node | Driver | PoolName | FreeCapacity | TotalCapacity | CanSnapshots | State | |====================================================================================================================| | DfltDisklessStorPool | minikube | DISKLESS | | | | False | Ok | | sda | minikube | LVM_THIN | linstor_sda/sda | 27.79 GiB | 29.93 GiB | True | Ok | +--------------------------------------------------------------------------------------------------------------------+
New storage-pool is created, and named as the device name automatically.
5. MinIO setup
Create storage class for MinIO first:
root@minikube:~# cat sc.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass annotations: storageclass.kubernetes.io/is-default-class: "true" metadata: name: "linstor-csi-lvm-thin-r1" parameters: autoplace: "1" storagePool: "sda" provisioner: linstor.csi.linbit.com reclaimPolicy: Delete root@minikube:~# kubectl apply -f sc.yaml
Deploy MinIO:
root@minikube:~# cat minio.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: minio-deployment
spec:
selector:
matchLabels:
app: minio
strategy:
type: Recreate
template:
metadata:
labels:
app: minio
spec:
volumes:
- name: storage
persistentVolumeClaim:
claimName: minio-pv-claim
containers:
- name: minio
image: minio/minio:latest
args:
- server
- /data
env:
# Minio access key and secret key
- name: MINIO_ACCESS_KEY
value: "minio"
- name: MINIO_SECRET_KEY
value: "minio123"
ports:
- containerPort: 9000
hostPort: 9000
volumeMounts:
- name: storage
mountPath: "/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: minio-pv-claim
labels:
app: minio-storage-claim
spec:
storageClassName: linstor-csi-lvm-thin-r1
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10G
---
apiVersion: v1
kind: Service
metadata:
name: minio-service
spec:
type: LoadBalancer
ports:
- port: 9000
nodePort: 32701
protocol: TCP
selector:
app: minio
sessionAffinity: None
type: NodePort
root@minikube:~# kubectl apply -f minio.yamlVerify this pvc in LINSTOR.
root@minikube:~# linstor volume-definition list +----------------------------------------------------------------------------------------------+ | ResourceName | VolumeNr | VolumeMinor | Size | Gross | State | |==============================================================================================| | pvc-8838db2c-d65d-4b6c-8bbf-660ea7718699 | 0 | 1000 | 9.31 GiB | | ok | +----------------------------------------------------------------------------------------------+
And take look at block device level in Ubuntu
root@minikube:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 30G 0 disk
├─linstor_sda-sda_tmeta 253:0 0 32M 0 lvm
│ └─linstor_sda-sda-tpool 253:2 0 30G 0 lvm
│ ├─linstor_sda-sda 253:3 0 30G 0 lvm
│ └─linstor_sda-pvc--8838db2c--d65d--4b6c--8bbf--660ea7718699_00000 253:4 0 9.3G 0 lvm
│ └─drbd1000 147:1000 0 9.3G 0 disk /var/lib/kubelet/pods/2141f8df-d320-4bd6-8af5-418a1c5dfa0b/volumes/kubernetes.io~csi/pvc-8838db2c-d65d-4b6c-8bbf-660ea7718699/mount
└─linstor_sda-sda_tdata 253:1 0 30G 0 lvm
└─linstor_sda-sda-tpool 253:2 0 30G 0 lvm
├─linstor_sda-sda 253:3 0 30G 0 lvm
└─linstor_sda-pvc--8838db2c--d65d--4b6c--8bbf--660ea7718699_00000 253:4 0 9.3G 0 lvm
└─drbd1000 147:1000 0 9.3G 0 disk /var/lib/kubelet/pods/2141f8df-d320-4bd6-8af5-418a1c5dfa0b/volumes/kubernetes.io~csi/pvc-8838db2c-d65d-4b6c-8bbf-660ea7718699/mountIn a browser, navigate to the IP address of this Ubuntu virtual machine at the exposed port 9000 or 32701, and login using the default credentials:
Access Key : minio Secret key : minio123
Upload some files, in this case, Fedora-Cinnamon-Live-x86_64-32-1.6.iso (1.9GiB) will be used. And check LINSTOR again.
root@minikube:~# linstor volume list +------------------------------------------------------------------------------------------------------------------------------------+ | Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State | |====================================================================================================================================| | minikube | pvc-8838db2c-d65d-4b6c-8bbf-660ea7718699 | sda | 0 | 1000 | /dev/drbd1000 | 2.14 GiB | InUse | UpToDate | +------------------------------------------------------------------------------------------------------------------------------------+
Use the official MinIO client to see files within the exact credentials above.
root@minikube:~# ./mc ls local [2020-06-15 21:33:35 CST] 0B my-bucket/ root@minikube:~# ./mc ls local/my-bucket/ [2020-06-15 21:34:41 CST] 1.9GiB Fedora-Cinnamon-Live-x86_64-32-1.6.iso
Conclusions:
Using LINBIT’s LINSTOR as a block storage orchestrator, not only replicates data to many different server nodes, but also supports disk-less mode to allow access to block storage from one node to another. Even more, when integrated with the Stork plugin it can give you features to run the pod on the same server node housing the data allowing for native storage performance.
