Move The System Offline

Overview

This section walks the installer through the process of moving a LILT deployed system to an offline location. Please follow this documentation only if the internal IP of the server has changed during the migration.

Required Tools

You will need to use SSH to connect to the systems.

On the Mac or Linux, you can open a terminal window and use ssh.
On Windows, you can use the PuTTY ssh client.

You will also need a web browser to browse the website post installation and verification.

Installer Privileges

Many sections of this document include text formatted to indicate console input and output. The commands are prefixed with a $ or # depending on if the administrator should run them as root or not. Additionally, to assist the user to know which machine to run commands on, the prefix node, master, and gpu will be given when the machine should be switched. All following commands after a switch should use those.

Non-root commands

$ ./run_a_command as a normal user
node$ ./run_a_command as a normal user on the node machine

Root commands

# ./run_a_command as root

Prerequisites

This upgrade document is only valid for an existing on-premise install previously installed using on-prem-installer scripts. Before running the scripts, we must prepare dependencies.

Data Location

Please ensure that:

Docker images and RPM packages are mounted and available at $package_dir to all of the nodes (master, non-GPU, and GPU).

In the typical installation, the system administrator will receive this data from LILT prior to installation, and the system administrator will mount the data.

Host Name

This install is only valid if the Kubernetes node name is consistent during the process, eg. node machines should have the host names as: localhost.localdomain or localhost.node1 etc. You can set it using below commands:

# hostname localhost.localdomain
# hostnamectl set-hostname NEW_HOSTNAME
# exit and re-login

This is due to the added persistent volumes which get created inside a k8s node. This field is immutable, so if k8s node is referring to the IP address, it will change after the migration and Persistent Volumes (PV’s) won’t work.

Backup

We will back-up the running system prior to upgrade. This backup is optional, but recommended, and will be used to restore the projects, documents and memory content in case of any issues.

InstallDir

Backup Old Install Directory (used for previous install), referred as backupInstall in next steps.

Database

Backup database by running below commands on the installer system:

$ mysqldump --host HOST --user root --password=ROOT_PWD lilt_dev > backup.sql

Documents, Memory

Backup documents by copying openebs mount folders in k8s-node.

Migration steps

After the server has been moved to the offline location, ensure its hostname still remains the same (eg. localhost.localdomain) and the IP address changes.
Kubernetes will stop working due to the updated IP address. Follow the steps below to complete the migration.

1. Stop the services

# systemctl stop kubelet docker

2. Switch Directory

# cd /etc/

3. Backup old kubernetes data

# mv kubernetes kubernetes-backup
# mv /var/lib/kubelet /var/lib/kubelet-backup

4. Restore certificates

# mkdir -p kubernetes
# cp -r kubernetes-backup/pki kubernetes
# rm kubernetes/pki/{apiserver.,etcd/peer.}

5. Start docker

# systemctl start docker

6. Re-init master with data in etcd

# kubeadm init -ignore-preflight-errors=DirAvailable-var-lib-etcd --pod-network-cidr=10.244.0.0/16

7. Update kubectl config

# cp /etc/kubernetes/admin.conf ~/.kube/config

8. Remove old nodes

# kubectl get nodes --sort-by=.metadata.creationTimestamp
# kubectl delete node $(kubectl get nodes -o jsonpath='{.items[?(@.status.conditions[0].status=="Unknown")].metadata.name}')

9. Check running pods

# kubectl get pods --all-namespaces -o wide

10. Label/ taint node

Ignore if any of the below commands shows an existing value warning.

# kubectl label nodes <nodeName> node-type=worker
# kubectl label nodes <nodeName> node-role.kubernetes.io/master=master
# kubectl label nodes <nodeName> node-role.kubernetes.io/worker=worker
# kubectl taint nodes <nodeName> node-role.kubernetes.io/master-

11. Verify Node

Check if node is in ready status

# kubectl get nodes -o wide

12. Restart pods

# kubectl get pods -n kube-system | awk '{print $1}' | xargs kubectl delete pod -n kube-system
# sleep 30
# kubectl get pods -n lilt | awk '{print $1}' | xargs kubectl delete pod -n lilt

13. Verify Pods

Ensure all pods are in running status.

# kubectl get pods -A -o wide

​Overview

​Required Tools

​Installer Privileges

​Non-root commands

​Root commands

​Prerequisites

​Data Location

​Host Name

​Backup

​InstallDir

​Database

​Documents, Memory

​Migration steps

​1. Stop the services

​2. Switch Directory

​3. Backup old kubernetes data

​4. Restore certificates

​5. Start docker

​6. Re-init master with data in etcd

​7. Update kubectl config

​8. Remove old nodes

​9. Check running pods

​10. Label/ taint node

​11. Verify Node

​12. Restart pods

​13. Verify Pods