> ## Documentation Index
> Fetch the complete documentation index at: https://support.lilt.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Move The System Offline

## Overview

This section walks the installer through the process of moving a LILT deployed system to an offline location. Please follow this documentation only if the internal IP of the server has changed during the migration.

## Required Tools

You will need to use SSH to connect to the systems.

* On the Mac or Linux, you can open a terminal window and use ssh.

* On Windows, you can use the PuTTY ssh client.

You will also need a web browser to browse the website post installation and verification.

### Installer Privileges

Many sections of this document include text formatted to indicate console input and output. The commands are prefixed with a \$ or # depending on if the administrator should run them as root or not. Additionally, to assist the user to know which machine to run commands on, the prefix node, master, and gpu will be given when the machine should be switched. All following commands after a switch should use those.

#### Non-root commands

```bash theme={null}
$ ./run_a_command as a normal user
node$ ./run_a_command as a normal user on the node machine
```

#### Root commands

```bash theme={null}
# ./run_a_command as root
```

## Prerequisites

This upgrade document is only valid for an existing on-premise install previously installed using on-prem-installer scripts. Before running the scripts, we must prepare dependencies.

### Data Location

Please ensure that:

* Docker images and RPM packages are mounted and available at *\$package\_dir* to all of the nodes (master, non-GPU, and GPU).

In the typical installation, the system administrator will receive this data from LILT prior to installation, and the system administrator will mount the data.

### Host Name

This install is only valid if the Kubernetes node name is consistent during the process, eg. node machines should have the host names as: localhost.localdomain or localhost.node1 etc. You can set it using below commands:

```bash theme={null}
# hostname localhost.localdomain
# hostnamectl set-hostname NEW_HOSTNAME
# exit and re-login
```

This is due to the added persistent volumes which get created inside a k8s node. This field is immutable, so if k8s node is referring to the IP address, it will change after the migration and Persistent Volumes (PV’s) won’t work.

## Backup

We will back-up the running system prior to upgrade. This backup is optional, but recommended, and will be used to restore the projects, documents and memory content in case of any issues.

### InstallDir

Backup Old Install Directory (used for previous install), referred as `backupInstall` in next steps.

### Database

Backup database by running below commands on the installer system:

```bash theme={null}
$ mysqldump --host HOST --user root --password=ROOT_PWD lilt_dev > backup.sql
```

### Documents, Memory

Backup documents by copying `openebs` mount folders in k8s-node.

## Migration steps

After the server has been moved to the offline location, ensure its hostname still remains the same (eg. localhost.localdomain) and the IP address changes.\
Kubernetes will stop working due to the updated IP address. Follow the steps below to complete the migration.

#### 1. Stop the services

```bash theme={null}
# systemctl stop kubelet docker
```

#### 2. Switch Directory

```bash theme={null}
# cd /etc/
```

#### 3. Backup old kubernetes data

```bash theme={null}
# mv kubernetes kubernetes-backup
# mv /var/lib/kubelet /var/lib/kubelet-backup
```

#### 4. Restore certificates

```bash theme={null}
# mkdir -p kubernetes
# cp -r kubernetes-backup/pki kubernetes
# rm kubernetes/pki/{apiserver.,etcd/peer.}
```

#### 5. Start docker

```bash theme={null}
# systemctl start docker
```

#### 6. Re-init master with data in etcd

```bash theme={null}
# kubeadm init -ignore-preflight-errors=DirAvailable-var-lib-etcd --pod-network-cidr=10.244.0.0/16
```

#### 7. Update kubectl config

```bash theme={null}
# cp /etc/kubernetes/admin.conf ~/.kube/config
```

#### 8. Remove old nodes

```bash theme={null}
# kubectl get nodes --sort-by=.metadata.creationTimestamp
# kubectl delete node $(kubectl get nodes -o jsonpath='{.items[?(@.status.conditions[0].status=="Unknown")].metadata.name}')
```

#### 9. Check running pods

```bash theme={null}
# kubectl get pods --all-namespaces -o wide
```

#### 10. Label/ taint node

Ignore if any of the below commands shows an existing value warning.

```bash theme={null}
# kubectl label nodes <nodeName> node-type=worker
# kubectl label nodes <nodeName> node-role.kubernetes.io/master=master
# kubectl label nodes <nodeName> node-role.kubernetes.io/worker=worker
# kubectl taint nodes <nodeName> node-role.kubernetes.io/master-
```

#### 11. Verify Node

Check if node is in ready status

```bash theme={null}
# kubectl get nodes -o wide
```

#### 12. Restart pods

```bash theme={null}
# kubectl get pods -n kube-system | awk '{print $1}' | xargs kubectl delete pod -n kube-system
# sleep 30
# kubectl get pods -n lilt | awk '{print $1}' | xargs kubectl delete pod -n lilt
```

#### 13. Verify Pods

Ensure all pods are in running status.

```bash theme={null}
# kubectl get pods -A -o wide
```