Deploy multi-region applications with k8s


## 1. Run 2 K8S clusters locally for practicing

Create 2 clusters with 1 master for each cluster. Image clusters put in different regions.

See [this post](/post/9-run-multi-node-k8s-cluster-locally).

## 2. Solution

In my case, I use MariaDB with `replication` architecture.

![Solution](/images/10-deploy-multi-region-applications-with-k8s/1-solution.jpg)

I use `bitnami/mariadb` 9.3.4 chart.

I need run mariadb with primary & secondary in different nodes, but `bitnami/mariadb` allow run 1 primary and many secondary nodes. So I need run some customization.

## 3. Customize `bitnami/mariadb` chart

3.1. Add primary.enabled to `values.yaml`

```yaml
primary:
  enabled: true
```

3.2. Add condition `.Values.primary.enabled` to all file in `templates/primary`

3.3. Add master's information to secondary in `values.yaml`.

```yaml
secondary:
  master:
    host:
    port:
```

3.4. Update env in `templates/secondary/statefulset.yaml`

```yaml
env:
  - name: MARIADB_MASTER_HOST
    value: {{ default (include "mariadb.primary.fullname" .) .Values.secondary.master.host }}
  - name: MARIADB_MASTER_PORT_NUMBER
    value: {{ default .Values.primary.service.port .Values.secondary.master.port | quote }}
```

See my final custom chart on [Github](https://github.com/phanletrunghieu/multi-region-k8s)

## 4. Deploy

Switch `kubectl` to cluster 1 and run

```yaml
helm -n app  upgrade --install app . --create-namespace -f values.region-1.yaml
```

Switch `kubectl` to cluster 2 and run
```yaml
helm -n app  upgrade --install app . --create-namespace -f values.region-2.yaml
```

Run multi-node kubernetes cluster locally within 15 minutes


## 1. Prepare tools

- multipass 1.6.2: `brew install --cask multipass`
- virtualbox 6.1.18: `brew install --cask virtualbox`

> Note: You can install tools with other tools.

## 2. Create 2 VMs

Create 2 nodes Ubuntu 20.04, 1G RAM, 1 CPU, 8GB disk.

```bash
multipass launch -vvv -n node-1 -m 1G -d 8GB -c 1 20.04
multipass launch -vvv -n node-2 -m 1G -d 8GB -c 1 20.04
```

Check in VirtualBox: `sudo VirtualBox`
> Note: Open VirtualBox with sudo

![Nodes in the cluster](/images/9-run-multi-node-k8s-cluster-locally/1-nodes-in-cluster.png)

## 3. Configure the network

Because 2 VMs use network NAT, it has no a special ip. So we need to create 2 new networks: Host-only (nodes will communicate to each other with this network), NAT network (to allow nodes be able to access the internet) and disable the current NAT.


3.1. In **VirtualBox**, open **Preferences**, create a new NatNetwork and enable DHCP

![Create NAT network](/images/9-run-multi-node-k8s-cluster-locally/2-create-nat-network.png)

![NAT network with DHCP](/images/9-run-multi-node-k8s-cluster-locally/3-nat-network-dhcp.png)

3.2. Shutdown nodes

![Shutdown nodes](/images/9-run-multi-node-k8s-cluster-locally/4-shutdown-nodes.png)

3.3. Add new networks

![Add new networks](/images/9-run-multi-node-k8s-cluster-locally/5-add-new-network-1.png)

![Add new networks](/images/9-run-multi-node-k8s-cluster-locally/5-add-new-network-2.png)

3.4. Start nodes

![Start nodes](/images/9-run-multi-node-k8s-cluster-locally/6-start-nodes.png)

3.5. If the `~/.ssh/id_rsa` file is not exist, generate a pair of ssh keys.

```bash
ssh-keygen -C "hieu@deptrai"
cat ~/.ssh/id_rsa.pub
```

3.6. SSH to node 1:

```bash
multipass shell node-1
```

3.7. Write your ssh public key to node-1
```bash
echo "your_ssh_public_key" | tee -a ~/.ssh/authorized_keys
```

Replace `your_ssh_public_key` with `~/.ssh/id_rsa.pub` file's content.

Example:

```bash
echo "ssh-rsa AAAAB3NzaC1yc2E..... hieu@deptrai" | tee -a ~/.ssh/authorized_keys
```

3.8. Enable 2 networks that we enabled

```bash
ip link | grep DOWN
```

They are usually `enp0s8` and `enp0s9`
![Start nodes](/images/9-run-multi-node-k8s-cluster-locally/7-network-down.png)

Run this command with `sudo`. Notice `enp0s8` and `enp0s9`.

```bash
sudo bash -c "cat > /etc/netplan/60-bridge.yaml" <<EOF
network:
  ethernets:
    enp0s8:
      dhcp4: true
      dhcp4-overrides:
        route-metric: 200
    enp0s9:
      dhcp4: true
      dhcp4-overrides:
        route-metric: 200
  version: 2
EOF
```
```bash
sudo netplan apply
ip addr show
```

The `enp0s9`'s ip is `192.168.56.108`

3.9. Exit ssh:
```
exit
```

3.10. Shutdown node-1 -> disable `NAT` at `Adapter 1` -> Start node-1

![Disable network at adapter 1](/images/9-run-multi-node-k8s-cluster-locally/8-disable-network-adapter-1.png)

3.11. Repeat steps `3.6`-`3.10` with `node-2`

Finally, we have IPs:
- Node-1: 192.168.56.108
- Node-2: 192.168.56.109

## 4. Setup a k8s cluster

4.1. Install k3sup

```bash
curl -sLS https://get.k3sup.dev | sh
sudo install k3sup /usr/local/bin/
```

4.2. Install Kubernetes in `node-1` with `master` role.

```bash
k3sup install --ip 192.168.56.108 --user ubuntu --context k3s-1 --k3s-version v1.19.8+k3s1
```

4.3. Install Kubernetes in `node-2` with `worker` role.

```bash
k3sup join --ip 192.168.56.109 --server-ip 192.168.56.108 --user ubuntu --k3s-version v1.19.8+k3s1
```

It will create `kubeconfig` file in your current directory.

## 5. Test

```bash
export KUBECONFIG=/Users/hieudeptrai/kubeconfig
kubectl get node -o wide
```

Done!

Tôi quản lý 1 nùi mật khẩu thế nào?


## 1. Nhu cầu

- Lưu được mật khẩu an toàn nhất có thể
- Đồng bộ trên các nền tảng: Android, iOS, browser,..
- Không giới hạn thiết bị
- Hỗ trợ đăng nhập 2 lớp
- Free càng tốt

## 2. Tool mình đã từng sử dụng

**Lastpass**

Tool khá xịn, đầy đủ các chức năng:

- Đồng bộ trên các nền tảng: Android, iOS, browser,..
- Không giới hạn thiết bị
- Đặt biệt là free

**1password**

Có đầy đủ các tính năng xịn xò, nhưng không miễn phí.

## 3. Giải pháp hiện tại mình đang dùng

Dạo 1 vòng tìm kiếm thì thấy những app có tính đăng nhập 2 lớp đều là gói premium hết.

Chuyển hướng qua tìm opensource thì thấy có thằng Bitwarden. App này có đủ tính năng mình cần và hỗ trợ self host.

### Tìm 1 con server free:

- EC2 t2.micro của Amazon
- f1-micro của Google Cloud

### Chuẩn bị 1 domain

Ví dụ `bws.example.com`

### Chuẩn bị thêm

- docker 19.03+
- docker-compose 1.26+
- nginx 1.14+
- certbot

```
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install python-certbot-nginx
```

- 1 SMTP account: có thể dùng `smtp.gmail.com` với account gmail luôn

### Deploy Bitwarden server

- Official (C#): https://github.com/bitwarden/server
- Non-Official (Rust): https://github.com/dani-garcia/bitwarden_rs

Có 2 option, ở đây mình dùng server Rust cho gọn nhẹ.

Tạo 1 file `docker-compose.yaml`

```yaml
version: '3'

services:
  bitwarden:
    image: bitwardenrs/server:1.16.3
    restart: always
    volumes:
      - ./bw-data:/data
    ports:
      - 8080:80 # web, api, ...
      - 3012:3012 # websockets
    environment:
      WEBSOCKET_ENABLED: 'true'
      SIGNUPS_ALLOWED: 'true'
      SHOW_PASSWORD_HINT: 'true'
      ROCKET_WORKERS: 20
      ADMIN_TOKEN: 'xxxxxxxhieudeptraixxxxxxx'
      DOMAIN: https://bws.example.com
      SMTP_HOST: smtp.gmail.com
      SMTP_FROM: abc@gmail.com
      SMTP_PORT: 587
      SMTP_SSL: 'true'
      SMTP_USERNAME: abc@gmail.com
      SMTP_PASSWORD: xxxxx
```

Chạy file docker-compose `docker-compose up -d`


Tạo 1 file `bitwarden` tại `/etc/nginx/sites-enabled`

```nginx
server {
    listen 80;
    listen [::]:80;

    server_name bws.example.com;

    client_max_body_size 128M;

    location / {
        proxy_pass       http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /notifications/hub {
        proxy_pass         http://localhost:3012;
        proxy_set_header   Host $host;
        proxy_http_version 1.1;
        proxy_set_header   Upgrade $http_upgrade;
        proxy_set_header   Connection "Upgrade";
    }

    location /notifications/hub/negotiate {
        proxy_pass http://localhost:8080;
    }

    location /admin {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_pass http://localhost:8080;
    }
}
```

Apply nginx config: `sudo nginx -s reload`

### Config thêm https cho an toàn hơn

```
sudo certbot --nginx -d bws.example.com
```

### Tạo user

Vào https://bws.example.com/admin, điền admin token trong file docker-compose.yaml vào.

![Đăng ký/đăng nhập Bitwarden](/images/8-toi-quan-ly-mat-khau-cua-minh-the-nao/1-dang-nhap.png)

### Cài extension/app

- Browser extension: https://github.com/bitwarden/browser
- Android: https://play.google.com/store/apps/details?id=com.x8bit.bitwarden
- iOS: https://apps.apple.com/us/app/bitwarden-password-manager/id1137397744

## 4. Một vài hình ảnh nhá hàng

Web

![Web](/images/8-toi-quan-ly-mat-khau-cua-minh-the-nao/2-web.png)

Xem password

![Xem password](/images/8-toi-quan-ly-mat-khau-cua-minh-the-nao/3-add-password.png)

Extension

![Extension](/images/8-toi-quan-ly-mat-khau-cua-minh-the-nao/4-extension.png)

Extension nhận diện mật khẩu

![Extension nhận diện mật khẩu](/images/8-toi-quan-ly-mat-khau-cua-minh-the-nao/5-extension-detect-password.png)

Setup K3s on Digitalocean


## 1. Create a Droplet

![Create a Droplet](/images/7-install-k3s-in-digitalocean/1-create-a-droplet.png)

Result:

![Result Droplet](/images/7-install-k3s-in-digitalocean/2-result-droplet.jpg)

Try to ssh to droplet.

```bash
ssh root@123.456.789.012
```

Install somethings:

```bash
apt install curl
```

If success, go to step 2.

## 2. Install `k3sup` on your computer

```bash
curl -sLS https://get.k3sup.dev | sh
```

## 3. Install k3s

```bash
k3sup install --ip 123.456.789.012 --user root
```

![Result 3-k3sup](/images/7-install-k3s-in-digitalocean/3-k3sup.png)

## 4. Test

After installing successfully, we have `kubeconfig` file.

```bash
export KUBECONFIG=/Users/hieudeptrai/kubeconfig
kubectl get node -o wide
```

Done!

Quản lý log tập trung trong microservice với ELK. Phần 1: trên docker swarm


## 1. Kiến trúc

![List node](/images/6-quan-ly-log-tap-trung-elk-phan-1-tren-docker-swarm/architect.png)

## 2. Khởi tạo docker swarm

Bạn có thể dùng nhiều server có cài sẵn docker để tạo thành 1 cluster. Tốt nhất là docker cùng version để tránh gặp những lỗi tào lao. Trong bài viết này mình dùng docker-machine để giả lập 3 node để demo.

### 2.1. Tạo 1 cluster với 3 node

```bash
docker-machine create -d virtualbox --virtualbox-memory "2048" node1
docker-machine create -d virtualbox --virtualbox-memory "1024" node2
docker-machine create -d virtualbox --virtualbox-memory "1024" node3
```

Fixing `out of memory` error of Elasticsearch

```bash
docker-machine ssh node1 sudo sysctl -w vm.max_map_count=262144
docker-machine ssh node2 sudo sysctl -w vm.max_map_count=262144
docker-machine ssh node3 sudo sysctl -w vm.max_map_count=262144
```

Kiểm tra các node đã tạo

```bash
docker-machine ls
```

![List node](/images/6-quan-ly-log-tap-trung-elk-phan-1-tren-docker-swarm/1-list-node.png)

### 2.2. Cho `node1` làm node manager.

```bash
docker-machine ssh node1
docker swarm init --advertise-addr 192.168.99.104
```

`192.168.99.104` là ip của node 1, đã kiểm tra ở trên.

![List node](/images/6-quan-ly-log-tap-trung-elk-phan-1-tren-docker-swarm/2-init-swarm.png)

Kết quả ở trên cho ta 1 command để các node khác có thể join vào cluster.

### 2.3. Join các node còn lại vào cluster

Ta lần lượt ssh vào từng node và chạy lệnh join ở trên.

```bash
docker-machine ssh node2
docker swarm join --token SWMTKN-1-5dk4ysohsctxbifjmb6x8xa5i6qps81au5k4fmzzqmne3l3oz0-ac99bh5vrw8zm3lychgpu3cwh 192.168.99.104:2377
```

```bash
docker-machine ssh node3
docker swarm join --token SWMTKN-1-5dk4ysohsctxbifjmb6x8xa5i6qps81au5k4fmzzqmne3l3oz0-ac99bh5vrw8zm3lychgpu3cwh 192.168.99.104:2377
```

> Chú ý: `token` trong lệnh `docker swarm join` sẽ khác nhau ở mỗi lần chạy. Nên chỗ này đừng copy của mình nhé. Lỗi đấy!

## 3. Deploy ELK lên docker swarm

SSH vào node manager `docker-machine ssh node1`.

Tạo 1 file `docker-stack.yml` có nội dung như sau:

```yaml
version: '3'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
    ports:
      - "9200:9200"
    volumes:
      - esdata:/usr/share/elasticsearch/data
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"
      ELASTIC_PASSWORD: "changeme"
      discovery.type: single-node
    deploy:
      placement:
        constraints: [node.role == manager]

  logstash:
    image: docker.elastic.co/logstash/logstash:7.8.0
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    depends_on:
      - elasticsearch
    deploy:
      placement:
        constraints: [node.role == manager]

  kibana:
    image: docker.elastic.co/kibana/kibana:7.8.0
    environment:
      ELASTICSEARCH_HOSTS: 'http://elasticsearch:9200'
    ports:
      - "5601:5601"
    depends_on:
      - logstash
    deploy:
      placement:
        constraints: [node.role == manager]

  logspout:
    image: gliderlabs/logspout:v3
    command: 'udp://logstash:5000'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - logstash
    deploy:
      mode: global
      restart_policy:
        condition: on-failure
        delay: 30s

  visualizer:
    image: dockersamples/visualizer:stable
    ports:
      - "8081:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
    labels:
      app: "visualizer"
    deploy:
      placement:
        constraints: [node.role == manager]

volumes:
  esdata:
    driver: local
```

Tạo 1 file `logstash.conf` có nội dung như sau:

```
input {
   udp {
   		port => 5000
   		codec => json
   	}
}

## Add your filters / logstash plugins configuration here

filter {
  if [docker][image] =~ /logstash/ {
    drop { }
  }
}

output {
    elasticsearch {
      hosts => "elasticsearch:9200"
      user => "elastic"
      password => "changeme"
    }

    stdout { codec => rubydebug }
}
```

Tiến hành deploy

```bash
docker stack deploy -c docker-stack.yml elk
```

Thường mình sẽ gặp 1 lỗi là logspout sẽ không kết nối được đến logstash, do logstash chưa init xong. Bạn hãy đợi 1 thời gian rồi chạy `docker service update --force elk_logspout` để restart lại logspout trên các node.

## 4. Xem log từ Kibana

Dùng browser truy cập vào 1 node bất kỳ trong swarm với port `5601`

![List node](/images/6-quan-ly-log-tap-trung-elk-phan-1-tren-docker-swarm/1-list-node.png)

Trong trường hợp của mình, mình sẽ truy cập vào `http://192.168.99.104:5601`.

![Kibana](/images/6-quan-ly-log-tap-trung-elk-phan-1-tren-docker-swarm/3-kibana-1.jpg)

Tạo 1 index pattern `logstash*`
![Kibana](/images/6-quan-ly-log-tap-trung-elk-phan-1-tren-docker-swarm/4-kibana-2.jpg)

![Kibana](/images/6-quan-ly-log-tap-trung-elk-phan-1-tren-docker-swarm/5-kibana-3.jpg)

![Kibana](/images/6-quan-ly-log-tap-trung-elk-phan-1-tren-docker-swarm/6-kibana-4.jpg)

![Kibana](/images/6-quan-ly-log-tap-trung-elk-phan-1-tren-docker-swarm/7-kibana-5.jpg)

![Kibana](/images/6-quan-ly-log-tap-trung-elk-phan-1-tren-docker-swarm/8-kibana-6.png)

Oke! Tới đây giao diện để truy xuất log đã hiện ra. Bạn có thể filter log theo thời gian, theo app,... Chỗ này bạn có thể tự mò nhé, giao diện kibana cũng khá dễ sử dụng.

Cùng nhau chia sẻ kinh nghiệm, mò mẫm kiến thức mới

Hlog