Kubernetes – nginx with virtualhosts on 3 nodes bare metal configuration

Hello everybody!

This is a simple but cool exercise you can do in order to better understand Kubernetes and it’s configuration.

Scenario: we want a persistent Volume on an external nfs storage (we call it storage1 inside the configurations. Remember to set your etc/hosts file in order to be resolve the address or just use the ip).

This is our first file, nfs_pv.yaml. It creates a new persistent volume on the nfs storage, on the path “/mnt/share” as you can see in the following code:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs
spec:
  capacity:
    storage: 50Gi
  accessModes:
    - ReadWriteMany
  storageClassName: my-local-storage
  nfs:
    server: storage1
    path: "/mnt/share/"

It’s very simple!!. Storage1 is the name resolution of my external nfs (set it on /etc/hosts file!).

Now we need a claim on this persistent volume. Just edit pvc.yaml file and insert the following code:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: my-claim
spec:
  accessModes:
  - ReadWriteMany
  storageClassName: my-local-storage
  resources:
    requests:
      storage: 50Gi

Now you should have two files inside you directory. Launch the command:

kubectl get pv
kubectl get pvc

You can control that the status of the volume and its claim are bound

Create a service that is responsible to give us network access to the pods that we are going to create. Let’s call this file as svc.yaml.

apiVersion: v1
kind: Service
metadata:
   name: www-np
spec:
   type: NodePort
   ports:
   - port: 80
     targetPort: 80
   selector:
     app: www2

We are now able to use port 80 to match the port 80 on the pod with nginx.

Create now the nginx deployment (deployment.yaml):

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: www2
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: www2
    spec:
      containers:
        - name: www2
          image: nginx:alpine
          ports:
            - containerPort: 80
          volumeMounts:
            - name: www-persistent-storage
              mountPath: /usr/share/nginx/html
            - name: directory-data
              mountPath: /etc/nginx/sites-enabled/            
            - name: nginx-config
              mountPath: /etc/nginx/ 
            

      volumes:
        - name: www-persistent-storage
          persistentVolumeClaim:
            claimName: my-claim
        - name: nginx-config
          configMap:
            name: confnginx

        - name: directory-data
          configMap:
            name: directory-data

explanation:
– /usr/share/nginx/html will be mapped on the nfs storage (/mnt/share/). the “www-persistent-storage” is related to the claimName “my-claim” that is linked to the nfs folder.
– The folder /etc/nginx/sites-enabled/ will be created by the deployment and it’ll contain directory-data (configMap that we’ll explain in a while).
– “replicas: 2” is the number of pod we want to create. They will balance requests on the web server.

We need to change two configurations for the virtualhost to work. The first one is Site-enabled folder which contains all the nginx.conf files of our virtualhosts. The other configuration is the nginx.conf to be saved in /etc/nginx on the pod. We used the configMaps in order to achive this.
We put inside a folder (“data” in this case) all the configuration we want to enable in our nginx installation.

kubectl create configmap directory-data --from-file=data/

This command will take all the files in the directory and save them in the configmap directory-data. In the deployment file we configured this part:

            - name: directory-data
              mountPath: /etc/nginx/sites-enabled/   

this allow us to save all the files of our virtualhosts in the directory “/etc/nginx/sites-enabled/

we are doing the same with confnginx


kubectl create configmap confnginx --from-file=nginx.conf

check the configmaps with this command:

# kubectl get configmaps
NAME             DATA   AGE
confnginx        1      122m
directory-data   2      146m

It’s possible to recover the content of the configmaps files. Here’s the command that allow you to read the configuration files (we’re offer to you the whole code):

# kubectl get configmap directory-data
NAME             DATA   AGE
directory-data   2      154m
[root@k8s-master boh_nginx]# k get configmap directory-data -o yaml
apiVersion: v1
data:
  example.com.conf: |
    server {
        listen       80;
        server_name  example.com www.example.com;

        #charset koi8-r;
        #access_log  /var/log/nginx/host.access.log  main;

        location / {
            root /usr/share/nginx/html/example.com/;
            index  index.html index.htm;
            try_files $uri $uri/ =404;
        }

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ \.php$ {
        #    proxy_pass   http://127.0.0.1;
        #}

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ \.php$ {
        #    root           html;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #    deny  all;
        #}
    }
  example2.com.conf: |
    server {
        listen       80;
        server_name  example2.com www.example2.com;

        #charset koi8-r;
        #access_log  /var/log/nginx/host.access.log  main;

        location / {
            root /usr/share/nginx/html/example2.com/;
            index  index.html index.htm;
            try_files $uri $uri/ =404;
        }

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ \.php$ {
        #    proxy_pass   http://127.0.0.1;
        #}

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ \.php$ {
        #    root           html;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #    deny  all;
        #}
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2019-08-13T10:14:33Z"
  name: directory-data
  namespace: default
  resourceVersion: "478167"
  selfLink: /api/v1/namespaces/default/configmaps/directory-data
  uid: 35f084a6-03ce-4a86-903e-2479a1d0bcce

As you can see we have example.com and example2.com files!
The following is the nginx.conf taken from the configmap:

# kubectl get configmap confnginx -o yaml
apiVersion: v1
data:
  nginx.conf: |
    user  nginx;
    worker_processes  1;

    error_log  /var/log/nginx/error.log warn;
    pid        /var/run/nginx.pid;


    events {
        worker_connections  1024;
    }


    http {

        default_type  application/octet-stream;

        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';

        access_log  /var/log/nginx/access.log  main;

        sendfile        on;
        #tcp_nopush     on;

        keepalive_timeout  65;

        #gzip  on;




        include /etc/nginx/sites-enabled/*.conf;
        server_names_hash_bucket_size 64;


    }
kind: ConfigMap
metadata:
  creationTimestamp: "2019-08-13T10:38:09Z"
  name: confnginx
  namespace: default
  resourceVersion: "480437"
  selfLink: /api/v1/namespaces/default/configmaps/confnginx
  uid: dc4278c3-6d62-4b6e-8bae-cde3d39b6748

From the folder that contains all these file we wrote during the tutorial lauch:

# kubectl apply -f ./

this will deploy all the configuration we made!

Don’t forget to set the hosts file to match the ip address of your k8s installation.
You need to launch this command if you’re using a baremetal setup in order to see which port is mapped on the nginx installation:

open your browser on:
www.example.com:30502

www.example.com:30502
www.example2.com:30502

enjoy!