You might have wondered what is the easiest way to implement logging and monitoring in kubernetes. I have recently implemented kubernetes cluster logging and monitoring using EFG stack (Elastic search, Fluentbit and Grafana). Here I will try to explain and give source code for implementation of EFG stack for your K8's cluster.
Fluent Bit is an open source and multi-platform Log Processor and Forwarder which allows you to collect data/logs from different sources, unify and send them to multiple destinations. It's fully compatible with Docker and Kubernetes environments.
Fluent Bit must be deployed as a DaemonSet so that it will be available on every node of your Kubernetes cluster.
Fluentbit will export the metrics from all the pods and it will create a new index in elastic search instance with prefix -{{ env "ENVIRONMENT" }} following by the current date.
In order to run fluentbit it will need RBAC authorization to it will first create Clusterrole and Clusterrolebinding. Please make sure the google account from which you are running has sufficient permission to create new Clusterrolebinding.
Env variables for fluentbit :
Envirnment variables are implented in deploy.sh file. Please make sure you have implemented correct values before running the script.
Usage :
$ ./deploy.sh
Fluentbit to grafana :
Once the index in elastic search is reflected you can import the elastic search index in grafana and create metrics.
EmoticonEmoticon