How to Easily Monitor Ant Media Server Instances in 3 Steps?

Before starting the blog post, I just want to thank you all people in the community that makes Ant Media Server Instances run in 120 countries.

So we listen our users and one of the common question we receive is how to Monitor Ant Media Server instances. It’s really a crucial thing when you have lots of instances running for your media streaming service. So we’ve implemented some monitoring methods to make monitoring Ant Media Server instances easy and adaptable for any system.
How It Works

For Ant Media Server side, it’s just an event producer for Apache Kafka. Ant Media Server sends instance and stream stats to Apache Kafka for every 15 seconds. You can consume Apache Kafka events with any other tool or pipeline. For our case, we used the pipeline below for collecting and visualization of the stats.

1. Ant Media Server sends events to Apache Kafka
2. Logstash consumes Apache Kafka Events and writes them to Elastic Search
3. Grafana collects the stats from Elastic Search and visualize them.

So in this blog post, we show up how to install the pipeline above to have a turnkey monitoring solution for Ant Media Server Instances.
Let’s get started by installing the components below:

• Install Apache Kafka
• Install Elasticsearch and Logstash
• Install Grafana

Install Apache Kafka

Apache Kafka is useful for building real-time streaming data pipelines to get data between the systems or applications.

1. Install Java because Apache Kafka requires Java.

   apt-get update && apt-get install openjdk-8-jdk -y
   

2. Download the Apache Kafka and then extract the archive file

   wget -qO- https://archive.apache.org/dist/kafka/2.2.0/kafka_2.12-2.2.0.tgz | tar -zxvf- -C /opt/ && mv /opt/kafka* /opt/kafka
   

3. Edit server.properties vim /opt/kafka/config/server.properties file as below.

   listeners=PLAINTEXT://your_server_ip:9092
   

4. Start Apache Kafka

   /opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties &
   /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties &
   

   Firstly, we’ve started ZooKeeper because Kafka needs ZooKeeper and then we’ve started Kafka.

5. Check if it’s working by running the command below

   netstat -tpln | egrep "9092|2181"
   

   if you see that the ports(9092 and 2181) are in listening mode, it means yeah it’s working. You’ve finished the first installation.

Run Apache Kafka as a systemd Service.

Running Apache Kafka as a systemd service will let us manage Apache Kafka services with systemctl commands. Follow the instructions below:

• Create systemd unit file for Apache Kafkavim /lib/systemd/system/kafka.service
• Copy and paste the below content into the kafka.service you’ve created above. Make sure that you set the correct JAVA_HOME path for your system in the content below

  [Unit]
  Description=Apache Kafka Server
  Requires=network.target remote-fs.target
  After=network.target remote-fs.target kafka-zookeeper.service
  
  [Service]
  Type=simple
  Environment=JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
  ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
  ExecStop=/opt/kafka/bin/kafka-server-stop.sh
  
  [Install]
  WantedBy=multi-user.target
  

• Create systemd unit file for Zookeepervim /lib/systemd/system/kafka-zookeeper.service
• Copy and paste the below content to the kafka-zookeeper.service file you’ve created above.

  [Unit]
  Description=Apache Zookeeper Server
  Requires=network.target remote-fs.target
  After=network.target remote-fs.target
  
  [Service]
  Type=simple
  Environment=JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
  ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
  ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
  
  [Install]
  WantedBy=multi-user.target
  

• Enable and reload the systemd daemon to apply new changes.

  systemctl enable kafka-zookeeper.service
  systemctl enable kafka.service
  

• Start Apache Kafka with  systemctl

  systemctl start kafka-zookeeper.service
  systemctl start kafka.service
  

Kafka Settings for Ant Media Server

If you want to monitor Ant Media Server, you need to set the IP address of your Apache Kafka in AMS_INSTALLTION_DIR/conf/red5.properties file.

• Open the following line with the editorvim /usr/local/antmedia/conf/red5.properties
• Edit the following line server.kafka_brokers=ip_address:port_number Replace ip_address:port_number with Apache Kafka IP Address and port number. Example: server.kafka_brokers=192.168.1.230:9092
• Restart Ant Media Server: service antmedia restart
• Check if it’s working. When you run the following command on Kafka server, if there is data flow, this means everything is configured properly./opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.230:9092 --topic ams-instance-stats --from-beginning Output should be something like below:

  {"instanceId":"a06e5437-40ee-49c1-8e38-273544964335","cpuUsage":
  {"processCPUTime":596700000,"systemCPULoad":0,"processCPULoad":1},"jvmMemoryUsage": 
  {"maxMemory":260046848,"totalMemory":142606336,"freeMemory":21698648,"inUseMemory":120907688},"systemInfo": 
  {"osName":"Linux","osArch":"amd64","javaVersion":"1.8","processorCount":1},"systemMemoryInfo":
  ...
  

Some Useful Apache Kafka Commands

• List all topics/opt/kafka/bin/kafka-topics.sh --list --bootstrap-server your_kafka_server:9092Example:

  /opt/kafka/bin/kafka-topics.sh --list --bootstrap-server 192.168.1.230:9092
  ams-instance-stats
  ams-webrtc-stats
  kafka-webrtc-tester-stats
  

• Monitor messages for a specific topic with Kafka Consumer as we’ve used above/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.230:9092 --topic ams-instance-stats --from-beginning

Install Elasticsearch and Logstash

Install Elasticsearch

1. Import GPG key and Repo

   wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
   sudo apt-get install apt-transport-https
   echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
   

2. Update package lists and install elastic search

   apt-get update && apt-get install elasticsearch
   

3. Enable and start elasticsearch service

   systemctl enable elasticsearch.service
   systemctl start elasticsearch.service
   

Install Logstash

Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it and then sends it to a “stash” like Elasticsearch

1. Update your package lists, then install logstash with the following command

   apt-get update && apt-get install logstash
   

2. Enable logstash service

   systemctl enable logstash.service
   

3. Configure logstash. Create /etc/logstash/conf.d/logstash.conf file and add below content. Please don’t forget to replace kafka_server_ip and make sure elasticsearch_ip is correct.

   #kafka
   input {
     kafka {
       bootstrap_servers => "kafka_server_ip:9092"
       client_id => "logstash"
       group_id => "logstash"
       consumer_threads => 3
       topics => ["ams-instance-stats","ams-webrtc-stats","kafka-webrtc-tester-stats"]
       codec => "json"
       tags => ["log", "kafka_source"]
       type => "log"
     }
   }
   
   #elasticsearch
   output {
     elasticsearch {
        hosts => ["127.0.0.1:9200"] #elasticsearch_ip
        index => "logstash-%{[type]}-%{+YYYY.MM.dd}"
     }
     stdout { codec => rubydebug }
   }
   

4. Save and close the file, then restart logstash service

   systemctl restart logstash
   

Test Elasticsearch and Logstash Configuration

You can test that Elasticsearch and Logstash are working correctly with the command below.
curl -XGET 'localhost:9200/_cat/indices?v&pretty'

Example output:

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open logstash-log-2020.03.23 mf-ffIHBSNO4s7_YoUr_Rw 1 1 1300 0 527.5kb 527.5kb


Install Grafana

Grafana is an open source metric analytics & visualization suite.

1. In order to install Grafana Server, run the following commands.

   sudo apt-get install -y software-properties-common wget apt-transport-https
   wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
   sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
   sudo apt-get update && sudo apt-get install grafana
   

2. Enable and start grafana server

   systemctl enable grafana-server
   systemctl start grafana-server
   

Configure Grafana

1. Login to Web panel(http://your_ip_address:3000/login) through your web browser. Default username and password is admin/admin
2. Click Add data source 
3. Select Elasticsearch 
4. Set setting as below

   URL : http://127.0.0.1:9200
   Index name: logstash-
   Time filed name: @timestamp
   Version: 7.0+
   

    

5. Click New dashboard 
6. Click Add Query 
7. Choose whatever you want to monitor.Query: ElasticSearch 
8. Ant Media Example Dashboard 

If you would like to use the same dashboard, you can download it from the below link.

https://mirror.antmedia.io/antmedia-dashboard.json

Bonus: Create Telegram Alert

You can define an alert in Grafana and it can notify you when specific events happen

1. Create a New Bot
   • Open Telegram and search for @BotFather user and send the following command.

     /newbot
     

   • When you create a new bot, you will get response like below.

     Use this token to access the HTTP API:
     1254341629:AAHYHhJK8TgsUXa7jqBK7wU1bJ8hzWhUFzs
     Keep your token secure and store it safely, it can be used by anyone to control your bot.
     

2. Create a Channel and retrieve the channel’s chat ID.
   • Create a channel in telegram and Invite your bot as an admin
   • Send a test message and Get the Chat ID
   • Use cURL or just visit the url below with your browser. Don’t forget to replace the access token

     https://api.telegram.org/bot{USE_YOUR_ACCESS_TOKEN}/getUpdates
     

   • You will get the lines like below.

     {"ok":true,"result":[{"update_id":222389875,
     "channel_post":{"message_id":2,"chat": 
     {"id":-1001181377238,"title":"test","type":"channel"},"date":1587016720,"text":"test"}}]}
     

   • Save the id number: -1001181377238 because we’ll need it in the next step

Configure Grafana Notification

We have configured the chatbot above. Now let’s start to configure Grafana Notification.

1. Login to Grafana web panel via http://your_grafana_server:3000
2. Click Alerting / Notification Channel 
3. Add New Channel 
4. Configure it as shown in the below screenshot 

   Name : name_of_your_notification.
   Type : Telegram
   Bot Api Token: your_bot_token_id
   Chat ID: your_channel_id
   

   If you click on the Send Test and there is a message on the telegram, everything is fine.

   

   Now you’ve set up notifications as you wish.

This is the whole setup of monitoring Ant Media Server. If you have any questions, please just drop a line to contact (at) antmedia.io