We are starting a series of blog posts where we will share how Ant Media Server is scaled up to 100K viewers step by step. Let’s start with 10K as the first step 🙂 In some use cases such as trivia quiz apps, webinars, and conference room applications; you may need Ultra-Low Latency using WebRTC with a large number of viewers and publishers and therefore you may need to scale up your solution to support 10K viewers which could be achieved with Ant Media Server scaling solution.
Today I’m going to show you how easy to perform a load test with 10K viewers which is just the beginning of our new blog series. In the future, we are going to publish new blogs about our load tests supporting an increased number of viewers that will go up to 100K.
In order to monitor the test, you need to have some tools as a prerequisite. We are going to explain how to use those tools step by step. By performing these steps you can also perform the test by yourself.
What do we need?
- 1 x Load Balancer (Nginx)
- 1 x Database server (MongoDB)
- 10 x Load Servers
- 20 x Ant Media Server Edge Cluster
- 1 x Ant Media Server Origin
- 1 x Kafka Server
- 1 x Monitoring Tool
- Test Tool
Now let’s see each item in detail.
Load Balancer
Since Nginx doesn’t require much from the system resources so we have used a server with 4 Cores. We are going to utilize Nginx for load balancing and use its round-robin mechanism to distribute the load to Ant Media Edge servers.
The configuration is as follows.
user nginx;
worker_processes auto;
pid /var/run/nginx.pid;
worker_rlimit_nofile 1048576;
events {
worker_connections 1048576;
multi_accept on;
use epoll;
}
http {
#Ant Media Edge
upstream antmedia_edge {
server edge_ip1:5080;
server edge_ip2:5080;
server edge_ip3:5080;
.
.
.
server edge_ip20:5080;
}
sendfile on;
tcp_nopush on;
tcp_nodelay on;
server_tokens off;
keepalive_timeout 300s;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# ssl settings
ssl_protocols TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;
# logs settings
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"'
'"$hostname" "upstream: $upstream_addr"';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log;
# gzip
gzip on;
gzip_disable "msie6";
gzip_http_version 1.1;
gzip_comp_level 6;
gzip_types text/plain text/css application/json application/javascript text/javascript application/x-javascript text/xml application/xml application/xml+rss application/vnd.ms-fontobject application/x-font-ttf font/opentype font/x-woff image/svg+xml image/x-icon;
# proxy settings
proxy_redirect off;
proxy_http_version 1.1;
proxy_read_timeout 10s;
proxy_send_timeout 10s;
proxy_connect_timeout 10s;
#redirect all http requests to https
server {
listen 80 default_server;
server_name _;
return 301 https://$host$request_uri;
}
#Edge Configuration
server {
listen 5080;
location / {
proxy_pass http://antmedia_edge;
proxy_http_version 1.1;
proxy_connect_timeout 7d;
proxy_send_timeout 7d;
proxy_read_timeout 7d;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
}
}
Database Server
In order to make Ant Media Server to work as in cluster mode, we need to install MongoDB and apply necessary configuration. You can access the setup guide from this link.
Load Servers
Load servers are configured to act as the viewers eventually the load itself. All the load will be created by using these instances and we will distribute the load to edge servers using load balancer. Based on our design, load servers are configured to get the stream from Edge Servers with the bit rate of 600 KBps and 480p where the adaptive bit rate by default is off and eventually, each load server is going to act as 1000 viewers to get 1000 streams.
Ant Media Server Cluster on AWS
After finishing the installation on AWS for a single instance, we took the image of that instance and using same image, we have set up other instances all at once. In addition, we preferred to use C5.2xlarge instances. Check this post for a detailed guide about setting up scalable ultra low latency streaming cluster on AWS.
How the test tool works
We run this test tool on Load instances. We send the broadcast via the FFmpeg tool, usage is as follows:
ffmpeg -re -i test.flv -codec copy -f flv rtmp://Ant_Media_Origin_IP/WebRTCAppEE/stream1
You can use the link below to download and install the test tool.
wget https://mirror.antmedia.io/webrtctest-release.zip apt update apt install unzip openjdk-8-jre unzip webrtctest-release.zip cd webrtctest-release
Test tool usage
bash run.sh -m player -n 1 -i stream1 -s nginx_load_balancer_ip -u false -k kafka_server_ip:9092
Kafka Configuration
Kafka collects the information from each server and sends them to the monitoring server. Before taking the instance image on AWS, configuring Kafka will ease the work.
You can review the document for Kafka installation from here.
Kafka settings are as follows.
vim /usr/local/antmedia/conf/red5.properties server.kafka_brokers=your_kafka_server_ip:9092 service antmedia restart
Install Grafana
We have used the most suitable monitoring tool for 10K testing, Grafana. On the backend, there is also Elasticsearch and Logstash working with Grafana.
Please refer to this link for installation.
Observation
Based on our observation, Ant Media Servers handled the capacity of 10K WebRTC viewers without any quality degradation and the average CPU load was around 80% and Memory usage was around 50%.
Our next blog will be about how Ant Media Server will handle 30K, 50K, and 100K. Stay tuned!
If you have any questions, please just drop a line to contact (at) antmedia.io
You may want to also check out Mass Broadcasting Made Easy with Ant Media WebRTC Solution – 70K Unlocked!
