How to Setup a Scalable Ultra Low Latency Streaming Cluster in AWS

In this blog post, we’re going to tell how to set up a Scalable Ant Media Server Cluster in Amazon Web Services. This solution really cut the costs for projects that need scalable ultra-low latency streaming. Let me explain how. Assume that your project needs scalable ultra-low latency streaming. There are a few options to start.

1. Implement your own ultra-low latency solution.
2. Use one of the Platform as a Service (PaaS) solutions.
3. Use a scalable software solution that you can run on your own cloud. You’ll just use your own Infrastructure as a Service (IaaS) to create your streaming service.

Actually, the first option is something like inventing the wheel again. It likely takes a long time according to your expertise.  The second option is ok if you have a good amount of budget. The third option seems best in terms of low cost and full control. It just requires some experience in the cloud.

You can also estimate your cost and server requirement through our cost calculator and down the page you can see the table for supported values as per server CPU resources.

Infrastructure as a Service is already rising

When you have your own scalable ultra-low latency streaming service, then you can take advantage of several things. For instance, you likely have an agreement with your Infrastructure as a Service (IaaS) provider(AWS, Azure,…) about some discounts. So that you can use these discounts for your streaming service as well.  Another option is that you will have full control over scaling your streamers and viewers. In other words, Infrastructure as a Service is already rising. Let it rise for you.

Let’s get started on how to start your own scalable ultra-low latency streaming service on AWS.

Getting Started for Ultra-Low Latency Streaming Service on AWS

Here is the system outline

Let me give some brief definitions.

• Load Balancer:  LB is the entrance point for the publishers and players. Load Balancer accepts the requests from publishers or players and forwards the requests to the available node in the cluster.
• Origin Auto-Scalable Group: Nodes(Instances) in the origin group accept the publish requests and ingest the incoming WebRTC stream. When an origin instance accepts a WebRTC stream, it saves the related information to the MongoDB Database Server.
  There may be one node or multiple nodes in the origin group. It may even be manually or auto-scalable. In our deployment, it’s auto-scalable in AWS.
• Edge Auto-Scalable Group: Node(Instances) in the edge group accepts the play requests. Then it learns from MongoDB which origin node has the related stream. After that, it gets the stream from the related origin node and sends the stream to the player.
• MongoDB Database Server: Ant Media Server uses MongoDB in clustering. Streams information is saved to MongoDB so that edge instances can learn any stream’s origin node.

Right now let’s start with installing MongoDB Server

Step 1: Install MongoDB Server

The procedure below shows how to start an instance in the AWS EC2 service as well. In other words, if you have no experience with AWS, you can even install MongoDB Server as follows. If you know how to start an instance in AWS, just skip to “Install MongoDB to Your Instance”

• Signup to AWS if you don’t have an account yet.  Login to AWS Management Console. Then click EC2 Service as shown in the image below.

• Click “Launch” Instance.

• Search for “Ubuntu” and Select “Ubuntu 20.04”.

• Choose Instance Types like m4.xlarge or m5.xlarge series. There are two points here.
  • The first one is you may optionally choose a bigger instance according to your streaming load.
  • The second one doesn’t use any m5a instances because they have ARM architecture.

Then click “Review and Launch”.

• Click “Configure Security Group” in the image.
• Add “22” and “27017” TCP ports as follows in the image. The warning is critical for security. We’ll restrict the source into a VPC later. Just click “Review and Launch”.

• In the coming window, it will ask to specify the key file. Choose “Create new key pair” and click the “Download Key Pair” button. After the key file is downloaded click “Launch Instances”.

• Right now, your instances should be launching as shown in the image.

• Go to EC2 Instances and Click the “Connect” button.

• It shows a dialog as follow and connect to the instance via ssh

• • Right now, you should connect to your instance. To Connect your instance, open a terminal and run a command something like below. Please change {YOUR_KEY_FILE} and {INSTANCE_PUBLIC_IP} with your own credentials. For our case, they are “ant.pem” and “3.108.40.66”.

  ssh -i {YOUR_KEY_FILE} ubuntu@{INSTANCE_PUBLIC_IP}

Install MongoDB to Your Instance

• After you get connected, run the following commands in order to install MongoDB to your instance

$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
$ echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
$ sudo apt-get update
$ sudo apt-get install -y mongodb-org

• Open /etc/mongod.conf file with an editor like nano and change bind_ip value to 0.0.0.0  to let MongoDB accept connections in all interfaces and save it.

sudo nano /etc/mongod.conf

Press “Ctrl + X” to save the file.

• Restart mongod and enable service.

sudo service mongod restart
sudo systemctl enable mongod.service

MongoDB installation is complete, just save your MongoDB instance’s local address somewhere. We will use it later.

Step 2: Install Scalable Origin Group

• Click “Auto Scaling > Launch Configurations” and Click “Create Launch Configurations”

• You can see the name field just under the Create Launch Configuration header. Give a name something like “OriginGroup”.
• In Launch configuration you need to search AMI of Ant media server using image Id as per your AWS region. For example, we are using ap-south-1 image Id in our cluster as shown in the below image.

• Choose instance type, in our sample we choose c5.xlarge. You can choose any instance type according to your project and after proceed to the next step.

• In the coming window, as shown in the image below, We need to give name and set User data.
  • Then click the “Advanced Details” title. You will see the “User data” text area. Right now, copy the text below, change the “{MongoIP}” field with the MongoDB IP Address in the script and paste it to the “User data”.
  • After that Click “Skip to review”

#!/bin/bash
cd /usr/local/antmedia
./change_server_mode.sh cluster {MongoIP}

The form should be something like below

• Now you have to create a new security group for Auto scaling group in which the below ports need to be whitelisted as shown in the image.

Important Note: You need to open the TCP port range(TCP:5000-65000) for the internal network as well so that streams can be played from each node. It should not be open to the public.

• Click “Create Launch Configuration”.
• After the launch configuration is created successfully, go to Auto Scaling Groups in the EC2 section and create Auto Scaling Group.

• Give a name to scaling group. We give “AMS-Origin-Group” as a name and then Switch to launch configuration as by default it is selected to Launch template. Select your launch configuration group that you have created earlier for the origin group as shown in the below image.

• Choose “ap-south-1a” subnet. We choose only one subnet to let all instances appear in the same subnet for having better connectivity.

• In Configure advance options you need to select the existing load balancer option that we will create later to attach auto scaling groups with it.

• Choose your scaling policy. In our sample below, our origin group will scale up to maximum of 10 instances by providing Average CPU Utilization with %60. Then Click Next and Next.

• Lastly, the review screen will come and click the “Create Auto Scaling group”.

Step 3: Install Scalable Edge Group

Installing scalable edge group almost same as scalable origin group. Please go to Step 2 again and follow the same steps one more time. Just don’t forget to change naming (for instance give group name as Edge Group) and configure scaling policy and instance type according to your needs. If you have any questions or problems with this, please let us know through support@antmedia.io.

Step 4: Install Load Balancer

• Click the “Load Balancing > Load Balancers” on EC2 Service and Click the “Create” button under Application Load Balancer.

• Give a name to your Load Balancer and the basic configuration should be like in the image below and choose ap-south-1a and ap-south-1b for availability zones.

• Now we need to choose a load balancer security group which we will create by clicking create new security group option.

• Before moving further in Load balancer configuration we need to create target groups for both Origin & Edge and forward with HTTP through 5080 port. In the Register Targets group, do nothing, just proceed because we bind target later. In the below example, we have created the origin target group and the same for Edge will be created.

• After creating Target Groups, again go to EC2 > Target Groups > Edit attributes and change the Load Balancing algorithm for Edge and Origin as below.

• Now continue load balancer configuration, choose both HTTP and HTTPS by clicking “Add listener”. The port settings should be like in the image below and we need to bind target groups now to forward requests to origin & edge target groups. Also in Auto scaling groups advance configuration please attach the created load balancer now.

• Now Choose your domain certificate in Secure listener settings for secure streaming (If you don’t know how to create a certificate for ACM, please follow this guide and create a load balancer. Also, don’t forget to add CNAME for your load balancer. For instance, every load balancer has a DNS name like “xxxx.ap-south-1.elb.amazonaws.com” so you need to add CNAME for your subdomain that points to your load balancer address.

Right now Everything is ok. Just let me give a piece of brief information about the difference between publishing and playing. In our load balancer configuration, we forward HTTP(80) and HTTPS(443) to Origin Group and we forward HTTP(5080) and HTTPS(5443) to Edge Group. It means that we should connect 80 or 443 ports to publish and connect 5080 or 5443 to play streams. Otherwise, play requests go to the origin group, and publish request goes to the edge group and it’s likely to create some performance issues according to your configurations.

Logging in Ant Media Server Web Panel

You can log in to the web panel via the https://your-domain-name/ and login with “JamesBond” and the first instances instance-id in your origin group. If you don’t know the instance-id, you need to change your password.

We are storing passwords with MD5 encryption in the latest version. You can encrypt your password basically as follows.

On the terminal program

echo -n 'new-password' | md5sum

or any MD5 encrypter page like: https://www.md5online.org/md5-encrypt.html

Please ssh to your MongoDB instance and write the below commands via terminal

$ mongo
> use serverdb
> db.getCollection('User').find()
> db.User.updateOne({"_id": "5e978ef3c9e77c0001228040"}, {$set:{password: "md5Password"}})

It gives you an output like this

{ "_id" : ObjectId("5e978ef3c9e77c0001228040"), "className" : "io.antmedia.rest.model.User", "email" : "JamesBond", "password" : "e4e6ca42342f95978a17c6257593c1e1", "userType" : "ADMIN" }

Enable IP Filtering

Please visit How to enable IP filter behind a load balancer?

Test Flight

For publishing please visit the https://your-domain-name/WebRTCAppEE/ and click the “Start Publishing” button. The default stream id is “stream1”
For playing please visit the https://your-domain-name:5443/WebRTCAppEE/ and click the “Start Playing” button. The default stream will be played

As you figure out, we connect the default HTTPS port(443) for publishing and the 5443 port for playing. Because we configure the load balancer to forward default port(443) to origin group and 5443 to edge group.

I hope this guide will help some guys to get started in clustering. It’s good to understand the general picture. Later, we have some plans to provide easier ways to install cluster in AWS. You can also have a look at the blog post about the Global Cluster setup. If you have any questions, please let us know through contact@antmedia.io