Ant Media Server (AMS) users are mostly people from Product Engineering teams, and IT departments, and most of them are also software developers. So we have received lots of requests for plugin mechanisms. We have been working on a mechanism that lets AMS users integrate their extensions with AMS. In this blog post, I will tell the interaction points ie. and interfaces between the AMS and plugins. I will also want to tell these on a sample plugin project.
Interaction Points (Interfaces and methods)
In this part, I will introduce the interactions (interfaces and methods) between the AMS and the plugin. This interface has methods that have parameters from FFmpeg types. You may check our FFmpeg blog post for a first insight.
Plugin Data Flow
From AMS to Plugin Interaction
There are two main interfaces to get decoded video frames or encoded video packets from the AMS.
IPacketListener Interface
Encoded packets and stream properties are sent to the plugin by this interface. In other words, you should implement this interface and register your concrete object to capture stream properties and also packets from AMS.
Methods of IPacketListener:
- AVPacket onPacket(String streamId, AVPacket packet)
packets are sent to the plugin with this method. A packet may be a video or audio packet. - void writeTrailer()
called while stream closing - void setVideoStreamInfo(String streamId, StreamParametersInfo videoStreamInfo)
video stream properties are sent to the plugin with this method. - void setAudioStreamInfo(String streamId, StreamParametersInfo audioStreamInfo)
audio stream properties are sent to the plugin with this method.
IFrameListener Interface
Decoded frames and stream properties are sent to the plugin by this interface. In other words, you should implement this interface and register your concrete object to capture stream properties and also frames from AMS.
Methods of IFrameListener:
- AVFrame onAudioFrame(String streamId, AVFrame audioFrame)
audio frames are sent to the plugin with this method. - AVFrame onVideoFrame(String streamId, AVFrame videoFrame)
video frames are sent to the plugin with this method. - void writeTrailer()
called while stream closing - void setVideoStreamInfo(String streamId, StreamParametersInfo videoStreamInfo)
video stream properties are sent to the plugin with this method. - void setAudioStreamInfo(String streamId, StreamParametersInfo audioStreamInfo)
audio stream properties are sent to the plugin with this method.
Use cases and onFrame return values:
The return parameters may be different according to the plugin use case. Now I will tell which values can be returned according to three type of use cases. I will tell it for onVideoFrame but it is also same for onAudioFrame.
onVideoFrame method has such a signature.
AVFrame onVideoFrame(String streamId, AVFrame videoFrame)
AVFrame is passed to the method and also it returns AVFrame.
The returning value is very important for the AMS execution. There may be 3 different use cases:
1. Asynchronous case:
Your plugin only gets the video frame and works on that video frame but doesn’t feed the manipulated frame to the server again. In this case, the work should be in a different thread and onVideoFrame should return the same AVFrame that passed to the method.
For example: you may have such a plugin that only collects the frame for statistics
Asynchronous Plugin
2. Synchronous case:
Your plugin gets the video frame and works on that video frame and also feeds the manipulated frame to the server again. In this case, you should make the work in the caller thread and return the manipulated AVFrame.
example: you may have such a plugin that adds watermark addition
Synchronous Plugin
3. Last point case:
Your plugin gets the video frame and works on that video frame and three is no need to publish this AVFrame or feed this frame to other plugins. In this case, your plugin is the last point for the AVFrame in the execution. For this case, you should return a null value.
For example, you may have such a plugin that streams the AVFrame with a new protocol that isn’t available in AMS or you may only record the frames.
Last Point Plugin
From Plugin to AMS Interaction
Interaction from the plugin to the AMS is managed through AntMediaApplicationAdaptor class.
Registration Methods
These methods are for subscribing or removing Packet or Frame Listeners to the AMS. You need to register your objects in the plugin to the AMS to receive packets or frames.
- public void addFrameListener(String streamId, IFrameListener listener)
- public void removeFrameListener(String streamId, IFrameListener listener)
- public void addPacketListener(String streamId, IPacketListener listener)
- public void removePacketListener(String streamId, IPacketListener listener)
Frame Listener Registration
Custom Broadcast
AMS has such ingestion cases as WebRTC, RTMP, and stream source pulling. But you may want to have such a broadcast that is fed by the plugin instead of these ingestion cases. In this case, you can create a CustomBroadcast and you may feed it from the plugin.
CustomBroadcast also implements the IFrameListener. You can create or stop a CustomBroadcast with the following methods:
- public IFrameListener createCustomBroadcast(String streamId)
- public void stopCustomBroadcast(String streamId)
For example, you may have such a plugin that receives frames from an external source and feed them to the CustomBroadcast. Then CustomBroadcast can publish this stream with WebRTC, HLS, or DASH.
Custom Broadcast
You need to create a CustomBroadcast first by calling createCustomBroadcast. Then you have to call setVideoStreamInfo and setAudioStreamInfo methods of the CustomBroadcast. After setting the stream info, you may feed the CustomBroadcast with onVideoFrame and onAudioFrame.
Custom Broadcast Usage
Plugin Deployment
Plugin main class should be in the io.antmedia.plugins package.
If REST service exists, then Rest Service Class should be in the io.antmedia.rest package.
After building your Plugin Project you should copy the jar file into /usr/local/antmedia/plugins directory.
Sample Plugin Project
I will explain the process to create a Plugin application, with an example project. This project gets the video/audio frames and packets then provides statistics for them with a REST method. You can find this project on GitHub.
Create the project:
- Implementing IFrameListener and IPacketListener Interfaces.
In the sample project, we implement these interfaces with SampleFrameListener, SamplePacketListener - Create Application Main class io.antmedia.plugin.SamplePlugin.java
Note that Package structure is important for this class. In this class, we have a register method that registers our listeners to the AMS. - Create a REST Service class io.antmedia.plugin.RestService.java
Note that Package structure is important for this class. This class has two REST methods. One of them is the register method that calls the SamplePlugin register method. The second one is the stats method that gets the stats from SamplePlugin and provides them from the REST method. - Building the project with maven with the following method:
mvn clean install -Dmaven.javadoc.skip=true -Dmaven.test.skip=true -Dgpg.skip=true
- Deployment
copy the generated jar file into /usr/local/antmedia/plugins - Restart the Server
sudo service antmedia restart
Usage
Call the register POST REST methods to register the stream as Frame Listener and Packet Listener as follows:
curl -X POST -H "Accept: Application/json" -H "Content-Type: application/json" http://localhost:5080/WebRTCAppEE/rest/sample-plugin/register/stream1
Call the stats GET REST method to get the frame and packet stats as follows:
curl -X GET -H "Accept: Application/json" -H "Content-Type: application/json" http://localhost:5080/WebRTCAppEE/rest/sample-plugin/stats
Conclusion
Consequently, we have responded to your valuable requests for the plugin by implementing this feature. It will be released with v2.4.0 but it can be available in the snapshot releases as well. Now we are looking for your contributions as plugins to expand our ecosystem together.
