Function as a Service has become a popular topic after the release of Amazon Lambda in 2014. Since then multiple open-source frameworks have been released for Kubernetes. This blog will summarize the thesis project about Evaluation of Function as a Service Frameworks for Kubernetes.
The benefits of a system using Function as a Service are the fine-grained and quick scalability to meet a certain demand. Also, when the system in idle functions can be removed to save computing resources and the cost of running the functions.
When deciding to build or extend a system there are many factors that need to be taken into account, which can be tricky and cumbersome, especially when choosing frameworks to base your system on. If situations such as short lived tasks or handling infrequent and sporadic amounts of users are use cases for you, Function as a Service can be the answer. Once the decision to base a system around Function as a Service has been made, the question becomes: should we choose a platform provided by a cloud provider or deploy a framework? Which platform or framework is most suitable for our needs? Hopefully after reading this post you will get a better understanding about Function as a Service and the available frameworks for Kubernetes.
I decided to take a deeper look at OpenFaaS, Kubeless and Fission. In the thesis I looked at the communities, developer experience, and, of course, performance. In this blogpost I will just show the developer experience and some of the performance results. The performance tests were conducted using a test script, an overview of which is shown in the figure below. A Google Kubernetes Cluster was used to deploy the frameworks on and I used “Hey”, a load generator.
For the developer experience I looked at how interchangeable the functions I developed were between each framework. Then, I performed multiple performance tests. The first performance test was looking at the time it takes to start a new function from zero, the cold start time. Thereafter, I performed two tests with a varying number of concurrent clients calling a function. Finally, I looked at increasing the number of gateways handling incoming requests to functions. This increases the success rate of requests done under load.
Experiments
Developer experience
The functions that were developed were easily interchangeable between the frameworks. You can see the differences, highlighted with the red boxes, in the source code in Figure X.
Cold start
The cold start test case showed that OpenFaaS is faster to start a new function from zero compared to Fission. Kubeless did not support the feature of scaling down to zero. In the reference figure you can see the response time when there is a function alive.
Low number of concurrent clients
The “low number of concurrent clients” test case showed that Fission is experiencing some performance problems and that OpenFaaS and Kubeless are performing similarly. The first figure shows the response times for all frameworks, where Fission is much slower than the other two. Because of this the second figure shows just OpenFaaS and Kubeless.
High number of concurrent clients
The second concurrent test case was with a high number of clients. Only OpenFaaS and Kubeless were able to handle this high number of concurrent clients. In it, both OpenFaaS and Kubeless perform similarly when looking at response time. However, OpenFaaS is not able to handle all incoming requests, which is shown in the table.
Increase success rate with gateways
To fix the success rate of OpenFaaS I tried increasing the number of gateways deployed. The purpose of a gateway is to handle incoming requests and forward them to functions. I thought that the gateway could be a bottleneck handling all of the concurrent clients at the same time. Configuring the number of gateways to equal the number of nodes in the cluster seems to be the optimal level.
Discussion
From the results I would say that Fission is not suitable to use in its current state and that Kubless seems to be the overall winner. Also, OpenFaaS had some problems handling high amounts of concurrent clients, which can be solved with more nodes and gateways deployed. However, OpenFaaS has more features. This can be appealing, which I went into more detail in the report.
Conclusion
In this post I have shown three frameworks running on a Google Kubernetes Cluster. Overall I would say that using either Kubeless and OpenFaaS would be a good pick for a framework to use. OpenFaaS has a broader feature set than kubeless, however struggles success rates high when handling a high amount of users.
