At Nylas, we use Kong API gateway to authenticate, control routing, transform requests and responses. All of these are done by creating custom Kong plugin. This blog explains:
At Nylas, APIs are supported by different backends. For example:
Email API
Calendar API
Scheduler
Contacts API
ExtractAI
Google email & calendar integration
Outlook email & calendar integration
IMAP integration
Calendar management
Embedded contextual email
Scheduling automation
Automated outreach
Intelligent data extraction
Ecommerce
Telehealth
Real estate
Financial technology
Educational technology
Expense management
Customer relationship management
Applicant tracking system
Communications platform
Marketplaces
Document management
Docs
SDKs
Tutorials
API status
Changelog
Support
Blog
Events
Guides
Inspiration gallery
Partners
Build vs. buy
Customer stories
Security & Compliance
Why Nylas
Newsroom/send) are handled by an email sending service./events and /calendars) are handled by a calendar service.API requests to different endpoints are routed to different backend services. To better manage the routing, we use Kong as our API gateway provider:
We choose Kong as our API gateway provider. Nylas handles thousands of requests per second. For such a large throughput, Kong can route these requests in the order of a few milliseconds.
We separate our API gateway into control plane and data plane:
The separation of control plane and data plane improves the reliability and scalability of our API gateway. Even if control plane crashes, Nylas can still process requests through data plane. Data plane and control plane are also provisioned differently. High memory usage in data plane won’t affect the critical functionalities running in control plane. You can read more about control-plane and data-plane in their documentation: https://docs.konghq.com/gateway/latest/plan-and-deploy/hybrid-mode/.
Another thing Kong provides are a number of built-in plugins. A plugin is a middleware which processes requests inside the API gateway layer. It sit between customers and API backend. Some of the built-in plugins we use are:
Built-in plugins come with limitations. It does not support complex plugin logic. Our goal is to do the followings in the API gateway layer:
No built-in plugins handle such complicated logic, so we built our own API gateway plugin.
A request lifecycle will look like this:
Why authenticate in API gateway layer? Can’t we build an authentication library and use it in every backend service? Let’s consider this alternative architecture:
There are a few reasons:
Authentication is tightly coupled with API gateway. It makes sense to do authentication using a Kong plugin.
Kong offers an open sourced Plugin Developer Kit (or “PDK”) in various languages. You can build a Kong plugin with Go, Javascript, Python, and Lua:
We chose Go as our plugin development language because it is fast, light-weighted and has good community support. As a side note, we also use Go for most of our backend services. Here is a simple hello world Go plugin:
package main
import (
"github.com/Kong/go-pdk"
"github.com/Kong/go-pdk/server"
)
func main() {
server.StartServer(New, Version, Priority)
}
var Version = "0.2"
var Priority = 1
type Config struct {
Message string
}
func New() interface{} {
return &Config{}
}
func (conf Config) Access(kong *pdk.PDK) {
message := conf.Message
if message == "" {
message = "hello"
}
kong.Log.Notice("Message: " + message)
}
The main plugin logic is written in Access(kong *pdk.PDK) function. You can also define plugin version and priority. Priority is the ordering of plugin execution. The higher the priority, the earlier it gets executed.
Kong offers a wide range of built-in functions. You can find the source code in https://github.com/Kong/go-pdk. Here are some examples:
| Plugin capability | Functions |
| Request parsing | kong.Request.GetScheme() kong.Request.GetHost() kong.Request.GetPort() kong.Request.GetMethod() kong.Request.GetPath() kong.Request.GetQueryArg() kong.Request.GetHeader(headerKey string) kong.Request.GetRawBody() |
| Request transformation | kong.ServiceRequest.SetScheme(scheme string) kong.ServiceRequest.SetPath(path string) kong.ServiceRequest.SetMethod(method string) kong.ServiceRequest.SetQuery(query map[string][]string) kong.ServiceRequest.SetHeader(name string, value string) kong.ServiceRequest.SetRawBody(body string) |
| Response parsing | kong.ServiceResponse.GetStatus() kong.ServiceResponse.GetHeader(name string) kong.ServiceResponse.GetRawBody() |
| Response transformation | kong.Response.SetStatus() kong.Response.SetHeader(k string, v string) |
| Change routing destination | kong.Service.SetUpstream(host string) kong.Service.SetTarget(host string, port int) |
| Logging | kong.Info(args …interface{}) kong.Notice(args …interface{}) kong.Warn(args …interface{}) kong.Err(args …interface{}) kong.Crit(args …interface{}) |
With these functions, you can do anything in the gateway layer. The built-in functions are the building blocks of custom plugins.
Kong’s PDK libraries for Go, Javascript, and Python are not native. They are wrapper of Lua functions. For example, this is the source code of kong.Service.SetUpstream(host string) function:
func (s Service) SetUpstream(host string) error {
return s.Ask(`kong.service.set_upstream`, bridge.WrapString(host), nil)
}
The Go function wraps the Lua command kong.service.set_upstream. It doesn’t actually do anything except executing Lua. For this reason, writing unit test for custom Kong plugin is a little tricky. We will have to mock Lua like this:
service := Service{
bridge.New(
bridgetest.Mock(
t,
[]bridgetest.MockStep{
{
Method: "kong.service.set_upstream",
Args: bridge.WrapString("farm_4"),
Ret: nil
},
},
),
),
}
assert.NoError(t, service.SetUpstream("farm_4"))
Unit test won’t be able to test the actual effect of Kong plugin. That is why we don’t solely rely on unit test for code quality.
Then how do we test the plugin functionality? The answer is building a developer environment on your laptop. You can run a minikube and deploy Kong with your plugin. Make a fake backend service and configure routing accordingly.
Once Kong is up, write a function test that calls Kong proxy. See whether the request/response is transformed, and see if the routing is expected.
You can read more about building developer environment in my previous blog: https://www.nylas.com/blog/how-we-test-microservices-locally-at-nylas/
After writing custom logic to the plugin, we can deploy it with Kong API gateway.
First, compile the plugin into an executable. The plugin must be runnable on linux. This is an example compile command:
GO111MODULE=on CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o bin/go-hello plugin/go-hello.go
Second, build a docker image. In your Dockerfile, use kong-gateway as base image and copy the executable into the container:
FROM kong/kong-gateway:2.8 ADD bin/go-hello /usr/local/bin/go-hello USER kong # Prove that go-hello works RUN ["/usr/local/bin/go-hello", "-dump"]
Run docker build command to build the kong-with-plugin-image:
docker build -f Dockerfile -t kong-with-plugin-image:latest .
Third, configure the Helm values.yaml file to register the plugin.
image: repository: kong-with-plugin-image tag: "latest" env: pluginserver_names: go-hello pluginserver_go_hello_socket: /usr/local/kong/go-hello.socket pluginserver_go_hello_start_cmd: /usr/local/bin/go-hello pluginserver_go_hello_query_cmd: /usr/local/bin/go-hello -dump plugins: bundled,go-hello
Run helm upgrade with the new values.yaml:
helm upgrade kong kong/kong --values values.yaml -n kong
If you deploy Kong with control-plane and data-plane, kong image should be changed in both planes. You also need to add the environment variables (such as pluginserver_names and pluginserver_go_hello_socket) to the values.yaml file for both control-plane and data-plane. During deployment, upgrade control plane first, and then data-plane.
Now the plugin should be available in Kong. You will see these logs proving that plugin binary is registered:
[proxy] 2022/07/14 22:46:01 [notice] 5591#0: *28 [kong] process.lua:263 Starting go-hello, context: ngx.timer
At last, you can use [deck command](https://docs.konghq.com/deck/latest/) or Admin API to enable this plugin. If you are a Kong enterprise user, go to Kong manager and enable it in the UI:
Instead of “nylas-proof-of-concept”, you will see “go-hello”.
Once deployed to production, you will need to check whether the plugin is behaving properly.
If the plugin crashes, Kong API gateway will “fail-open” where requests pass through Kong as if there is no plugin. After the fail-open, plugin binary will restart immediately. However, if the fundamental issue is not fixed, plugin will keep crashing and restarting.
You won’t see any alert for Kong plugin crashes. The only way to check is looking at logs. Here is an example of a plugin crash logs:
kong-data-plane-kong-9b4cd96b6-nq52b proxy 2022/07/08 06:15:39 [error] 2104#0: *1840 [kong] pb_rpc.lua:404 [nylas-auth] closed, client: 10.177.59.116, server: kong, request: "GET /calendars HTTP/1.1", host: "api-staging.nylas.com" kong-data-plane-kong-9b4cd96b6-nq52b proxy 2022/07/08 06:15:39 [notice] 2104#0: signal 17 (SIGCHLD) received from 2106 kong-data-plane-kong-9b4cd96b6-nq52b proxy 2022/07/08 06:15:39 [notice] 2104#0: *23 [kong] process.lua:279 external pluginserver 'nylas-auth' terminated: exit 2, context: ngx.timer kong-data-plane-kong-9b4cd96b6-nq52b proxy 2022/07/08 06:15:39 [notice] 2104#0: *23 [kong] process.lua:263 Starting nylas-auth, context: ngx.timer
Search with keyword signal 17 (SIGCHLD) or external pluginserver '<your-plugin-name>' terminated. If you see many error logs, it is an indication that your custom plugin is crashing.
Here is another tip: use the file-log plugin and log with kong.Notice(args ...interface{}) in your custom code. It can narrow down your search for bugs. Write as many logs as possible locally, and remove those logs before deploying to production.
If you are a Kong Enterprise user, you can always reach out to Kong Support. Although Kong does not support custom plugin development, they can still offer some guidance in their best efforts.
Lastly, we recommend running Kong plugin in a developer environment (minikube). Try to reproduce the bug locally, and write comprehensive function tests. If a bug cannot be reproduced locally, it may not be caused by Kong, but by the upstream service.
After reading this blog, be sure to checkout my example plugin: https://github.com/quzhi1/KongPlayground. This repository contains the plugin code, the deployment configuration, and a developer environment to play with.
The process of developing Kong custom plugins can be summarized in this meme:
Special thanks to all Nylanauts who helped building and deploying Kong plugins:
You can sign up Nylas for free and start building!
We’re excited to announce the launch of the Nylas Scheduler v3! Efficient scheduling is essential…
Key Takeaways Managing calendar availability is essential for professionals, teams, and businesses to stay organized…
Key Takeaways This article addresses the challenges of webhook integration and introduces the PubSub Notification…
Zhi is a staff engineer who leads the Developer Velocity team at Nylas. In 2021, he left Stripe and became a Nylanaut. Zhi is also a history buff who loves traveling.