Challenges Using Istio



The team and I were searching for solutions that enabled us to secure the communication between the microservices that we have running in our Rancher cluster. Our first application consisted of a set of services that essentially performed CRUD operations using the typical RESTful architecture (GET, POST, PUT, etc).

We decided to use Istio because it offered a compelling solution that incorporated the use of mTLS (mutual TLS authentication). For us, this meant that we didn’t need to manage certificates ourselves our worry about the individual TLS implementation of each service in its respective language. In fact, it also meant we could make all of our requests through standard HTTP requests and the Istio sidecar would tunnel those requests through a TLS connection.

This worked perfectly for our first application. We enabled a namespace wide policy that enforced each service to only accept mTLS connections.

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "default"
  namespace: "with-istio"
spec:
  peers:
  - mtls: {}

This was followed with a destination rule that told each service to communicate using a TLS connection.

apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
  namespace: "with-istio"
spec:
  host: "*.with-istio.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

The result was what we expected. Each deployment had a sidecar proxy available, and all communication was secured using TLS. We verified this using the Kiali graph.

Unfortunately, the simplicity of Istio falls apart when dealing with more complex GRPC connections and services with their own TLS management. There were many challenges using Istio that we didn’t anticipate.

Problems with using Istio

mTLS on TCP Traffic

The first problem we encountered was a misunderstanding into what type of traffic Istio secures. According to Istio, the following protocols are supported:

  • grpc
  • http
  • http2
  • https
  • mongo
  • mysql
  • redis
  • tcp
  • tls
  • udp

What’s not clearly documented is that Istio ignores plain TCP traffic. So if you have a service that has a port name with the pre-fix “tcp” in it, Istio will ignore it and not secure it using mTLS.

  ports:
  - name: tcp-foo
    port: 9000
    protocol: TCP
    targetPort: 9000

The following diagram is taken from Kiali, which shows how the httpbin application (which uses http protocol) is secured using mTLS but the tcp-echo service (which uses a tcp port) is not.

This might be an intended feature of Istio, but the documentation for this behavior is not outlined clearly on Istio’s website. In fact, the only reason we discovered this behavior was from a response on a GitHub issue.

While this helped us perform a workaround for one of the problems (listed below) it also meant that we could not secure that traffic using mTLS. A possible solution would be to stand up a gateway and virtual service in front of these services, but that will have to be replicated multiple times.

GRPC and HTTP2

Another issue that we encountered was using the GRPC protocol with Istio. When we try to communicate to an Istio service that has a GRPC protocol we receive a 403 from envoy.

curl -v --http2 http://grpcbin:4200
#results in 403 from Envoy

According to this GitHub issue, it’s due to how envoy treats the optional h2c header. This might be fixed in the latest version of Istio (1.2.5), we have yet to perform an upgrade.

mTLS over HTTPS

The final challenge we encountered was making HTTPS requests from inside of the container. We have two services which use their own TLS certificates that can’t be turned off. Service B is expecting a HTTPS request from service A. The problem is that Istio expect the container to perform HTTP requests, and not HTTPS requests. If we perform a HTTPS requests, we receive a SSL error. However, if we do an HTTP requests, Service B complains that there’s no certificate available.

Istio does provide a solution to this in their section title Create an HTTPS service with Istio sidecar with mutual TLS enabled. The solution is to re-install Istio with strict mTLS enabled (istio-demo-auth). We choose not to do this though, because Istio warns against using it in production:

The demo configuration profile is not suitable for performance evaluation. It is designed to showcase Istio functionality with high levels of tracing and access logging.

The solution that we came up with was to just label the service with the tcp pre-fix so that Istio will ignore the traffic.

Conclusion

Istio is a powerful tool that secures all of your microservices in its service mesh. In our testing, if the application consists of various RESTful services all communicating using HTTP 1.1, without their own TLS validation, then Istio is the obvious choice for securing the communication. However, the more complex your communication needs get, the more inflexible Istio becomes.


See also