This indicates a problem within our walls. We can see in our Pingdom tests that we’re spending a lot of wait time before getting back any information. This will simulate issues within our infrastructures, such as slow or overburdened servers, or ingress pods with insufficient capacity to manage the load. In a similar fashion to how we injected latency at our edge, next let’s add latency to our ingress controllers on our actual worker nodes, slowing traffic just as it comes into our purview. Injecting Latency at Our Origins (Kube-Proxy Latency)

This is a perfect use of our Pingdom toolsets.Ģ. This is likely an internet problem, and we’ll need to follow up with our internet provider for further debugging. Our internal metrics show no issues at all. With these clues, we can safely deduce the problem will exist before the application establishes the socket, and the problem lives outside of our infrastructure. We can see ISP-caused latency is represented by connect time increases.Įven with the high delay shown by our external monitors, our internal monitors show no such latency increase. This is due to the time it takes for our packets to reach our system to make the TCP three-way handshake and establish a socket that can pass information between the user’s browser and our servers. With the latency injection, we can now see a much longer Connect time for our services. | tc qdisc add dev eth1 root netem delay 200ms Then, the following will cause any packet to take 200ms longer before making it out of the network queue. To this end we’ll use Nginx to pass traffic into the DigitalOcean Load Balancers and the Linux Traffic Control tool to manage our networking latency at various levels of the stack to see how it changes the behavior of our application. To do this, we’ll need to inject a server between the internet and our internal systems and slow down the packet queue to observe the results before it hits our network. Injecting Latency at Our Edge (Internet Latency)įirst, we’ll add latency at our edge level. Now let’s inject latency at four different parts of our architecture to see what we can learn from Pingdom.ġ. It’s important to outline where exactly these connections are occurring because different points of failure will represent differently to an end user and could be critical to identifying where in this stack to begin your investigations. We’ve color-coded our infrastructure to map to the representative part of the Pingdom Speed Test results, so we can design our tests to show the changes clearly. Here we can see a common pattern of external provider load balancers, our clusters Nginx ingress, Flask application front end, and a database. We’ll use a standard infrastructure for the purposes of our demonstration. Our application stack mapped with Pingdom site speed metrics. By using LinkerD agents in between every service of our stack, we can see the different behaviors the Pingdom agents will observe from the outside in. We can use it to watch the bytes traverse our network where our logs may not be so verbose. This serves as an injected monitor into all calls occurring between our services once they enter our front doors. In addition to Pingdom, we’ll also use LinkerD as an intermediary for our Kubernetes cluster to walk through a simple application from the Pingdom call to the application’s response. By correlating your Pingdom alerts with your infrastructure, you can shave valuable time off your next escalation and better design your tests to conform with your environment. Then, we will see how we can use Pingdom in a real-world situation to figure out where latency is occurring. We’ll create latency at four levels of our architecture: our edge, our origins, our app, and our database. In this article, we’ll look at how Pingdom can help you diagnose latency in your DigitalOcean Kubernetes deployments. So how does a Pingdom call work, and how can you use it to debug what’s happening inside the system? If you know what clues to look for, it can provide a great place to triage where a problem is occurring in the system. SolarWinds ® Pingdom ® is an external synthetic monitoring agent designed to monitor your systems from the outside in.

Monit digitalocean