Lock Contention Hypothesis

Oct 13, 2025 by magicalceylontours.com 143 views

Introduction

In a recent migration of our search service to Kubernetes (k8s), we encountered a peculiar issue where every few minutes, a request would take an unusually long time to complete, resulting in high latencies. The p99.99th percentile (p9999) of these requests was a mere 30ms, indicating that the issue was not a consistent performance bottleneck. However, the occasional spikes in latency were causing significant problems for our application. In this article, we will delve into the root cause of this issue and explore the solution we implemented to resolve it.

The Problem

Our search application is built using Facebook's Thrift framework for server and thread management. The I/O threads utilize epoll_wait, a high-performance I/O event notification mechanism, while the CPU threads rely on futex and spinlocks for synchronization. The workloads themselves access a large memory-mapped file (mmap) for random reads, which is locked into memory using the mlock system call. Despite this complex architecture, the application does not make any sysfs calls, making it difficult to pinpoint the source of the issue.

Initial Investigation

To identify the root cause of the high latency spikes, we began by disabling cadvisor, a Kubernetes component responsible for monitoring and collecting metrics from containers. To our surprise, disabling cadvisor resolved the issue, indicating that the problem was related to cadvisor's presence. However, we were unable to determine the exact reason for this behavior.

cadvisor's Role

cadvisor runs with default settings and a 30-second interval, which seemed to be causing the high latency spikes. However, we were unable to find any direct evidence of cadvisor's involvement in the issue. The application's architecture and workload characteristics did not seem to be related to cadvisor's functionality.

Lock Contention Hypothesis

Given the application's use of futex and spinlocks for synchronization, we hypothesized that lock contention might be the root cause of the issue. However, the application does not make any sysfs calls, which seemed to rule out this possibility. Nevertheless, we decided to investigate this hypothesis further.

Tuning cfs_period_us

As a last resort, we decided to tune the cfs_period_us parameter for the cfs (Control Group Scheduler) to a higher value (625ms vs 100ms default). To our surprise, this change resolved the issue, indicating that lock contention was indeed the root cause of the problem.

Conclusion

In conclusion, our investigation revealed that cadvisor was causing high latency spikes in our search application. While the exact reason for this behavior was unclear, tuning the cfs_period_us parameter resolved the issue, pointing to lock contention as the root cause. This experience highlights the importance of thorough investigation and experimentation in resolving complex performance issues.

Recommendations

Based on our experience, we recommend the following:

Monitor cadvisor's impact: If you're experiencing high latency spikes in your application, consider disabling cadvisor to determine if it's the root cause of the issue.
Investigate lock contention: If you suspect lock contention, investigate the application's synchronization mechanisms and consider tuning the cfs_period_us parameter.
Thoroughly investigate performance issues: Complex performance issues often require thorough investigation and experimentation to resolve.

Future Work

While we were able to resolve the issue, we plan to further investigate the root cause of the problem and explore ways to improve the application's performance. Specifically, we aim to:

Analyze cadvisor's behavior: We plan to analyze cadvisor's behavior and metrics to determine if there are any issues with its implementation or configuration.
Optimize synchronization mechanisms: We will investigate ways to optimize the application's synchronization mechanisms to reduce lock contention and improve performance.

Introduction

In our previous article, we explored the issue of high latency spikes in our search application, which was caused by cadvisor, a Kubernetes component responsible for monitoring and collecting metrics from containers. We also discussed how tuning the cfs_period_us parameter resolved the issue, pointing to lock contention as the root cause. In this article, we will answer some frequently asked questions (FAQs) related to this issue.

Q: What is cadvisor, and why is it causing high latency spikes?

A: cadvisor is a Kubernetes component that collects metrics from containers and provides insights into their performance. However, in our case, cadvisor was causing high latency spikes due to lock contention. The exact reason for this behavior is unclear, but tuning the cfs_period_us parameter resolved the issue.

Q: What is lock contention, and how does it relate to cadvisor?

A: Lock contention occurs when multiple threads or processes try to access shared resources simultaneously, leading to delays and performance issues. In our case, lock contention was caused by cadvisor's presence, which was interfering with the application's synchronization mechanisms.

Q: Why did tuning the cfs_period_us parameter resolve the issue?

A: Tuning the cfs_period_us parameter resolved the issue by reducing the frequency of lock contention. The cfs_period_us parameter controls the period between two consecutive scheduling decisions, and by increasing it, we reduced the number of scheduling decisions, thereby reducing lock contention.

Q: What are the implications of this issue for Kubernetes users?

A: This issue highlights the importance of monitoring and understanding the impact of Kubernetes components on application performance. It also emphasizes the need for careful tuning of system parameters to avoid performance issues.

Q: How can I prevent similar issues in my own applications?

A: To prevent similar issues, we recommend:

Monitoring Kubernetes components: Regularly monitor Kubernetes components, such as cadvisor, to detect any performance issues.
Tuning system parameters: Carefully tune system parameters, such as cfs_period_us, to avoid performance issues.
Investigating lock contention: Investigate lock contention in your application to identify potential performance bottlenecks.

Q: What are the next steps for resolving this issue?

A: We plan to further investigate the root cause of the issue and explore ways to improve the application's performance. Specifically, we aim to:

Analyze cadvisor's behavior: We will analyze cadvisor's behavior and metrics to determine if there are any issues with its implementation or configuration.
Optimize synchronization mechanisms: We will investigate ways to optimize the application's synchronization mechanisms to reduce lock contention and improve performance.

Conclusion

In conclusion, our investigation revealed that cadvisor was causing high latency spikes in our search application due to lock contention. By tuning the cfs_period_us parameter, we resolved the issue, highlighting the importance of careful tuning of system parameters to avoid performance issues. We hope that this Q&A article will help others who may be facing similar performance issues in their applications.

Recommendations

Based on our experience, we recommend the following:

Monitor Kubernetes components: Regularly monitor Kubernetes components to detect any performance issues.
Tune system parameters: Carefully tune system parameters to avoid performance issues.
Investigate lock contention: Investigate lock contention in your application to identify potential performance bottlenecks.

Future Work

We plan to continue investigating the root cause of the issue and exploring ways to improve the application's performance. Specifically, we aim to:

Analyze cadvisor's behavior: We will analyze cadvisor's behavior and metrics to determine if there are any issues with its implementation or configuration.
Optimize synchronization mechanisms: We will investigate ways to optimize the application's synchronization mechanisms to reduce lock contention and improve performance.