Performance Testing & Observability
Load, stress, spike, and soak testing wired into real monitoring.
Problem
Performance issues on TenxYou and Freshterra typically surfaced after release through slow pages or timeouts, with no repeatable way to simulate real traffic patterns before shipping.
Challenge
Build a performance testing setup that could model different failure modes — sudden traffic spikes, sustained load, and long-running sessions — and make results visible enough for the team to act on quickly.
Architecture
k6 scripts define load, stress, spike, and soak scenarios against staging environments for TenxYou and Freshterra. Results and application logs flow into Grafana, Prometheus, and Loki so a latency spike in a dashboard can be traced back to the responsible service.
Technical Decisions
Four distinct scenario types
Load, stress, spike, and soak tests were modeled separately because they surface different problems — load testing checks steady-state capacity, spike testing checks reaction to sudden traffic, and soak testing surfaces slow leaks that only show up over time.
Metrics and logs in one place
Pairing Prometheus metrics with Loki logs in the same Grafana view made it possible to go from 'latency is up' to 'here's the failing service' without switching tools.
Implementation
- Wrote k6 scenario scripts for load, stress, spike, and soak profiles against TenxYou and Freshterra staging environments.
- Configured Prometheus to scrape performance metrics from test runs and application services.
- Built Grafana dashboards correlating latency, error rate, and throughput across scenarios.
- Wired Loki log streams into the same dashboards for faster root-cause tracing.
Results
- Established a repeatable performance testing process across two production platforms.
- Gave the team dashboard-level visibility into latency and error-rate behavior under load, rather than relying on user reports.
Lessons Learned
- Modeling scenario types separately (load vs. spike vs. soak) surfaces different bugs — a single generic load test would have missed most of them.
- Correlating metrics and logs in the same dashboard cuts diagnosis time significantly compared to checking each tool separately.
Future Improvements
- Add automated performance regression gates tied to CI.
- Extend soak testing to run continuously against a long-lived environment.