gke-ai-troubleshooting-tpu-performance-degradation

Diagnose GKE Cloud TPU training throughput drops and step-time regressions (15%+ TPU duty-cycle drop) using ML Diagnostics Workload Monitoring (`gcloud alpha mldiagnostics monitored-events` / `hypercomputecluster.googleapis.com/v1alpha`) and 1-minute Cloud Monitoring system metrics (`kubernetes.io/node/accelerator/*`). Distinguishes hardware and network fabric throttling from workload resource bottlenecks (HBM capacity, host memory, or host CPU saturation). Use when TPU training throughput or du

Install

KitScout test notes

Not tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.

Similar tools

• observability-and-instrumentation 117,009 ★
  Instruments code so production behavior is visible and diagnosable. Use when adding logging, metrics, tracing,
• honeycomb-tracing 113,555 ★
  Trace Paperclip runs, requests, and full CI suites in Honeycomb. Use to find traces, diagnose latency, design
• azure-monitor-opentelemetry-ts 49,642 ★
  Auto-instrument Node.js applications with distributed tracing, metrics, and logs.
• devops-troubleshooter 49,642 ★
  Expert DevOps troubleshooter specializing in rapid incident response, advanced debugging, and modern observabi
• distributed-tracing 49,642 ★
  Implement distributed tracing with Jaeger and Tempo for request flow visibility across microservices.