gke-ai-troubleshooting-tpu-mxla-hang

Diagnose GKE Cloud TPU multi-slice training hangs (Megascale `HANG_DETECTED` logs and hang monitored events) using the ML Diagnostics `Megascale XLA (MXLA) Hang Analyzer` (`gcloud alpha mldiagnostics monitored-events`) and 1-minute Cloud Monitoring multi-slice latency metrics (`kubernetes.io/container/multislice/*`). Distinguishes XLA compiler/HLO launch divergence and host data-input stalls from TPU chip, SparseCore, ICI, or network fabric faults. Use when multi-slice TPU training jobs freeze w

Install

KitScout test notes

Not tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.

Similar tools

• observability-and-instrumentation 117,009 ★
  Instruments code so production behavior is visible and diagnosable. Use when adding logging, metrics, tracing,
• honeycomb-tracing 113,555 ★
  Trace Paperclip runs, requests, and full CI suites in Honeycomb. Use to find traces, diagnose latency, design
• azure-monitor-opentelemetry-ts 49,642 ★
  Auto-instrument Node.js applications with distributed tracing, metrics, and logs.
• devops-troubleshooter 49,642 ★
  Expert DevOps troubleshooter specializing in rapid incident response, advanced debugging, and modern observabi
• distributed-tracing 49,642 ★
  Implement distributed tracing with Jaeger and Tempo for request flow visibility across microservices.