💻kismetlabs.xyz

I'm Kevin, hardware engineer and writer. My life's work is to design technology for the benefit of humanity.

  • 📝Blog
  • 🖥️Projects
  • 🔍Consulting
  • 💼Experience
  • 🎓Education
  • 💾About

Résumé|Topics

Blog

Deep dives, build logs, and notes on hardware bring-up, racks, power, and more.

Show all

Tags

Home / Blog
  • Electrical 6.13

    Protection at the utility connection: relays, settings and anti-islanding

    The relay functions that protect the substation and the point of interconnection, worked on the 75 MVA example transformer: full-load currents and CT ratios, the overcurrent and inrush problem, a percentage differential check, passive anti-islanding and its non-detect zone, and reclosing onto an island.

    #electrical#ep12#protection#relays#anti-islanding#worked-example
  • Electrical 6.15

    Vendor diligence, energy-services terms and bridging power for an on-site plant

    How to turn the generation, battery and microgrid design into contract terms and review steps: what an availability number means in hours, unit versus plant guarantees, a remedy worth in dollars, a ramp and ride-through acceptance test, deviation requests, and a bridging-power hand-over.

    #electrical#ep14#vendors#contracts#bridging-power#worked-example
  • Electrical 6.14

    Power-system studies for a data center site: load flow, motor starting, transient stability and EMT

    What each study answers and the toy version of it for the 62.4 MW example site: voltage and losses by load flow, voltage dip from motor starting, critical clearing time from a swing-equation model, and capacitor-bank resonance, with the study types a utility or a grid operator asks for.

    #electrical#ep13#studies#load-flow#stability#worked-example
  • Electrical 6.12

    Microgrids for a data center: islanding, load shedding, black start and resynchronization

    The operating modes of a site microgrid (grid-parallel, islanded, black start), what the controller does in each, an islanding timeline, load shedding against available generation, a block-by-block black start and the synchronizing window for reconnecting, with the IEEE 1547 and 2030.7 requirements that frame them.

    #electrical#ep11#microgrid#islanding#black-start#worked-example
  • Electrical 6.11

    Generator and battery response to a training load: step loads, ride-through and grid-forming storage

    A swing-equation model of a gas plant hit by load steps, a battery that takes the fast part of a synchronized training load, droop sharing with a grid-forming inverter, ride-through energy, inverter fault current, and the standards that govern generator transients and battery safety.

    #electrical#ep10#generation#battery#grid-forming#worked-example
  • Electrical 6.10

    On-site generation: engines, turbines and fuel cells for a 62 MW site

    Choosing and sizing behind-the-meter generation for the example site: unit counts after site derating, the availability of an N+k plant, NOx in tons per year against permit thresholds, a phased build, fuel systems and the codes that apply.

    #electrical#ep9#generation#engines#turbines#fuel-cells#worked-example
  • Electrical 6.2

    Where the power comes from: energy sources, siting and nameplate for a 62 MW site

    The first post of the electrical series starts at the generator: the US generation mix, where each kind of source is concentrated, why data center clusters sit where they do, and the arithmetic of how much nameplate, storage and gas a flat 62 MW load needs.

    #electrical#ep1#generation#siting#worked-example
  • Part 3 · 1 of 1

    Slurm and Kubernetes in industry: who runs what, and why

    A reading post that sets the two lab clusters against public evidence: a scorecard of what each does well, which labs and companies use which, how people run both at once, and what changes for GB200 NVL72 racks.

    #homelab-series#p3#slurm#kubernetes#scheduling#industry
  • Electrical 6.3

    Utility interconnection and the medium-voltage substation for a 62 MW site

    Twenty of the example halls on one site, with the interconnection standards and power quality at the point of interconnection: service voltage by arithmetic, a two-transformer substation with a normally open tie and its fault duty, the interconnection study sequence a utility runs, the 2024 Virginia load-loss event, and a mock cost for the substation.

    #electrical#ep2#interconnection#substation#worked-example
  • Electrical 6.18

    400 V and 800 V DC distribution for GPU racks: what changes in conversion and protection

    A reading post with three pieces of arithmetic: current, conductor count and conversion stages for AC and 800 V DC at 1 MW per rack, and what the NVIDIA and Open Compute material says changes in grounding, protection and safety.

    #electrical#ep17#dc-power#reading
  • Kubernetes 2.6

    Kubernetes 2.6: building, upgrading and replacing nodes with kubeadm

    A three-node cluster built with kubeadm on fresh VMs: what a single control plane does when its API server stops, a one-minor upgrade with probes running, replacing a worker, an expired join token, and an etcd snapshot.

    #kp7#kubernetes#kubeadm#cluster-lifecycle#bare-metal
  • Electrical 6.17

    Arc-flash basics: incident energy, clearing time and what a label says

    Incident energy at the switchboard and panel from the coordination post, using the open-air Lee and Doughty equations checked against a published worked example, and how the short-time delay that fixes coordination raises the energy from 0.9 to 15.6 cal/cm2.

    #electrical#ep16#arc-flash#safety#worked-example
  • Electrical 6.16

    Commissioning documents: basis of design, sequence of operations and an integrated test script

    The three documents that turn a power design into a tested hall, written for the distributed 2.4 MW example: a basis of design, a sequence of operations simulated against its targets, and an integrated systems test with a record that catches an unbalanced load.

    #electrical#ep15#commissioning#worked-example
  • Electrical 6.15

    Metering, monitoring and PUE: branch circuits, stranded power and alerts

    Emulated branch-circuit and facility meters for a 12-rack hall feeding Prometheus: per-rack draw against capacity, stranded power, PUE, and how fast alerts fire for a lost feed, a silent meter and a cooling fault.

    #electrical#ep14#metering#pue#prometheus#worked-example
  • Kubernetes 2.5

    Kubernetes 2.5: a Go admission webhook and controller

    A small admission webhook and a small controller in Go, run on the three-server k3s cluster: what the webhook does when it is down, slow or in its own way, how its certificate renews without a restart, and how a controller with two replicas fails over.

    #kp6#kubernetes#k3s#go#admission#controllers
  • Electrical 6.9

    GPU load transients: what a synchronized training job asks of the source

    A toy model of a hall of 1,920 GPUs training in lockstep: the swing every iteration, the step at a checkpoint, what a smoothed source and its storage have to do, and why sizing on the average load is not enough.

    #electrical#ep8#gpu#transients#worked-example
  • Kubernetes 2.4

    Kubernetes 2.4: certificates, rotation and secrets at rest

    cert-manager with a private CA and a certificate that renews itself, what a running server does with the new file, rotating k3s's own certificates, and turning on encryption of secrets in etcd on three servers.

    #kp5#kubernetes#k3s#certificates#secrets#cert-manager
  • Electrical 6.8

    Grounding, harmonics and power quality: neutral current and distortion

    Neutral current from unbalanced and rectifier loads, transformer heating from harmonics, voltage distortion at the bus, and what high-resistance grounding changes about a ground fault, all with textbook arithmetic and assumed load spectra.

    #electrical#ep7#grounding#harmonics#power-quality#worked-example
  • Kubernetes 2.3

    Kubernetes 2.3: tenants with RBAC, quotas, pod security and an audit log

    Two tenant namespaces on the three-server k3s cluster: client-certificate users, role bindings that stop at the namespace, an escalation attempt, quota and default sizes, pod security, network isolation, and the audit log of what was refused.

    #kp4#kubernetes#k3s#rbac#multi-tenancy#audit
  • Electrical 6.7

    UPS topologies and batteries: efficiency, losses and runtime

    Double conversion, line-interactive and eco mode compared on an assumed efficiency model, the UPS loss of each redundancy design from the redundancy post, and the battery energy and runtime for one 1,200 kW block.

    #electrical#ep6#ups#batteries#worked-example
  • Kubernetes 2.2

    Kubernetes 2.2: replacing flannel with Cilium and writing network policy

    Switching the three-server k3s cluster from flannel to Cilium: what stalled during the switch, the same network policy before and after, an HTTP rule and an egress rule by name, Hubble, and what DNS and CNI failures look like.

    #kp3#kubernetes#k3s#cilium#network-policy
  • Electrical 6.6

    Busway, rack distribution and 415 V versus 208 V

    Current, tap-off size, busway rating, voltage drop and loss for one row of racks at 208, 415 and 480 V, and the transformer stage that a 208 V design adds.

    #electrical#ep5#busway#voltage#worked-example
  • Kubernetes 2.1

    Kubernetes 2.1: three control-plane servers and etcd on k3s

    Moving the k3s control plane from one server on SQLite to three servers with embedded etcd: the migration, what the API does with one and two servers down, a snapshot and restore, and a rolling restart.

    #kp2#kubernetes#k3s#etcd#high-availability
  • Electrical 6.5

    Short-circuit current and selective coordination: transformer to rack breaker

    A worked example on one 480 V path: fault current at three points from textbook formulas, then time-current curves for three breakers, showing where a rack fault takes out the feeder and how short-time delays fix it at a cost.

    #electrical#ep4#short-circuit#coordination#worked-example
  • Electrical 6.4

    Redundancy architectures for one example hall: 2N, distributed and catcher

    One 2.4 MW example hall in four redundancy designs: sizing, one and two blocks out, an unbalanced-load failure, then a mock reliability model and a mock cost breakdown that put a price on each extra minute of availability.

    #electrical#ep3#redundancy#worked-example
  • Part 0 · 3 of 3

    The foundations: Slurm, Kubernetes and the stacks that combine them

    A reading post on how Slurm and Kubernetes answer different questions, why drain, cordon and taint are easy to confuse, and what Slinky, Soperator and SUNK do when a datacenter wants both.

    #homelab-series#p0#slurm#kubernetes#scheduling
  • Part 8 · 4 of 4

    Power and cooling at GPU rack density: air, cold plates and what changes

    A reading post with no lab: what one air-cooled 8-GPU server draws and exhausts, the rack-level arithmetic for air against water, the cooling approaches in ASHRAE's liquid-cooling paper, and what operations teams check when a rack is liquid-cooled.

    #homelab-series#p8#gpu#power#cooling
  • Part 8 · 3 of 4

    What the product names mean: HGX, DGX, MGX and SuperPOD

    A reading post with no lab: how NVIDIA's HGX boards, DGX systems, MGX modular designs and SuperPOD reference architectures relate, what a DGX H100 contains, what a scalable unit is, and which decisions each step leaves to the buyer.

    #homelab-series#p8#gpu#architecture#datacenter
  • Part 8 · 2 of 4

    How GPU servers connect to each other: InfiniBand, RoCE and rail-aligned fabrics

    A reading post with no lab: what RDMA changes, how InfiniBand and RoCE differ in flow control and congestion handling, what rail-aligned leaf and spine fabrics look like in NVIDIA's SuperPOD reference design, and the bandwidth arithmetic per GPU.

    #homelab-series#p8#gpu#networking#infiniband
  • Part 8 · 1 of 4

    How GPUs connect: NVLink, NVSwitch, the topology matrix and an NVL72 rack

    A reading post with no lab: what NVLink and NVSwitch add over PCIe, how to read nvidia-smi topo -m and NCCL's path levels, and what the vendor bandwidth figures imply for a gradient exchange, worked as arithmetic, then the same fabric at rack scale in an NVL72 rack: nodes, switch trays, IMEX, the fabric managers and the bring-up order.

    #homelab-series#p8#gpu#nvlink#topology
  • Part 7 · 4 of 4

    Reading and using NUMA topology on a virtual two-node machine

    A guest with two virtual sockets and two NUMA nodes on a single-socket host: reading the layout with lscpu, numactl and hwloc, setting memory policies and watching where pages land, benchmarking local against remote memory, and finding out why Slurm's CPU binding did nothing on the lab cluster.

    #homelab-series#p7#numa#linux#scheduling
  • Part 7 · 3 of 4

    Scheduling GPU work on Kubernetes with emulated GPUs

    Four emulated GPUs per node advertised as a counted resource, drills for fragmentation and preemption with the real Kubernetes scheduler, device allocation with the Kubernetes example driver and a CEL selector, and a paper exercise on MIG partition layouts.

    #homelab-series#p7#gpu#kubernetes#scheduling
  • Part 7 · 2 of 4

    GPU fleet monitoring with an emulated exporter

    Eight emulated GPUs on two nodes exposing the metric names of a GPU telemetry exporter, four alert rules for a clock cap, a driver error, a missing GPU and an idle GPU, and drills that time how fast each alert appears.

    #homelab-series#p7#gpu#monitoring#prometheus
  • Part 7 · 1 of 4

    An AWS autoscaling group with Terraform, on an emulator

    A network, a launch template and an autoscaling group for GPU worker nodes written in Terraform and applied to a local AWS emulator, driven through scaling events with the AWS API, with a list of where the emulator differs from the real service.

    #homelab-series#p7#cloud#terraform#autoscaling
  • Part 6 · 7 of 7

    Software RAID and a management network, on VMs

    A disk failure and rebuild on software RAID 1 and RAID 5 with checksums, a management network made of VLANs where a BMC emulator is reachable from one segment only, and NIC bonds tested with link failures.

    #homelab-series#p6#bare-metal#raid#networking
  • Part 6 · 6 of 7

    Provisioning blank VMs over the network with PXE

    An isolated provisioning network, a server that gives blank machines an address, a boot loader and unattended install answers, and a timeline of two empty VMs becoming installed, reachable machines in under five minutes.

    #homelab-series#p6#bare-metal#pxe#provisioning
  • Part 6 · 5 of 7

    Hardening a VM and checking Secure Boot with a TPM

    A VM scanned and hardened against a published baseline with before and after scores and a list of what was not fixed, and a second VM that boots with Secure Boot and a virtual TPM, tested with an unsigned kernel module and a secret sealed to the boot state.

    #homelab-series#p6#bare-metal#security#secure-boot
  • Part 6 · 4 of 7

    NetBox as the rack inventory: devices, power and BMCs

    NetBox on the cluster as the source of truth for an example rack: devices, power cabling and BMC records, a script that cross-checks it against the emulated BMC fleet and the power plan, and an onboarding and offboarding drill.

    #homelab-series#p6#bare-metal#netbox#dcim
  • Part 6 · 3 of 7

    Commissioning checks on lab VMs: drift and burn-in report

    A baseline checked without changing anything, a drill that changes five things by hand and times how fast a scheduled check notices, and a burn-in and acceptance report with a pass and a fail case.

    #homelab-series#p6#bare-metal#ansible#commissioning
  • Part 6 · 2 of 7

    A rack power plan: from the outlet back to the transformer

    A worked example for an example rack: outlet map, circuit limits, losing a feed and staged power-on, then the data center electrical chain behind it: voltages, redundancy models, fault current, grounding, GPU load swings and commissioning, with scripts that check the arithmetic.

    #homelab-series#p6#bare-metal#power#paper-exercise
  • Part 6 · 1 of 7

    Emulated BMCs: provisioning, firmware rollout and fault drills

    Six emulated Redfish management controllers on one machine, a scripted setup, a staged firmware rollout, and drills that show how fast a hang, an outage and a flap are noticed.

    #homelab-series#p6#bare-metal#redfish#monitoring
  • Part 5 · 6 of 6

    Self-service with Crossplane: a developer API on the cluster

    Defining a WebApp resource with Crossplane so that a developer writes five lines and the cluster creates the Deployment and Service, with schema limits, a narrow provider identity, and measurements of how it behaves when things are changed by hand.

    #homelab-series#p5#platform#crossplane
  • Part 5 · 5 of 6

    Gateway API and network policy: routing and isolating traffic on k3s

    Turning on the Gateway API in the k3s ingress controller, splitting traffic between two versions of an application by weight and by header, and restricting which pods can reach it with a network policy.

    #homelab-series#p5#platform#gateway-api#security
  • Part 5 · 4 of 6

    Policy and supply chain: Kyverno, signed images and an SBOM

    Adding admission policy with Kyverno to the k3s cluster, then signing an image, attaching a software bill of materials, and making the cluster refuse pods whose images are unsigned or tampered with.

    #homelab-series#p5#platform#kyverno#security
  • Part 5 · 3 of 6

    GitOps with Argo CD: the cluster follows a git repository

    Delivering a small application to the k3s cluster from a git repository with Argo CD, and measuring how fast it applies changes, corrects drift, prunes objects, and recovers from a bad commit.

    #homelab-series#p5#platform#argocd
  • Part 5 · 2 of 6

    Scaling on a request-rate metric with KEDA

    Replacing the CPU autoscaler on the model server with KEDA scaling on a Prometheus query, comparing the two under the same load, and what happens when it scales to zero.

    #homelab-series#p5#platform#keda
  • Part 5 · 1 of 6

    Queues and quotas on Kubernetes with Kueue

    Putting a quota-based job queue in front of a three-node cluster with Kueue, borrowing and reclaiming quota between two teams, and what admission does and does not guarantee.

    #homelab-series#p5#platform#kueue
  • Part 0 · 2 of 3

    The foundations: Go vs Python

    How Go and Python differ in typing, concurrency, memory and deployment, followed by a case study: the same log-parsing pipeline written in both and timed on one lab VM.

    #homelab-series#p0#go#python
  • Part 4 · 5 of 5

    A Go CLI tool for the cluster: capacity and model server status

    A small Go program that reads node capacity and model server status from the Kubernetes API with a read-only identity, tests that need no cluster, and what its numbers showed about requests.

    #homelab-series#p4#ml-platform#go
  • Part 4 · 4 of 5

    Terraform on Proxmox: one VM from code, with drift and a rebuild

    Defining a virtual machine in Terraform with a least-privilege API token, catching a manual change as drift, and destroying and rebuilding the machine from code in under half a minute.

    #homelab-series#p4#ml-platform#terraform
  • Part 4 · 3 of 5

    Ray on Kubernetes: a data pipeline on one, two and three workers

    Running a Ray cluster on the k3s cluster with the KubeRay operator, timing a CPU-bound pipeline on one, two and three workers, and killing a worker in the middle of a run.

    #homelab-series#p4#ml-platform#ray
  • Part 4 · 2 of 5

    Moving data to a model server: local disk, NFS and object storage

    Measuring read and write speed from a local disk, an NFS share and a MinIO object store on the three-VM cluster, with the spread between repeated runs.

    #homelab-series#p4#ml-platform#storage
  • Part 4 · 1 of 5

    A model server on Kubernetes under load

    A small image-classification service on the k3s cluster, with Prometheus, load tests, autoscaling, alert rules, and a node powered off while it serves traffic.

    #homelab-series#p4#ml-platform
  • Part 2 · 1 of 1

    Kubernetes on the same three VMs

    A three-node k3s cluster next to Slurm: node-failure timings, drain with a disruption budget, a minor upgrade, autoscaling, taints, and what went wrong.

    #homelab-series#p2#kubernetes
  • Part 1 · 1 of 1

    A three-node Slurm cluster from scratch

    A small Slurm cluster on Proxmox, set up with Ansible, with a node drain test and the things that went wrong.

    #homelab-series#p1#slurm
  • Part 0 · 1 of 3

    The foundations: Proxmox VE and Tailscale quick start

    How I found Proxmox VE, why LXC containers and Tailscale fit a home lab, and the machines the Slurm lab runs on.

    #homelab-series#p0#proxmox#tailscale
  • Series intro

    Learning infrastructure by building it: series introduction

    A plan for working through schedulers, Kubernetes, ML platforms, bare-metal bring-up and cloud, in small labs, starting with a Slurm cluster.

    #homelab-series#series-intro
  • No posts match.

Built with Astro and Tailwind CSS and hosted on Cloudflare Pages.

Interested in working together? Send me a message.

BlogProjectsConsultingExperienceEducationAbout

© 2026 Kevin Pham. All rights reserved.