Utility interconnection and the medium-voltage substation for a 62 MW site

Goal

The previous post looked at where electricity is generated and why data centers cluster where the grid can reach them. This post follows the chain one step further, to the point where the site meets the grid. The posts after it go inward along the same chain: the one-line diagram of the hall, short-circuit duty from the utility source, busway, UPS, grounding and power quality, and finally the load.

The short-circuit post later takes this post’s 500 MVA utility source as its starting point, and the redundancy post prices a hall without the service that feeds it. Here we define that service.

One example hall is 2.4 MW of IT. A large site is many of them. We scale to twenty halls and ask four questions. At what voltage does the site connect? How are the transformers and the medium-voltage bus arranged so that one failure does not take the site down? What does the utility study and require before it energizes the service? And what does that add to the cost?

As in the rest of the series, where a utility’s data, a standard or a manufacturer’s rating disagrees with a number here, the number here is the one that is wrong. The values below are assumptions chosen to show the method.

Terms used in this post

  • Point of interconnection (POI): the place where the customer’s facilities connect to the utility system, usually a substation bay.
  • Large load: a customer whose demand is large enough to need special study. RMI describes loads above about 20 MW as typically connecting at the transmission level, and utilities and market operators set their own thresholds.
  • Customer interconnection facilities: the substation bays, breakers, protection, transformers and short lines the customer needs to reach the POI. RMI says these costs are typically allocated entirely to the customer.
  • Network upgrades: changes the utility makes elsewhere on its system, such as reconductoring, uprating transformers or adding reactive support. Cost allocation for them varies by jurisdiction.
  • System impact study: the utility’s study of what the new load does to its system, which can include power flow, stability, short-circuit and electromagnetic transient simulations.
  • Bus tie: a breaker between two sections of a bus. Normally open, it lets one source feed both sections after the other is lost.
  • Metal-clad switchgear: medium-voltage switchgear in which each breaker sits in its own grounded compartment, normally drawout.
  • E-house: a factory-built enclosure, delivered by truck or crane, that holds switchgear, protection and controls and is connected on site.
  • Ride-through: the ability of a load to stay connected through a voltage or frequency disturbance on the grid.

What I built

  • bmc/interconnect.py, a script that computes the service current at five voltages for the site, checks the transformer size against the loss of one, and computes the fault current at the 13.8 kV bus with the bus tie open and closed.
  • bmc/diagrams.py draws the substation figure.
  • A mock cost for the customer substation, added to the cost model from the redundancy post.

Two 138 kV lines, two 75 MVA transformers and a split 13.8 kV bus with a normally open tie

How it works

Site load. Twenty halls of 2.4 MW of IT at a PUE of 1.3 is 62.4 MW at the meter. At a power factor of 0.98 that is 63.7 MVA. These are the same assumptions as the earlier posts.

Service voltage. The current at the service shows why a site this size connects at transmission voltage. I = S / (√3 × V).

VoltageCurrent for 63.7 MVALevel
13.8 kV2,664 Adistribution
34.5 kV1,066 Asub-transmission
69 kV533 Asub-transmission
138 kV266 Atransmission
230 kV160 Atransmission

At 13.8 kV the site would need a bus above 2,600 A and several large feeders, and the utility would need to find that capacity on a distribution circuit. At 138 kV the same power fits on conductors of a few hundred amperes. In practice the voltage is set by what the utility has at the site, not by the customer, and the utility confirms it during the study.

Transformers and N-1. The service is two 138/13.8 kV transformers, each sized to carry the whole site alone. The site needs 63.7 MVA, so I chose 75 MVA units, which load to 85 percent when one is out. The ratings of real large transformers are stated at several cooling stages, so a real selection reads the rating at its highest stage. The two transformers are fed from two lines, and the figure shows each with its own breaker.

The bus tie and the short-circuit duty. The 13.8 kV bus is split in two. With the tie open each transformer feeds its own half. If a transformer is lost, closing the tie lets the other carry everything. The fault current says why the tie is normally open. With a 10 percent transformer impedance and an assumed 5,000 MVA utility source on the 138 kV side:

Bus conditionFault current at 13.8 kVFault MVA
One transformer, tie open27.3 kA652
Two transformers in parallel, tie closed48.3 kA1,154
Infinite utility behind one transformer31.4 kA751

Metal-clad switchgear is commonly built in standard short-circuit ratings such as 25, 31.5, 40, 50 and 63 kA, though the standard values for a given product come from its own listing. A 31.5 kA board would pass with the tie open and fail with it closed. A 50 kA board passes both. The choice is between paying for the higher rating and a closed tie, or keeping the tie open in normal operation and accepting a brief interruption to one side when a transformer is lost. A closed-transition transfer, where the tie closes before the failed source’s breaker opens, briefly makes the 48.3 kA case, so the switchgear has to be rated for it even if it lasts a few cycles.

Tie it back to the earlier posts. The 652 MVA in the first row is the right order of magnitude for the 500 MVA utility source I assumed in the short-circuit post. A real utility value, a real transformer impedance and the real tie status move every downstream fault number, which is why the utility’s available fault current data is the first input a study asks for.

Fit with 2N and distributed redundancy. The two sides of the bus map to side A and side B of a 2N hall. The shared parts are the point to examine: two lines from the same substation or the same right of way share a failure, and the diagram only shows two lines. Whether the utility’s two feeds are electrically and physically independent is a study and a negotiation, not something the customer draws.

The utility’s process. RMI describes six stages: a preliminary inquiry with size, voltage, location and date; an interconnection request; a feasibility study; a system impact study, which RMI calls the most critical step; a facilities study with construction-level design and cost; and an interconnection agreement, which may require significant financial security. It says the whole sequence takes anywhere from several months to several years. The ESIG large loads report says that steady-state, positive-sequence phasor-domain and short-circuit studies should be performed for every large load, and that an electromagnetic transient study may be required depending on the facility. It also says performance requirements such as voltage and frequency ride-through, power factor and ramp limits are incomplete or absent across NERC, states, market operators and utilities. Study durations differ by region, and one report puts MISO’s study timeline to an agreement at 120 days against 545 days for NYISO (Modo Energy).

What the utility will ask the design to show. From the ESIG recommendations: a steady-state and an electromagnetic transient model of the facility, submitted with the application and updated as the design changes; the ramp the load can produce, which ties to the GPU load post; the ride-through behavior of the load; the power factor; and how conformity will be verified during commissioning. A site with on-site generation or storage adds a generator or storage interconnection process on top, and ESIG notes that processes for co-located load and generation are limited and inconsistently defined.

Standards and power quality at the point of interconnection. The site’s equipment and its effect on the grid are judged against standards, and which one applies depends on how the site connects. IEEE 1547-2018 covers distributed resources connected to the utility. As NREL’s overview quotes it, a resource that finds itself in an unintentional island must detect it, cease to energize and trip within 2 seconds. IEEE 2800-2022 covers inverter-based resources connected to transmission, and applies at the plant level and not only to each unit. A joint industry webinar lists its capabilities: voltage and frequency ride-through, active and reactive power control, frequency response, power quality and negative-sequence behavior. A standard becomes a requirement when the grid operator or the utility adopts it in its interconnection rules, so the first question for a site with batteries or fuel cells is which version the utility applies.

Two other standards set what the site may do to the voltage. IEEE 519 sets harmonic limits at the point of common coupling. A transcription of the 2022 tables gives voltage distortion limits by bus voltage: for 1 kV up to 69 kV, 3.0 percent for any individual harmonic and 5.0 percent THD; above 69 kV up to 161 kV, 1.5 and 2.5 percent; above 161 kV, 1.0 and 1.5 percent; and for 1 kV and below, 5.0 and 8.0 percent. Current limits for systems from 120 V to 69 kV depend on the ratio of the utility’s short-circuit current to the site’s maximum demand current (Isc/IL). Below 20 the total demand distortion limit is 5 percent, from 20 to 50 it is 8, from 50 to 100 it is 12, from 100 to 1,000 it is 15 and above 1,000 it is 20 percent, with individual limits that fall as the harmonic order rises, and power generation facilities are held to the lowest-ratio limits whatever their ratio. The blog is a secondary source, so a compliance design uses the standard itself.

IEEE 1453 covers flicker, the repetitive voltage changes that make lights visibly fluctuate. PacifiCorp’s engineering handbook applies it as a utility policy: the short-term index Pst is measured over a 10-minute interval and the long-term index Plt over two hours, the compatibility level is Pst 1.0 and Plt 0.8 for low and medium voltage, and the planning levels for new installations are Pst 0.9 and Plt 0.7 at medium voltage and Pst 0.8 and Plt 0.6 at high and extra-high voltage. A synchronized training load has exactly the shape the index is built to measure. Turning 0.47 percent at 40 changes a minute into a Pst value needs the flicker meter algorithm or the standard’s curve, and a utility study does that with the load’s real profile.

The 2024 Virginia load-loss event. NERC’s incident review describes a lightning arrestor failure on a 230 kV line on 10 July 2024 that produced a lasting fault and a six-fault disturbance. About 1,500 MW of load in the same area dropped off, and all of it was data center load. Voltage during the faults ranged from 0.25 to 0.40 per unit. The data centers had transferred to backup. NERC says static UPS designs usually look like a short loss of load that returns quickly, and rotary UPS designs like a loss of load that does not return quickly. Most of the loss traced to the interaction between the line’s automatic reclosing sequence and a data center protection scheme that counts voltage disturbances in a window, with most of the sustained reduction at the third voltage depression, which matched the third reclose.

The lesson ties back to the UPS post. The transfer logic, its disturbance count and its return-to-utility behavior are not only a site-internal setting. A transfer that is reasonable for one site’s reliability can remove a large share of a region’s load in one event, which is why the utility asks for ride-through behavior and why a disturbance-count setting is an interconnection input.

Commands and results

python3 bmc/interconnect.py

The script prints the service current table and the fault duty table above, and a pass or fail for each standard switchgear rating with the tie open and closed. It uses the standard per-unit formula I = S_base / (√3 × V × Z) with the utility impedance added in series, and no motor or generator contribution.

Power quality of the training swing. The script’s poi function applies the swing of the GPU load post, 23 MW every 1.5 s, which is 40 changes a minute if all halls run in lock step. At the 138 kV point of interconnection with the assumed 5,000 MVA utility the rapid voltage change is 0.47 percent. On the MV bus with one transformer it is 3.6 percent and with the tie closed 2.0 percent. The short-circuit ratio at the point of interconnection is 79. The table above is for systems up to 69 kV, where a ratio of 79 would fall in the 50 to 100 row with a 12 percent total demand distortion limit. At 138 kV the point falls in the 69 to 161 kV class, which has its own, tighter current table that the source did not reproduce. The 138 kV planning level for flicker is Pst 0.8 and Plt 0.6. Whether 0.47 percent at 40 changes a minute passes depends on the flicker curve in the standard and on the utility’s own planning level, and a battery that smooths the swing, as in the previous posts, is the usual way to reduce it.

Mock cost of the customer substation. I added a substation line to the cost model. The unit prices are assumptions: 4 million dollars for each 75 MVA transformer, 1.2 million for each of four HV breaker bays, 2.5 million for the 13.8 kV switchgear lineup, and 6 million for the ground grid, relaying, controls, civil work and the line tap. That totals 21.3 million dollars, which is about 444 dollars per kW of IT for the 48 MW of IT, or about 8.5 percent of the 2N hall cost of 5,237 dollars per kW in the redundancy post. Utility network upgrades and any security the agreement requires come on top, and the RMI description of cost allocation means they can be larger than the customer’s own facilities. Long lead times for large transformers and HV equipment mean the schedule can be set by these items and not by the data hall.

Prefabricated substation equipment. The medium-voltage lineup and its protection are a natural candidate for factory assembly. Flex names MV substation e-houses and LV power pods as the modules that compress the schedule most, and Siemens describes a modular medium-voltage skid with switchgear and transformer built as a single factory-tested unit. The schedule savings quoted by vendors range from 30 to 50 percent and are marketing figures, so they are a claim to test against a project schedule, not an input. The engineering lives in the interfaces. For an e-house those are: the shipping and crane limits that cap the size of each piece; the cable terminations for the incoming HV or MV and the outgoing feeders; the connection to the site ground grid; the protection and controls communication with the rest of the plant; and the split of factory acceptance testing and site acceptance testing, which the commissioning post discusses. A design for offsite manufacture fixes these interfaces on drawings before the equipment is built.

Challenges

  • Two lines are not necessarily two sources. The drawing shows two independent lines. If they share a substation bus, a tower or a right of way, one event takes both. The N-1 transformer arrangement protects only the part of the system inside the fence.
  • Closing the tie changes the fault duty. The switchgear rating decides whether the tie can be closed in normal operation. A rating chosen for the open-tie case silently excludes the closed-tie case.
  • The utility’s model decides what you are allowed to do. The study uses the load model the customer submits. A load that ramps faster or rides through worse than the model is an operating risk and, under some agreements, a compliance one.
  • Backup transfer is an interconnection issue. The 2024 event came from backup transfer logic doing what it was set to do. Changing the setting can be a reliability change for the site and for the grid.
  • The schedule is set by equipment you do not control. Transformers and HV breakers have long lead times, and the study sequence takes months to years, so the first design decision is the one the utility makes.

Differences with a production setup

  • Real utility data. The source impedance, X/R ratio, minimum and maximum fault currents and the two-line independence come from the utility’s study. My 5,000 MVA source is an assumed value.
  • Real transformers. Impedance, tap range, cooling stages, loss data and tolerance come from the manufacturer’s test report. The 10 percent figure and the 75 MVA size are chosen for the example.
  • Protection. A real substation has line differential and transformer differential protection, breaker failure, bus protection and a protection coordination study with the utility. None of it is in my script.
  • Station design. Breaker-and-a-half, ring bus and double-bus layouts at HV have different maintenance and failure behavior, and the choice is made with the utility. The figure shows the simplest arrangement.
  • Grounding and safety. A substation ground grid is designed to limit step and touch voltage under a fault, using standards such as IEEE Std 80. The sizing needs soil resistivity data for the site.
  • Codes. Medium-voltage installations fall under the NEC for the customer side and under the National Electrical Safety Code for utility-owned equipment, and which applies where depends on who owns what at the POI.
  • Cost. The unit prices are assumptions. A real estimate comes from vendor quotes, the utility’s facilities study and the contractor’s pricing.

FAQ

Why does a 62 MW site connect at 138 kV?

The current at 13.8 kV would be over 2,600 A for the whole site, which a distribution circuit cannot supply. At 138 kV it is about 266 A. RMI says loads above roughly 20 MW typically connect at the transmission level.

Why keep the bus tie open?

Closing it puts both transformers in parallel and raises the fault current from 27.3 kA to 48.3 kA in this example. An open tie holds the fault duty at the lower value.

What is the most important utility study for the customer?

RMI calls the system impact study the most critical step, since it determines the upgrades needed and who pays for them.

What does ride-through have to do with the design?

Under the NERC review, load that transferred to backup in response to repeated voltage dips was the main cause of the 1,500 MW loss in 2024. The site’s transfer logic and its settings are what the utility will ask about.

Is an e-house cheaper?

Vendors claim 30 to 50 percent shorter schedules, which is a marketing claim. The measured value depends on the project’s sequence, shipping constraints and how well the interfaces are defined before manufacture.

What does the customer own?

It depends on the utility and the agreement. RMI says customer interconnection facilities are typically allocated entirely to the customer, and ownership of the POI substation depends on the utility’s rules.

Repeating with real hardware

There is no home-scale version of a transmission interconnection. The repeat that fits is a desk review of a real utility’s large load requirements and the single-line diagram of a real customer substation, checking each number in this post against them.

Next

Take the quiz to test your understanding: Electrical series quiz

The next post in this series covers redundancy architectures and single-line diagrams for the hall.

Comments