BART's power surge problem, which had been frying train electronics and took some 85 cars out of service at its height, has gone away and service is more or less back to normal. Here it is from BARTs own statement:
After swapping out the generation of train cars most prone to damage (our "C" cars) to help establish regular service again, the spikes in voltage along the track have gone away. So we have since moved the C cars back to add more cars to the line, and the problem has not reoccurred.
"The problem has gone away," said Paul Oversier, Assistant General Manager of Operations, "but we need to get to the bottom of this. We don't want our customers to suffer through another round of this so we need to get to the root cause."
In other words, they haven't fixed the problem. It went away on its own.
Henry Kolesar, Group Manager, Vehicle Maintenance Engineer at BART's Hayward repair facility, gave Streetsblog a tour and showed us the parts that are causing the problems. Kolesar also endeavored to explain exactly what's going on with the electrical engineering (and we think we get it).
The key component that got fried is called a thyristor. That's already been widely reported. So what is a thyristor exactly? The motors on BART trains run on 1000 volts of power from the third rail. But BART trains are controlled by computers. The computers do all the calculations about how fast the motor needs to turn to assure that the train is going at the proper rate and that it's stopping and starting exactly where it needs to. The computers figure out how much voltage to apply to the motors to achieve the proper speed and acceleration. Here's the problem: the computers run on 36.5 volts.
So you need a device that can take the signal from the 36.5 volt computer and use it to regulate the 1,000 volts of power coming into the train motor. Obviously, you can't just plug the motor into the computer's circuits, or it would turn into a smoking glob of mush.
In come solid-state switches, such as thyristors, that take a low voltage signal from a computer and interpret it to adjust the high voltage appropriately for the motors.
To keep the high voltage away from the low voltage, the thyristor--which looks a bit like a silver dollar--is encased in a stack of porcelain insulators that do not conduct electricity. The insulators have corrugations (gaps and ridges along the edges) that keep the high power electricity from creeping along the edges or sparking through the air and ruining the components.
Of course, anyone who has watched a BART train (or the overhead power collector on a Muni train or trolley bus) long enough has seen the occasional sparks coming from the contacts that connect the train to the third rail. That's because electricity, at high voltage, can travel short distances through thin air (at really high voltages, electricity can travel very far, as happens with lightening). So in that split second when a third-rail collector shoe comes out of contact with the third rail, electricity will actually jump through the air.
And, "When the 1000 volts of electricity jumps or creeps across the surfaces where it's not supposed to, it causes damage," explained Kolesar (as you can see in the photo of the burned insulator below).
So it's really, really, really important that the insulators around the thryistors have enough corrugations to stop the high voltage electricity from traveling through the air to the low voltage part of the device.
The size of the insulators is dictated by math. There has to be a minimum distance between those ridges in the corrugations to keep the electrical components safe. If you're trying to run something on 600 volts, you need a certain size ridge; if you run on 1000 volts, you need more, and so on.
In other words, "What's been happening to our trains--it's mathematically impossible for the voltage to do what it's been doing," said Kolesar.
But it's happening. Now, before you go there, this is not something that can endanger the passengers. It all takes place in the electrical boxes under the train. But it means the car's controls get fried and the motors won't work anymore. The train's overall performance decreases. Eventually, if enough thyristors get fried, the whole train has to be taken out of service.
So why is it happening?
At this point, the engineers just don't know. The thyristor packs on BART trains are actually rated to take 2.5 times the voltage the train actually uses. Kolesar tested them in the shop at five times the rated voltage, and they still tolerated it. "But they flashed in service," he said.
There are things that could explain what's going on. For example, if enough dirt collected on the thyristor insulators the voltage could travel through it and burn out the thyristor. But "the ones that failed were clean," said Kolesar.
There are some wilder ideas. You know those home air filters you can buy with an ionizer? Ionized air is supposed to be healthy, in theory. But sufficiently ionized air can build up around electronic components and make the air more conductive to electricity: in other words, make it easier for sparks to travel. "We are also focusing on the propulsion box to see whether the arcing was originating there," said Paul Oversier, Assistant General Manager of Operations for BART, in the prepared statement. "According to all the equations, this shouldn't be happening."
But so far, said Kolesar, they haven't been able to identify a source of ionized air, although they have some theories they're currently testing out. And there's a problem with that theory anyway: why is it happening now? He said there are no new components on the trains that can explain it. Sabotage is another thing they've discussed, but Kolesar says--without getting into details that might give anyone any ideas--they've ruled that out too.
Kolesar says reports about BART using non-standard parts--and its contribution to the problem--are exaggerated. Although BART runs at an unusually high voltage for a Metro rail system, it's not unprecedented, and the thyristors themselves are used in many different applications. So they really are off-the-shelf components, although BART has had some trouble finding replacements quickly.
For Kolesar, however, the most frustrating part is that they haven't been able reproduce the problem and it has now mysteriously gone away. Nothing drives an engineer crazier than to have the problem stop without knowing why it happened in the first place, he said.
For now, they're taking slices of rails, brakes, and other components that are regularly changed during maintenance and sending them to a lab. One theory is that a manufacturer may have changed the formula sightly in the steel they use, so as it wears it is producing particles that are interacting with the electrical systems to produce ionized gas. Kolesar is working with different departments at BART, trying to figure out what may have gotten introduced into the system. "Have you changed suppliers of rail? Or third rail pads?" he asked.
For riders, though, the bottom line is that the problem still isn't solved. That means one of these days a bunch of BART cars are going to fail again, with the resulting delays and headaches for commuters. The good news is that some really smart people are working hard trying to figure it out.
Kolesar, meanwhile, claims he had a full head of hair before the problem started.
