Leoch DJW12-5.0(F2 12V5.0Ah) Maintenance-free Sealed Lead-acid Battery cohort testing and refilling (WHL #79)
Batteries. Plain old lead-acid batteries for today.
Before I left to Scotland, IT dropped a bunch of UPS batteries in my office. And since I wasn’t present, they asked my colleague to get them tested, you know, just sort out the bad ones. Sure thing.
It’s 119 of them, plus two dozen more of a larger size (9Ah) that are more common in those desktop APC units that I also use at home. All in all over 200kg of lead. Well, thanks a lot…
Thing is: I don’t have time for that. And it’s not my job. And with the limited equipment I have at hand (at work…), it took me well over a month to get them all tested alongside regular work. Four a day, six a day, maybe eight if it’s a really long one or if the batteries are pretty dead. Should take as little time as possible, so the testing plan was:
a) Measure voltage.
If they’re well beyond end-of-discharge voltage, forget about them. I do not know how long they sat before they were brought here, could be months given it was our IT team, but any battery that has either sat over-discharged for a significant of time, or has a rate of self-discharge so that it kills itself within days – toss it. This, interestingly, already accounts for 31 units, plus one that was a bit puffy, and three more that showed a certain usable voltage, but were of such high impedance that they could not deliver any current.
b) Discharge to a common level.
While it would be great to have an exact level to start any further tests, I’m having two issues here: One is that I do not have any 2/4 quadrant power supplies, so that I need to use a regular load that is inherently inaccurate when discharging things (say a resistor – that needs constant supervision to stop the process). Two is that even if I have all of them linked together and know they’re the same voltage – I simply cannot test them all at once, so they sit and discharge and there goes my comparison base. So my solution was an old desktop UPS unit, actually the very same APC 700VA one that I own (and have modified) myself. Add extension leads to not wear out the internal ones that are also a bit short, and then discharge batteries as needed. UPS load is the same old laptop for all units, so same discharge profile for everyone. Sure, temperature and the time they sat might have differed, but I always used discharged batteries the same day, none sat overnight. This limits my throughput, but we want a level field for every player, right?
c) Charge to a common level.
Now that we do have empty batteries, charging them is the final step for a first result. I do have a dual lab power supply in my office that was set to 1.0(0)A and 14.0V, had two pairs of identical leads for the outputs, and just connected them. I noted the time of both start of charge and end of charge, which I defined as the moment they drop below one amp of charging current. Time difference in hours times 1A equals charge taken in amp hours, simple as that. Of course there’s a few gotchas here, for example once again watching the whole thing all the time wasn’t possible. But since my time resolution is 0.1 hours or six minutes, and charging current actually declines rapidly, backdating a couple minutes is pretty easy. I re-did a couple units that I completely missed and I also tried again units that I had to use a different power supply for (0.1 to 0.2Ah difference despite much lower resolution, so pretty alright). Stopping the charge cycle over night is a bit of an issue and likely diminishes readings by around 0.1 to 0.2Ah, especially when stopped on high SOC – so I avoided that and only tested batteries that would finish before the end of the work day. Also, please note that this figure is not total battery capacity, which would be tremendously sad for “5.0Ah” rated batteries with none even reaching 3.5Ah. A lot of energy can be added after this drop of charging current, and I did not discharge the batteries as low as the datasheet allows for in the C/20 or other light use scenarios. But this 1A rule is a practical way to compare identical batteries with each other.
d) Weigh them.
This is purely optional and I did that on my own time after clocking out, but weight is pretty much the only thing that distinguishes the remaining 85 identical batteries and can be measured in a matter minutes instead of hours. It turns out that, except for real outliers, this is a waste of time and cannot predict battery performance with suitable accuracy. However, I suspect (more on that later) that weight loss might be a good indicator, but that would of course require taking that data before assembly into a new UPS battery pack. Any battery above 1860 grams was perfectly fine and any below 1785 grams was toast…but that was like three units each, so 5% of the total population. Not a great indicator at all.
Anyway, this data yields this graph, already color-coded with later results (uuuh, foreboding)
How do we separate the “OK” ones from the “Meh” units? Well…since they’re now all charged to a somewhat similar level, let’s discharge them again. On a higher load. The initial discharge was like 40 to 50W to get them all to a very low level independently of their impedance, this one is more on the 150 to 200W side. It’s another computer, a desktop this time, that also shows the same power profile for all candidates since it just POSTs and does not find a boot disk, so it sits around consuming a pretty static amount of power after around a minute of mayhem. Side note: I initially tried using a 400W heating plate which was my best option of “low hundreds of watts, does not rotate or cause damage when unsupervised”, but that draws so much power, it triggers the low voltage threshold of any battery including the very best ones. Frankly, 400W at 12ish volts is shy of 40 amps including the converter efficiency, so that is a bit much for those small 5Ah cells with rated currents up to a 5-minute-discharge at 20A.
Running this test yields 27 batteries that do not even make it to the 30 second mark (ranging from 1.3Ah to 3.1Ah), there’s 49 batteries that do offer one to six minutes of runtime (ranging from 1.0Ah to 3.3Ah), and there’s eight more units from 0.5Ah to 2.0Ah that developed one or more significantly hot cells during the test – checked by touch, verified with my thermal imaging camera. Those have been moved to the category of bad batteries since I would not want any candidates for an imminent cell defect in the “yup, those work” group.
Does the runtime indicate anything vs. the weight graph, can we sort that way?
Nah, we can’t. We can establish a new lower bound of “good” batteries at 1813 grams since none below that was able to deliver 30s or more of high-current runtime, but that’s about it. 70% of the “meh” group is also above that threshold, so it is not all that useful. There’s a couple of low-capacity high-current batteries, and a couple of high-capacity low-current ones. Depending on what you define as passing goal, either of them might qualify.
We can also plot runtime vs. measured charge. Is that helpful? Nope again.
It defines a lower threshold of 2.1Ah for 5+ minutes of high-current usage and 2.2Ah for 6 minutes, but this selection rule obviously requires the longer test to be run beforehand. Let’s keep the 2.1Ah mark since the other does not really improve things – 2.1Ah requirement rules out 70% of “meh” batteries and also 16% of the good ones, but another 53% of the good ones pass that limit without actually reaching five or more minutes of runtime. This includes the one unit that scored the highest capacity of them all – would you want to skip this 3.4Ah/4min battery in favour of a 2.1Ah/5 minute one? So that’s also total crap for easy selection. Let’s just say it’s not easy and there’s a reason for why there’s no ten dollar China tester for testing that just nails it.
But: This very last test actually allows for a quick experiment. What if those batteries, which should be around three to four years of age, could (in theory) be revitalized instead of binned and replaced? Every UPS manufacturer of course recommends changing them every so often, every one of them offers replacements (or even replacement plans) for a pretty penny, and every one of them does charge the batteries a liiitle more than necessary so that you get maximum performance. Sure, in this world of safety protocols, avoiding responsibilities by just throwing money at it, as well as having less qualified staff to do important work (this is where the money could be spent otherwise), this might be the way forward (except for the environment). But the lead-acid battery is such an old concept that we do know how it ages – it loses water via the H2/O2 split when charged over a certain threshold, and this is why forklifts get the occasional sip of distilled water when charging, as did car batteries before the “sealed battery” was invented. You know, AGM and non-spillable batteries are cool inventions, but they fundamentally use the same chemistry that just loses water more slowly, but “cannot” be refilled – they’re binned instead.
The Eaton 9SX 6000 UPS from my example does have an indicated charge voltage of 209 volts. At 15 batteries in series, that’s 13.93V per unit. Might be 210V, which is 14V sharp. The Leoch DJW12-5.0 datasheet is clear on this: Standby Use: 13.5V~13.8V at 25°C with a temp coefficient of -0.02V/K. So they’re already 0.2V above rated use voltage and then, since they’re used in a rack cabinet with high-power servers, 35°C ambient of the non-ventilated battery tray means another 0.2V over spec. Anyone knows why things degrade faster when run way over spec? 
Anyway, I modified two batteries that were conveniently damaged, as in “the lid fell off”. Beneath the lid is six white fluffy rings that I probably do not want to know what it is made of, and rubber plugs inside to each of the six cells of the battery. One battery was filled ahead of testing, marked discreetly and then came out as one of only five units that were able to provide six minutes of power in the heavy-use scenario (it’s the 2.2Ah one, so significantly lower capacity than the other 3.0 to 3.3Ah ones, but made it to 2.8Ah in a second test later on). The other one was tested as-is, 2.4Ah and only one minute of runtime. It was then refilled with 4 to 7.5ml of water per cell until they all looked alike, and re-tested. From testing unmodified batteries I know that the discharge-charge cycle does not alter parameters beyond my overall resolution, so it’s not like the simple act of using them resolves some kind of lazy battery effect. This 2.4Ah/1min battery also scored six minutes of runtime in a second test – just barely, but even five minutes are hugely impressive. So without doing the same thing to all other 74 eligible batteries with their 444 refilling holes, I think adding a sip of water to restore a) filling level and b) acid density to near factory condition makes a huge difference and could save a lot of them from being binned prematurely. Again, very few people would actually do this in a commercial environment due to red tape and the blame game if for some reason something goes wrong, but if these batteries are thrown out and can be had for nothing – why not refresh them before they go into your home UPS?
Conclusion, findings, advice:
1) Test your UPS on a regular basis: One cell or entire battery might already be dead, which will only become obvious when you actually need the UPS to work. Here, a full 29% of the population was not fit for service and likely every of the eight UPSes they came from had one or more of these inside. Glad we exchanged them, sad there were so many.
2) There is no obvious sign for a good battery. There’s obvious signs for bad ones, mainly standby voltages of less than nine volts and physical deformation of the battery case, but good batteries and bad batteries weigh the same, look the same, smell the same (please do not lick them). One of them however fucks you over when you need it the most.
3) Servicing an unserviceable “sealed” (my ass!) battery can improve performance short-term and likely will do so long-term as well – it’s just that adding 90 controlled sips of water per UPS is no option in a commercial environment, but might very well be for home use. Currently new lead-acid batteries start at 90 cents per Ah (car battery size), smaller ones specifically for UPS use are significantly more expensive. While I can easily refill 15 tiny batteries in an hour, I do not make 70€/hour at work (75Ah equivalent), let alone after taxes – so preserving existing units is of course way more economical than just buying new.
4) The first version of a re-assembled pack for the Eaton UPS clocked in at 11:50 minutes for supporting a 2000W space heater, with a specced runtime of “15 minutes” by Eaton itself. Those fifteen batteries were chosen by the capacity figures alone. An optimized pack based on both capacity and high-current durability plus temperature-optimized placement of the individual batteries gave a 13:20 minute runtime. Not saying this is “good as new” numbers, but this will certainly do for the spare unit that the IT apprentices can play with until it gets deployed to a rack cabinet with like 200W worth of switches and a fibre modem. 43 batteries will be binned, 15 are in that pack, which makes 61 leftover units…
5) LiFePO = LFP batteries are currently 10x more expensive than lead-acid ones, but that number gets smaller every year due to the massive ramp-up in battery production for electric cars and other high-volume products. Given the ongoing cost of lead-acid battery replacement that isn’t simply necessary for LFP, the ~100€/year for that Eaton unit (~3000€) will sooner or later trigger the transition to a LFP UPS. That one will also not require permanent overcharging of the battery (as it will fucking burn your fab down…no, it won’t, it’s LFP), so fixed cost will be lower as well. Said Eaton UPS has a constant standby power draw of 95 watts when fully charged – that’s 830kWh/a which outweighs the battery cost by far (~4x!). And this doesn’t even include cooling, since rule of thumb is one Watt goes to the A/C systems for every Watt that is used in the server cabinet. I’d even predict LFP is the standard battery in UPSes by 2030 for those reasons. Until then: There’s a ton of old lead-acid batteries that can be had for cheap!
