Tough choices... The algorithmic trading server I worked on a few years ago used three different numeric types to represent quantities of money (all USD, in its case). We used scaled integers (32 bit scaled by 100) to represent stock prices (trades, bid, and ask) because that was known to be sufficient precision for the task and was fast because those integers are native types. We used scaled longs (64 bits scaled by 10000) for foreign exchange prices (also trades, bid, and ask) for the same reasons. Finally, we used BigDecimal (Java's built-in decimal with arbitrary precision) for aggregate calculations because it was the only built-in decimal type with enough precision, and because calculations like that were relatively infrequent and the performance penalty wasn't too bad.
But using several different types for the same logical purpose had very bad consequences for our system's reliability. Why? Because on every calculation involving money in the entire system, the programmer had to carefully think about the operands, make sure they were in the same value representation, convert them correctly if not, do the calculation, then convert the result (if necessary) to the desired type. At the same time, the programmer had to correctly anticipate and handle any possible overflow.
So how bad could that be? Here's a real-world example from that same algorithmic trading server. Periodically we would look at the bid/ask stack for particular stocks within 5% of the last trade price. For a highly liquid stock that might involve several thousand entries, where each entry was a price and a quantity. We needed to get the sum of the extended price for each entry, and then an average price for the ask stack and separately for the bid stack. The prices were in scaled integers, the quantities also in integers - but their product could exceed what could be held even in our scaled longs. So we had to convert each price and each quantity to a BigDecimal, then do the multiplication to get the extended price. Then we had to sum all those extended prices separately for the bid and ask stack. We could sum the quantities safely in a long, so we did. Then we had to convert that summed quantity to a BigDecimal so we could divide the sum of extended prices by the summed quantity. The result was a BigDecimal, but we needed a scaled integer for the result – so we had to convert it back. That means we needed to (very carefully!) check for overflow, and we also had to set the BigDecimal's rounding mode correctly. Nearly every step of that process had an error in it, due to some oversight on the programmer's part. We spent days tracking those errors down and fixing them. And that's just one of hundreds of similar calculations that one server was doing!
Ideally there would be one way to represent the quantity of money that would use scaled integers when feasible, and arbitrary precision numbers when it wasn't. It's certainly possible to have a single Java class with a single API that “wrapped” several internal representations. This is much easier if the instances of the class are immutable, which would be good design practice in any case. The basic rule for that class at construction time would be to choose the most performant representation that has the precision required. We could have used such a class on that algorithmic trading server, and that would have saved us many errors – but none of us thought of it at the time...
Note: this is the last post of this series on representing and calculating monetary values in Java. I've provided a link for the series at right. If perchance some other semi-coherent thought occurs to me, I reserve the right to add to this mess...
Thursday, April 20, 2017
Wednesday, April 19, 2017
Storing and transmitting monetary values...
Storing and transmitting monetary values... Most financial applications these days are not standalone applications, but rather components of much larger and more complicated systems. In a past life I was the CTO of a firm that made stock and option trading software, and I once made a map of all the systems that our servers communicated with. There were about a dozen separate machines in our own server room that talked to each other. There were also over two hundred other applications (in over thirty other organizations) that our servers talked to. Those communications paths used a dozen or so different protocols, not counting minor variations. In addition to all these connections, our servers also stored financial information: in memory, on disk, and in logs. That system was not an outlier – I've worked on dozens of other systems just as complex, or even more so.
This complexity has lots of implications when it comes to representing monetary values. What jumps right out to anyone examining all these needs is that there is no “one way” to represent a monetary value that is correct, best, or optimal.
Often the representation has to match an existing standard. For instance, perhaps the monetary value must be in the form of an ASCII string, like this:
When storing or transmitting large numbers of values, often a compact (or compressed) encoding is essential. I once worked on a server that did algorithmic trading (that is, the program decided what stocks to buy and sell). This server needed to store (both on disk and in memory) huge tables of stock pricing data. On busy days this could amount to several hundred million entries. The original design of the server represented the monetary values in a fixed length binary structure that occupied 22 bytes, so on busy days we were storing a couple of gigabytes of data – and moving that data between disk, memory, and CPU cache. We observed that this was a bottleneck in our code, so we came up with a variable-length encoding that averaged just 7.5 bytes per value. We rolled our own on this one, as the numeric representation we were using didn't have a compact encoding. The performance impact was stunning, much more than the 3x you might naively expect. The biggest impact came from fewer CPU cache misses, the second biggest from faster encoding/decoding time. The latter we didn't expect at all. The real lesson for me: compact encodings are very valuable!
Databases present another challenge for representing monetary values. The numeric column types available often can be specified with the precision required – but the API will require converting the value to or from a string or an unlimited precision value (such as BigDecimal). Sometimes the values don't need to be queried, but rather simply stored, as key-accessed blobs. In those cases, a compact encoding is valuable, especially for large tables of values accessed as a whole.
From all this we can derive another requirement for a monetary value representation: flexibility of encodings. There should be a variety of supported, standard encodings including (especially) a compact, variable-length binary encoding, a fixed-length binary encoding. There should be support for conversions to and from unlimited precision numbers as well as native types. There should be support for conversions to and from strings. Also very important, for flexibility: support for adding encodings – because there's no way to anticipate all the strange things the world will require of you...
This complexity has lots of implications when it comes to representing monetary values. What jumps right out to anyone examining all these needs is that there is no “one way” to represent a monetary value that is correct, best, or optimal.
Often the representation has to match an existing standard. For instance, perhaps the monetary value must be in the form of an ASCII string, like this:
$4532.97. The standard might specify that it be in a fixed-length field, with spaces padding to the left as required. Or the standard might specify that the value be in XML, encoded in UTF-16, with a tag of “AMT” for the amount, and “CUR” for the three-character ISO currency code. Some older protocols have specified binary formats, with a numeric code for the currency and packed BCD for the value. I could go on and on with these, because many clever and/or crazy people have devoted a great amount of time to inventing all these schemes.When storing or transmitting large numbers of values, often a compact (or compressed) encoding is essential. I once worked on a server that did algorithmic trading (that is, the program decided what stocks to buy and sell). This server needed to store (both on disk and in memory) huge tables of stock pricing data. On busy days this could amount to several hundred million entries. The original design of the server represented the monetary values in a fixed length binary structure that occupied 22 bytes, so on busy days we were storing a couple of gigabytes of data – and moving that data between disk, memory, and CPU cache. We observed that this was a bottleneck in our code, so we came up with a variable-length encoding that averaged just 7.5 bytes per value. We rolled our own on this one, as the numeric representation we were using didn't have a compact encoding. The performance impact was stunning, much more than the 3x you might naively expect. The biggest impact came from fewer CPU cache misses, the second biggest from faster encoding/decoding time. The latter we didn't expect at all. The real lesson for me: compact encodings are very valuable!
Databases present another challenge for representing monetary values. The numeric column types available often can be specified with the precision required – but the API will require converting the value to or from a string or an unlimited precision value (such as BigDecimal). Sometimes the values don't need to be queried, but rather simply stored, as key-accessed blobs. In those cases, a compact encoding is valuable, especially for large tables of values accessed as a whole.
From all this we can derive another requirement for a monetary value representation: flexibility of encodings. There should be a variety of supported, standard encodings including (especially) a compact, variable-length binary encoding, a fixed-length binary encoding. There should be support for conversions to and from unlimited precision numbers as well as native types. There should be support for conversions to and from strings. Also very important, for flexibility: support for adding encodings – because there's no way to anticipate all the strange things the world will require of you...
Paradise ponders, thermostat monitor, borosilicate glass, and excellent beef soup edition...
Paradise ponders, thermostat monitor, borosilicate glass, and excellent beef soup edition... After all the work we've had done to our downstairs furnace, it still has the original problem that started last December: it cycles on and off every 5 minutes or so. The changes made have greatly increased the air flow, especially in our basement cattery, and that's a very good thing. I'm happy to have made the changes even if they didn't fix the issue. Nevertheless, we still need to find (and fix!) the cause of this rapid cycling.
One of the possible culprits is our Nest thermostat. When the furnace is all buttoned up, there isn't any exterior indication of whether the thermostat is “calling” for heat. So yesterday afternoon I built a thermostat monitor. This is a ridiculously simple piece of electronics: just three pre-packaged 24V (AC or DC) LED monitors. One of them monitors the 24V power to the thermostat, another monitors the “call” signal, and the last one monitors the fan signal. The fan signal is used only when we want to run the fan without heat. I packaged them in a Ziploc food container, and mounted it to the furnace with a magnet. The really important one is the call signal. Here's some photos of my monitor in place:
After I put that in place, I sat and watched two complete cycles of the heater. The call signal never wavered – which tells us that the thermostat is doing its job perfectly, and the problem is somewhere in the heater. In the first photo above, if you embiggen it and look just to the left of those labels, near the bottom of the labels, you'll see a dim neon bulb showing. That is normally solidly on, indicating that all is well. I noticed something last night that we've all managed to miss before. When the heater has been on for a couple of minutes, the burner goes out (gas turned off) while the fan is still going. For the next 30 seconds or so after that, the neon light is blinking a two digit code: 3 short blinks, then 1 long blink, so “31”. Then after the 30 seconds is up, the fan stops and the neon light turns back to solidly on.
That 31 code means a problem in the low pressure sensor. This is a safety device that ensures the exhaust gases are going out the vent and not into the mechanical room – a good thing! My nose doesn't detect any exhaust gases in there, so I suspect the problem is not a lack of draft, but rather a malfunction in that switch. In any case, this is a certifiable clue as to the source of our cycling problem. That means we might actually be able to get it fixed! Yay!
Yesterday was Tuesday, and that means beef soup at Los Primos. Debbie didn't feel up to going out, so I ran up and got two takeout orders of soup. The folks there were disappointed not to see Debbie, and asked after her health. They told me to make sure she shows up next Tuesday. :) The food there is really special, but the people at the family-owned restaurant are even more so...
A few months ago, we bought eight of these tumblers. I thought I'd posted about them earlier, but I can't find anything on them. Debbie and I both tend to drink big glasses of stuff: water, milk, iced tea, and the like. Cold drinks, mostly, and often iced. These tumblers caught my eye because they're insulated – there's two layers of glass with a vacuum between them, just like an old-fashioned vacuum thermos bottle. Plus they're big: 20 ounces. The glasses are made from borosilicate glass, a material I know well for its unusual combination of strength and low thermal expansion coefficient. It's also very clear. The fact that they have a low thermal expansion coefficient means they're safe for both hot and cold drinks, plus you can throw them in the dishwasher. We've been using them for all our cold drinks, and we both love them. I'm especially appreciative of the rounded rim (comfortable in my mouth) and the sturdiness – we still have all 8 of them, despite daily use for 3 or 4 months now. Looking on the Amazon page I see that there are a few very negative (one star) reviews, mainly for three reasons: their weight, the thickness of the rim, and their alleged fragility. We actually like the weight and the thick rim, and I'm guessing that most people do as the reviews are overwhelmingly positive. As to the fragility, I can only say that these are lasting longer than the majority of glassware we own. :) Anyway, these are resoundingly endorsed by the Dilatushes of Paradise!
One of the possible culprits is our Nest thermostat. When the furnace is all buttoned up, there isn't any exterior indication of whether the thermostat is “calling” for heat. So yesterday afternoon I built a thermostat monitor. This is a ridiculously simple piece of electronics: just three pre-packaged 24V (AC or DC) LED monitors. One of them monitors the 24V power to the thermostat, another monitors the “call” signal, and the last one monitors the fan signal. The fan signal is used only when we want to run the fan without heat. I packaged them in a Ziploc food container, and mounted it to the furnace with a magnet. The really important one is the call signal. Here's some photos of my monitor in place:
After I put that in place, I sat and watched two complete cycles of the heater. The call signal never wavered – which tells us that the thermostat is doing its job perfectly, and the problem is somewhere in the heater. In the first photo above, if you embiggen it and look just to the left of those labels, near the bottom of the labels, you'll see a dim neon bulb showing. That is normally solidly on, indicating that all is well. I noticed something last night that we've all managed to miss before. When the heater has been on for a couple of minutes, the burner goes out (gas turned off) while the fan is still going. For the next 30 seconds or so after that, the neon light is blinking a two digit code: 3 short blinks, then 1 long blink, so “31”. Then after the 30 seconds is up, the fan stops and the neon light turns back to solidly on.
That 31 code means a problem in the low pressure sensor. This is a safety device that ensures the exhaust gases are going out the vent and not into the mechanical room – a good thing! My nose doesn't detect any exhaust gases in there, so I suspect the problem is not a lack of draft, but rather a malfunction in that switch. In any case, this is a certifiable clue as to the source of our cycling problem. That means we might actually be able to get it fixed! Yay!
Yesterday was Tuesday, and that means beef soup at Los Primos. Debbie didn't feel up to going out, so I ran up and got two takeout orders of soup. The folks there were disappointed not to see Debbie, and asked after her health. They told me to make sure she shows up next Tuesday. :) The food there is really special, but the people at the family-owned restaurant are even more so...
Tuesday, April 18, 2017
Inexact results and monetary calculations...
Inexact results and monetary calculations... There are a few things worth pondering about making calculations with money where the results are not exact quantities.
Consider the case of inexact results that are expected to be inexact. A common example of this is a compound interest calculation that involves logarithms, powers, and roots – not much chance of an exact answer on those! In cases like this, what we need for money is for the answer to be rounded to the nearest conventional unit of money (for instance, the nearest penny in USD). This is, however, a bit more complicated than you might think.
Consider a case in USD, where we want to round to the nearest penny. Suppose our unrounded result was 0.121 – that's easy, the rounded result is 0.12, rounded down (toward zero). Similarly, 0.346 would rounded up to 0.35. Both of those are obvious and uncontroversial. But suppose our unrounded result was 0.115? Do we round that down to 0.11, or up to 0.12? In both cases, the difference between the rounded and unrounded values is the same: 0.05. How do we choose between rounding up or rounding down?
Most of us old enough to predate the “new math” were taught in elementary school to round such values up, all the time. I have no idea what the hell kids are taught these days, except I'm confident it's not very useful. However, always rounding halfway values up over a large number of results introduces a bias toward larger average results. Bankers and their customers noticed this a long, long time ago. Some clever accountants came up with a simple solution that's easy to implement and removes the bias: “half even” rounding. In this kind of rounding, a result like 0.115 is rounded so that the next most significant digit is rounded to an even value – so round up to 0.12 in this example, because 2 is even and 3 is not. If the unrounded result was 0.165, though, then we'd round down to 0.16, because 6 is even. I should note that one could also use a “half odd” rounding to accomplish the same thing.
In some kinds of financial calculations, though, even this solution isn't what's needed. For instance, in certain kinds of models you always want to round toward zero. There are a half-dozen or so different kinds of rounding that are occasionally useful in monetary calculations. The way monetary quantities are represented really needs to support all of these rounding flavors.
Now lets consider a different case: where results are expected to be exact, and an inexact result indicates a mistake of some kind. I ran into a case like this in a stock trading application, where we multiplied the number of shares bought or sold times the sales price to get a “lot price”. Since shares of stock are indivisible (e.g., you can't buy 1.5 shares of IBM), that result should always be exact, to the penny. If it's inexact, then something is wrong – perhaps someone mistakenly entered a fractional share quantity, or there's a bug in the program. For these sorts of situations, it is very useful to know whether a monetary quantity is exact or inexact (e.g., has been rounded). The way monetary quantities are represented should support this.
Finally, sometimes in monetary calculations we really don't want inexact values to be rounded. For example, suppose we had 10,000 USD that we want to divide in thirds and distribute to three accounts. The rounded, inexact result of that would be 3,333.33 USD that we put into each account – but that only adds up to 9,999.99 USD – we “lost” a penny and now our books are out of whack. The way financial applications generally solve problems of this type is to use some algorithm to choose a lucky account, and they give that account the extra penny (in this case). These algorithms themselves are interesting, as they need to be repeatable (so you can't just roll the dice) for audits, but they are not the problem I'm discussing here. It's knowing that we have an inexact result, and how much is “left over” to distribute that I care about today.
This sort of problem always has a division operation at the root of it. The general solution is really simple: you need a division operation that gives you the floor of the quotient, and the remainder. So the result of the division in the example above would be 3,333.33 with a remainder of 0.01. The way monetary quantities are represented must include the division-with-remainder operation. Ideally it would allow returning quotients that are floor, ceiling, nearest toward zero, or nearest away from zero (all with appropriately adjusted remainders) because all of these are useful in some financial applications.
Consider the case of inexact results that are expected to be inexact. A common example of this is a compound interest calculation that involves logarithms, powers, and roots – not much chance of an exact answer on those! In cases like this, what we need for money is for the answer to be rounded to the nearest conventional unit of money (for instance, the nearest penny in USD). This is, however, a bit more complicated than you might think.
Consider a case in USD, where we want to round to the nearest penny. Suppose our unrounded result was 0.121 – that's easy, the rounded result is 0.12, rounded down (toward zero). Similarly, 0.346 would rounded up to 0.35. Both of those are obvious and uncontroversial. But suppose our unrounded result was 0.115? Do we round that down to 0.11, or up to 0.12? In both cases, the difference between the rounded and unrounded values is the same: 0.05. How do we choose between rounding up or rounding down?
Most of us old enough to predate the “new math” were taught in elementary school to round such values up, all the time. I have no idea what the hell kids are taught these days, except I'm confident it's not very useful. However, always rounding halfway values up over a large number of results introduces a bias toward larger average results. Bankers and their customers noticed this a long, long time ago. Some clever accountants came up with a simple solution that's easy to implement and removes the bias: “half even” rounding. In this kind of rounding, a result like 0.115 is rounded so that the next most significant digit is rounded to an even value – so round up to 0.12 in this example, because 2 is even and 3 is not. If the unrounded result was 0.165, though, then we'd round down to 0.16, because 6 is even. I should note that one could also use a “half odd” rounding to accomplish the same thing.
In some kinds of financial calculations, though, even this solution isn't what's needed. For instance, in certain kinds of models you always want to round toward zero. There are a half-dozen or so different kinds of rounding that are occasionally useful in monetary calculations. The way monetary quantities are represented really needs to support all of these rounding flavors.
Now lets consider a different case: where results are expected to be exact, and an inexact result indicates a mistake of some kind. I ran into a case like this in a stock trading application, where we multiplied the number of shares bought or sold times the sales price to get a “lot price”. Since shares of stock are indivisible (e.g., you can't buy 1.5 shares of IBM), that result should always be exact, to the penny. If it's inexact, then something is wrong – perhaps someone mistakenly entered a fractional share quantity, or there's a bug in the program. For these sorts of situations, it is very useful to know whether a monetary quantity is exact or inexact (e.g., has been rounded). The way monetary quantities are represented should support this.
Finally, sometimes in monetary calculations we really don't want inexact values to be rounded. For example, suppose we had 10,000 USD that we want to divide in thirds and distribute to three accounts. The rounded, inexact result of that would be 3,333.33 USD that we put into each account – but that only adds up to 9,999.99 USD – we “lost” a penny and now our books are out of whack. The way financial applications generally solve problems of this type is to use some algorithm to choose a lucky account, and they give that account the extra penny (in this case). These algorithms themselves are interesting, as they need to be repeatable (so you can't just roll the dice) for audits, but they are not the problem I'm discussing here. It's knowing that we have an inexact result, and how much is “left over” to distribute that I care about today.
This sort of problem always has a division operation at the root of it. The general solution is really simple: you need a division operation that gives you the floor of the quotient, and the remainder. So the result of the division in the example above would be 3,333.33 with a remainder of 0.01. The way monetary quantities are represented must include the division-with-remainder operation. Ideally it would allow returning quotients that are floor, ceiling, nearest toward zero, or nearest away from zero (all with appropriately adjusted remainders) because all of these are useful in some financial applications.
Paradise ponders, filling station, fuzzy eyeballs, and giant wrench edition...
Paradise ponders, filling station, fuzzy eyeballs, and giant wrench edition... Yesterday was another really busy day around the place. I got dragged away from my own projects several times to help the guys working on our “filling station”. There were decisions to make, some education on how the components worked, tools to lend, and hardware to scrounge for.
In the first photo below, Joe is using a fancy core drill (with a water-cooled diamond bit!) to drill holes through the back wall of the filling station. These holes allow the two pieces of 1" black iron pipe through the wall. I'll be grouting them later to make them water-tight. One of the pipes is for gasoline, the other for diesel. The second photo shows the inside of the filling station, with most of the plumbing done. The fuel comes in through the pipes on the bottom, then on each leg (for each fuel) goes through a shutoff valve, a filter/water separator, and a gauge. The pipes on top will be extended toward the opening, where a swiveled hose with a nozzle will be attached. The other end of the pipes is currently just open underneath the tanks; there's a bit of plumbing left to get done there, too. But look at all that lovely progress! I think that by next week I should be calling the local petroleum distributor to order me up some fuel. Woo hoo!
Yesterday Debbie and I took a jaunt up to our optometrist to pick up her reworked new glasses. The first time we got them she was getting headaches when wearing them. When we took them back, we discovered that the optometrist had made a prescription mistake. He wrote it as 53° axis (for astigmatism), but it was supposed to be 153°! They immediately ordered new lenses for her, and yesterday we got them. Her first reaction: she couldn't see well at distance. Now she's got to wear them for a week to see if her eyeballs will relax so she can see well with them. If not, it's back to the optometrist we go – for round three!
Mark T. was here yesterday to finish up putting the risers. There was a bit of a holdup, though, as he was trying to remove our old aluminum riser heads from the PVC pipe that holds them. The pipe is thick-walled 3" ID PVC, male threaded, and the riser screws onto the pipe. Well, these old riser heads had been in place for 20 years or so, and they weren't going to come off easily. Mark only had one pipe wrench big enough to grip them, and that one just barely. So I grabbed my giant water pump wrench, which has jaw that open to 5" (commercial photo at right). I don't often need something that big, but when I do it's really hard to beat this thing. It's got gripping power every bit as good as a conventional pipe wrench, but it's much easier to use. Mark and I got those things apart in no time. :)
In the first photo below, Joe is using a fancy core drill (with a water-cooled diamond bit!) to drill holes through the back wall of the filling station. These holes allow the two pieces of 1" black iron pipe through the wall. I'll be grouting them later to make them water-tight. One of the pipes is for gasoline, the other for diesel. The second photo shows the inside of the filling station, with most of the plumbing done. The fuel comes in through the pipes on the bottom, then on each leg (for each fuel) goes through a shutoff valve, a filter/water separator, and a gauge. The pipes on top will be extended toward the opening, where a swiveled hose with a nozzle will be attached. The other end of the pipes is currently just open underneath the tanks; there's a bit of plumbing left to get done there, too. But look at all that lovely progress! I think that by next week I should be calling the local petroleum distributor to order me up some fuel. Woo hoo!
Yesterday Debbie and I took a jaunt up to our optometrist to pick up her reworked new glasses. The first time we got them she was getting headaches when wearing them. When we took them back, we discovered that the optometrist had made a prescription mistake. He wrote it as 53° axis (for astigmatism), but it was supposed to be 153°! They immediately ordered new lenses for her, and yesterday we got them. Her first reaction: she couldn't see well at distance. Now she's got to wear them for a week to see if her eyeballs will relax so she can see well with them. If not, it's back to the optometrist we go – for round three!
Monday, April 17, 2017
Things that go wrong with monetary calculations...
Things that go wrong with monetary calculations... As a programmer, it's very easy to forget about the error-producing “edge cases” in calculations. These are all things that can occur in any calculations, including monetary calculations. For a robust financial application, all of these must be handled in some way. The big three:
- Division by zero. This is probably the classic “gotcha” in financial calculations, probably because there are so many ways for it to happen that programmers somehow fail to foresee. The standard floating point implementations will return a value of infinity for this operation (either positive or negative). Some libraries will throw an exception. Either method works.
- Invalid operation. The canonical examples are zero divided by zero, or the square root of a negative number. There are others, too. I've had this error crop up in several implementations of bond and stock options models I made. The standard floating point implementations will return a value of Not A Number (abbreviated as NaN) for these operations. Other libraries will throw an exception. Again, either method works.
- Inexact result. An inexact result occurs when the result of any calculation cannot be represented exactly because the number representation cannot hold either the number of significant digits or the exponent size, or both. In either binary or decimal, the result of 1/3 is one example of such a result. Standard floating point implementations silently round such results, providing no indication that the result is inexact.
Paradise ponders, walks and drives edition...
Paradise ponders, walks and drives edition... Yesterday afternoon I took Mako (our giant field spaniel puppy) for a walk up my usual route, 1.5 miles round trip. It was an absolutely gorgeous day, as you can see in the photos below. Spring is springing, and green is popping out all over. I tied Mako to a fence post to take these photos, and as you can see in the first photo he had himself all wrapped up within seconds!
Mako had an adventure on this walk that neither of us expected. He behaves a lot like Mo'i used to, snuffling along through the grasses in search of something he can eat (a vole or mouse, perhaps). Yesterday he came across a hole about 8" in diameter, hidden in the grass. Before I knew what was happening, he had his head in it right down to his shoulders. A second or so later he came flying backwards out of the hole, followed quickly by a spittin' mad groundhog. That old ground hog charged Mako fearlessly, and looked damned effective with his teeth and claws. Mako tumbled backwards and out of range, then promptly pooped. :) When we continued our walk, he gave that groundhog hole very wide berth!
I saw two other interesting animals on our walk: a white-tailed kite and a very grey red fox. Those kites are beautiful birds that were quite common where we used to live in California. This is the first one I've seen here. It hovered for five minutes or so in a few locations nearly straight overhead, so I had some excellent viewing. The fox we've seen before; apparently these fields are well within its territory. Cover is scarce right now, with the alfalfa just barely emerging, so the chances of spotting the fox now are much higher. Mako never saw either animal. :)
When I got back from the walk, Debbie and I took a drive out toward Hardware Ranch, and then a few miles up Ant Flats Road. We took the Tesla, and the bumpy Ant Flats Road was a challenge for her because of the pain in her knee's incision. We probably won't do that again for a few weeks, until she's feeling better. But ... we did see some animals, especially birds: a golden eagle, Sandhill cranes, blue herons, a pheasant, and lots of deer. The right-hand fork of Blacksmith Fork River was running at around 8x normal volume, so the waterfalls along the way were really pretty (photos below).
The first photo is of a man-made water feature in the front yard of the cabin on Miller's Ranch. They get to look at this out their windows. The other photos are two angles of the same natural falls, just a mile or so from Hardware Ranch along Ant Flats Road. We've been by this dozens of times, so we're very familiar with its normal flow – a small fraction of what you see here.
Mako had an adventure on this walk that neither of us expected. He behaves a lot like Mo'i used to, snuffling along through the grasses in search of something he can eat (a vole or mouse, perhaps). Yesterday he came across a hole about 8" in diameter, hidden in the grass. Before I knew what was happening, he had his head in it right down to his shoulders. A second or so later he came flying backwards out of the hole, followed quickly by a spittin' mad groundhog. That old ground hog charged Mako fearlessly, and looked damned effective with his teeth and claws. Mako tumbled backwards and out of range, then promptly pooped. :) When we continued our walk, he gave that groundhog hole very wide berth!
I saw two other interesting animals on our walk: a white-tailed kite and a very grey red fox. Those kites are beautiful birds that were quite common where we used to live in California. This is the first one I've seen here. It hovered for five minutes or so in a few locations nearly straight overhead, so I had some excellent viewing. The fox we've seen before; apparently these fields are well within its territory. Cover is scarce right now, with the alfalfa just barely emerging, so the chances of spotting the fox now are much higher. Mako never saw either animal. :)
When I got back from the walk, Debbie and I took a drive out toward Hardware Ranch, and then a few miles up Ant Flats Road. We took the Tesla, and the bumpy Ant Flats Road was a challenge for her because of the pain in her knee's incision. We probably won't do that again for a few weeks, until she's feeling better. But ... we did see some animals, especially birds: a golden eagle, Sandhill cranes, blue herons, a pheasant, and lots of deer. The right-hand fork of Blacksmith Fork River was running at around 8x normal volume, so the waterfalls along the way were really pretty (photos below).
The first photo is of a man-made water feature in the front yard of the cabin on Miller's Ranch. They get to look at this out their windows. The other photos are two angles of the same natural falls, just a mile or so from Hardware Ranch along Ant Flats Road. We've been by this dozens of times, so we're very familiar with its normal flow – a small fraction of what you see here.
Sunday, April 16, 2017
Chicken pot pie...
These deviations from the recipe had another result as well: we ended up with twice as much filling as we were supposed to. I ended up vacuum-bagging (for freezing) half the filling, and baking the rest with the puff pastry top. When it was done, we tucked into it with enthusiasm and managed to put away about 1/3 of that pan. Debbie actually had more than I did! I took about half the rest and put it in a refrigerator container. The remainder is now in two more vacuum-bags, waiting to cool down before I vacuum them and toss them in the freezer. We have much (yummy!) chicken pot pie in our future!
While I was cooking this, I had to add quite a bit of broth and cream to get enough liquid (because we added so much good stuff). The sauce is thickened with a roux, and I hadn't changed that from the recipe. That meant the sauce was way too thin, so I whipped up some more butter-and-flour roux in a little frying pan. That worked great – when I threw that roux in, the sauce thickened right up. I like the flavor and texture of a roux way better than cornstarch, so I was glad I did it that way...
How about scaled integers for monetary amounts?
How about scaled integers for monetary amounts? A friend recently wondered why I wouldn't simply use scaled integers for monetary amounts. For instance, if I determined that all I needed was 10 significant integer digits, plus 4 decimal places, then I could exactly represent any decimal value within that range by multiplying it times 10,000 and using the resulting integer. For example, I could represent 382.03 as 3,820,300. When it came time to present that number to a human, I'd just divide by 10,000.
Scaled integers work particularly well for addition and subtraction, and for many financial applications that's the bulk of what they do. Consider this addition example, unscaled on the left and scaled by 10,000 on the right:
Multiplies aren't quite as lovely, though. The scale factor gets multiplied along with the actual number, so you get a result that has to be divided by the scale factor to get a correctly scaled result. Example:
And then there's division, where the scale factor essentially is canceled out – requiring you to multiply the result by the scale factor to re-scale it.
If these numbers at our desired precision all fit into a native integer type, this would be a bit unwieldy, a little less performant than native, but workable. In an earlier post I figured that we needed a range that encompassed at least 30 decimal digits just to represent amounts of money. The binary equivalent of 30 decimal digits is about 100 bits. The largest native integer in Java (the long) has 63 significant bits – not even close.
Well, what if we used two longs? That would give us 126 significant bits – plenty of room. Addition and subtraction are still simple with this scheme. Multiplication is a bit harder, but still workable. Division is a bear, though, and substantially slower than a native implementation. Those aren't necessarily deal-killers, just a consideration. A similar issue arises from the fact that with this scheme it takes 16 bytes to store any number. That's expensive in database, mass storage, network transmission, and CPU cache (for any application that uses lots of values, i.e. most of them). But still not necessarily a deal killer.
But, as my mother-in-law would say, there's a worser problem with scaled integers. It derives from the fact that you don't just represent monetary values in an application – you also do math with them. I've used the simple example of multiplying price times quantity to get extended price, but many financial applications do much more than such simple math.
Just to pick one example out of my checkered past: I once was charged with building applications that modeled the performance of complex bonds (that is, those with fancy terms in them, not just simple interest), over wide ranges of multiple environmental variables (LIBOR rate, inflation rate, etc.) in combination with each other. These models had multi-dimensional tables with millions of entries, each of which contained a calculated probable value. In some of the models I built, these values could be as small as 10^-10, with around 8 significant digits. That's not something exotic and unusual, either – it's a perfectly normal sort of financial application.
Here's the real point, though: financial applications need to do math with money, and we really can't predict what the range of numbers they'll need will be. This point has been driven home for me by a number of bad experiences out in that pesky real world. Every application I've ever worked on that used fixed point numeric representation (which scaled integers are an example of) has run into problems with the range of numbers they could represent. The failure modes can be very bad, too – especially if the fixed point implementations aren't good at catching overflows (and many of them don't even try, because of the performance penalty).
This hard stop on the range of numeric values held is the real deal-killer for me with fixed point representations. The performance and size issues just make it a little bit worse. In my opinion, fixed point representation and manipulation of monetary values is a dangerous source of fragility in financial applications. Further, it's one that is very difficult to repair – once the decision to use a particular fixed point representation is made, that decision creates tendrils of dependency that find their way into every nook and cranny of the application.
So how do you avoid this? There's a good solution, but it comes with its own costs: decimal floating point. That will be the subject of a few more posts...
Scaled integers work particularly well for addition and subtraction, and for many financial applications that's the bulk of what they do. Consider this addition example, unscaled on the left and scaled by 10,000 on the right:
773.32 7733200
27.99 279900
------ ------- 801.31 8013100
Multiplies aren't quite as lovely, though. The scale factor gets multiplied along with the actual number, so you get a result that has to be divided by the scale factor to get a correctly scaled result. Example:
4.45 44500
6.02 60200
------ ---------- 26.789 2678900000 rescaled to 267890 And then there's division, where the scale factor essentially is canceled out – requiring you to multiply the result by the scale factor to re-scale it.
773.32 7733200
27.99 279900
------ ------- 27.63 27.63 rescaled to 276300 (results are rounded)
If these numbers at our desired precision all fit into a native integer type, this would be a bit unwieldy, a little less performant than native, but workable. In an earlier post I figured that we needed a range that encompassed at least 30 decimal digits just to represent amounts of money. The binary equivalent of 30 decimal digits is about 100 bits. The largest native integer in Java (the long) has 63 significant bits – not even close.
Well, what if we used two longs? That would give us 126 significant bits – plenty of room. Addition and subtraction are still simple with this scheme. Multiplication is a bit harder, but still workable. Division is a bear, though, and substantially slower than a native implementation. Those aren't necessarily deal-killers, just a consideration. A similar issue arises from the fact that with this scheme it takes 16 bytes to store any number. That's expensive in database, mass storage, network transmission, and CPU cache (for any application that uses lots of values, i.e. most of them). But still not necessarily a deal killer.
But, as my mother-in-law would say, there's a worser problem with scaled integers. It derives from the fact that you don't just represent monetary values in an application – you also do math with them. I've used the simple example of multiplying price times quantity to get extended price, but many financial applications do much more than such simple math.
Just to pick one example out of my checkered past: I once was charged with building applications that modeled the performance of complex bonds (that is, those with fancy terms in them, not just simple interest), over wide ranges of multiple environmental variables (LIBOR rate, inflation rate, etc.) in combination with each other. These models had multi-dimensional tables with millions of entries, each of which contained a calculated probable value. In some of the models I built, these values could be as small as 10^-10, with around 8 significant digits. That's not something exotic and unusual, either – it's a perfectly normal sort of financial application.
Here's the real point, though: financial applications need to do math with money, and we really can't predict what the range of numbers they'll need will be. This point has been driven home for me by a number of bad experiences out in that pesky real world. Every application I've ever worked on that used fixed point numeric representation (which scaled integers are an example of) has run into problems with the range of numbers they could represent. The failure modes can be very bad, too – especially if the fixed point implementations aren't good at catching overflows (and many of them don't even try, because of the performance penalty).
This hard stop on the range of numeric values held is the real deal-killer for me with fixed point representations. The performance and size issues just make it a little bit worse. In my opinion, fixed point representation and manipulation of monetary values is a dangerous source of fragility in financial applications. Further, it's one that is very difficult to repair – once the decision to use a particular fixed point representation is made, that decision creates tendrils of dependency that find their way into every nook and cranny of the application.
So how do you avoid this? There's a good solution, but it comes with its own costs: decimal floating point. That will be the subject of a few more posts...
Paradise ponders, recoveries, flowers, stairs, and risers edition...
Paradise ponders, recoveries, flowers, stairs, and risers edition... Debbie had a milestone day yesterday for her post-surgery recovery. Her bandages came off in the morning (she couldn't do it – got queasy – so I did it for her :). Then she took a shower; oh, so good. Then she got down the stairs to our basement cattery to see her babies. Then we had a steak dinner. Then we went up to Aggie's Creamery and got ice cream cones. A great day for her! But she was tired for some reason after all that. :)
We've got a few flowers in our yard. The daffodils have started to bloom, and then the ground cover (at right) we have in several places is also out. After all the destruction of our yard last year (and continuing this year), I'm amazed anything at all has survived. The sweet peas are starting to come up, too – probably another two or three weeks and they'll be a big burst of color to the west of our house...
Yesterday I completed the installation of the stairs I built into our sun room (photos below). All my careful measuring paid off: the stairs fit perfectly on the first try. The landing is level with our bedroom floor, as intended, and the little “lip” I machined out fit over the door sill exactly as intended. I'm very pleased with the way the finish blends with our tile floor, too. Next step: some more careful measurements for the rail (it will be on the left side). Once I make those, I'll send them off to the folks at Lazy K Wrought Iron to get a rail made. At that point our sun room will (finally!) be complete. We'll make another road trip up there to get them!
Mark T., Dave, and Dave's three sons were here all day yesterday installing 6" diameter irrigation pipe to move the line of risers that used to be in the middle of our back yard to just outside the fence. The new line of risers (visible if you embiggen the photo at right) is a couple of feet onto the property of our friend and neighbor Tim D. Once my sprinklers are installed (that's the next thing Mark T. will be working on), I won't need them at all – but Tim will, to irrigate the 2.5 acre field to the left (north) in the photo. It's a big chunk of work to move all that, and Mark and his helpers did a really nice job of it. The only bit they have left is the easy part: gluing the risers in place onto those vertical 3" pipes. Then it's on to the sprinklers in our yard!
Yesterday I completed the installation of the stairs I built into our sun room (photos below). All my careful measuring paid off: the stairs fit perfectly on the first try. The landing is level with our bedroom floor, as intended, and the little “lip” I machined out fit over the door sill exactly as intended. I'm very pleased with the way the finish blends with our tile floor, too. Next step: some more careful measurements for the rail (it will be on the left side). Once I make those, I'll send them off to the folks at Lazy K Wrought Iron to get a rail made. At that point our sun room will (finally!) be complete. We'll make another road trip up there to get them!
Saturday, April 15, 2017
A slightly subtle issue with binary representations of money...
A slightly subtle issue with binary representations of money... This is an issue that I ran into a few times on a stock trading application that initially used double precision floating point to represent monetary values.
Consider the fractional value (in base ten) 11/16. In exact radix form, that's 0.6875. Rounded to one decimal digit, that would be 0.7; to two digits 0.69, and to three digits either 0.687 or 0.688 (depending on the rounding rules you choose). That's the way we usually think of numbers; hopefully you find none of that surprising.
But now look at what happens in binary representations of the same number. The fractional form is 1011/10000 and the radix form is 0.1011. If you round that form to three bits (roughly equivalent to one decimal digit), you get 0.110 (or possibly 0.101 with different rounding rules). When you convert that value to decimal for presentation to one of those pesky humans, you get ... 0.75 (or possibly 0.625).
Oops.
There's nothing actually wrong with the way binary is rounding, it's just (very) unexpected to the average person looking at a rounded monetary value. The “expected” value of 0.7 for single digit rounding is (in binary) 0.10110011001100... There's simply no way to round in binary and get a result like that!
This rounding problem crops up most often when doing division operations. In those stock trading applications, we kept getting it in two places: stock quotes in fractional values (still common in some exchanges, though thankfully not in the U.S.), and when calculating average price for a series of related trades. The latter problem caused us much grief when the system on the other end represented money using decimal numbers – our average price calculation would give a different result than theirs, we'd get a mismatch, and a human would have to intervene to figure it all out. That human was not a programmer, so from their point of view our binary rounding was simply wrong. We never really fixed this problem until we switched to decimal representation...
Consider the fractional value (in base ten) 11/16. In exact radix form, that's 0.6875. Rounded to one decimal digit, that would be 0.7; to two digits 0.69, and to three digits either 0.687 or 0.688 (depending on the rounding rules you choose). That's the way we usually think of numbers; hopefully you find none of that surprising.
But now look at what happens in binary representations of the same number. The fractional form is 1011/10000 and the radix form is 0.1011. If you round that form to three bits (roughly equivalent to one decimal digit), you get 0.110 (or possibly 0.101 with different rounding rules). When you convert that value to decimal for presentation to one of those pesky humans, you get ... 0.75 (or possibly 0.625).
Oops.
There's nothing actually wrong with the way binary is rounding, it's just (very) unexpected to the average person looking at a rounded monetary value. The “expected” value of 0.7 for single digit rounding is (in binary) 0.10110011001100... There's simply no way to round in binary and get a result like that!
This rounding problem crops up most often when doing division operations. In those stock trading applications, we kept getting it in two places: stock quotes in fractional values (still common in some exchanges, though thankfully not in the U.S.), and when calculating average price for a series of related trades. The latter problem caused us much grief when the system on the other end represented money using decimal numbers – our average price calculation would give a different result than theirs, we'd get a mismatch, and a human would have to intervene to figure it all out. That human was not a programmer, so from their point of view our binary rounding was simply wrong. We never really fixed this problem until we switched to decimal representation...
Paradise ponders, less controlled chaos edition...
Paradise ponders, less controlled chaos edition... Well, yesterday our furnace guys were here and worked all day long on fixing our furnace issues. As always, there were about a bazillion little decisions that needed to be made (by me!) along the way. On the other hand, they got nearly everything done. The furnace now sits almost two feet higher off the ground than it used to, and underneath it is a ginormous filter. The goal there was to reduce the resistance of air entering the furnace. On that same note, they added two new return grills and enlarged two others, approximately tripling the square inches of return grills in our house. The difference is very obvious down in our basement “mechanical room” (where the furnace is located): the return air duct is rattling and flexing in response to the much higher volume of air now traveling through it. The guys also installed a large new vent (where the hot air comes out) in the cattery, and the warm air floods out of there in a way that delights the inhabitants.
That's the good furnace news. Here's the not-so-good: all of this work was done to cure a problem our furnace had. It cycled on-and-off every few minutes, as the plenum above the burners overheated. The diagnosis for this problem was insufficient air flow through the furnace, which made good sense to me. Also, we eliminated all the other causes any of us could think of. I'm sure you've guessed by now that the furnace is still cycling. Sigh. They'll attack it again on Monday. I don't really feel badly about this, though, as all of these changes are going to improve the heating in the house anyway. But sigh nonetheless...
The heater guys were here until after 6 pm, so after they left I started my evening chores and was contemplating going to bed. But just then the lawn sprinkler guys showed up, with a fifth-wheel trailer full of 6" irrigation pipe. And they went right to work to install it. I had been expecting them all day, but apparently there was something wrong with their trailer that took nearly all day to fix. They're supposed to be here again today to finish installing the new risers. That will make my friend and neighbor Tim D. a happy guy...
That's the good furnace news. Here's the not-so-good: all of this work was done to cure a problem our furnace had. It cycled on-and-off every few minutes, as the plenum above the burners overheated. The diagnosis for this problem was insufficient air flow through the furnace, which made good sense to me. Also, we eliminated all the other causes any of us could think of. I'm sure you've guessed by now that the furnace is still cycling. Sigh. They'll attack it again on Monday. I don't really feel badly about this, though, as all of these changes are going to improve the heating in the house anyway. But sigh nonetheless...
The heater guys were here until after 6 pm, so after they left I started my evening chores and was contemplating going to bed. But just then the lawn sprinkler guys showed up, with a fifth-wheel trailer full of 6" irrigation pipe. And they went right to work to install it. I had been expecting them all day, but apparently there was something wrong with their trailer that took nearly all day to fix. They're supposed to be here again today to finish installing the new risers. That will make my friend and neighbor Tim D. a happy guy...
Friday, April 14, 2017
Monetary values in computer programs...
Monetary values in computer programs... Many programmers who first encounter the problem of representing a monetary value in a program don't immediately see much of a challenge. At first blush, it seem simple to represent a monetary value: all you need is a number plus a way to identify the currency. Right?
Wrong.
The currency identification really is easy. There's a widely accepted standard (ISO 4217) that specifies a three-character code for every currency in the world. It's used in every multi-currency application I've seen in the past 15 years or so. For instance, United States dollars have the code “USD”. But the amount of that currency isn't just a simple number.
The first little complication is that monetary amounts aren't always integers. You might have $5 (a nice integer value), but you might also have $5.27. Different currencies around the world conventionally have anything from 0 to 4 decimal places for a “usual” amount of money – but they will occasionally have even more. For instance, the price in USD for a gram of gold is often specified with three or even four decimal places, like $40.625. Other commodities or financial instruments may have even more decimal places. At a job I had in the early 2000s I ran into a case where prices in USD were specified to seven decimal places (that's 1/100,000th of a cent!). The takeaway here is that the number of decimal places needed isn't infinite, but it's also not as small as you might think. I'd feel pretty safe with 8 decimal places for USD, or 10 for any currency...
The next complication is that monetary amounts can be quite large. Corporations these days deal with amounts as large as 100s of billions USD, or 12 digits of integer value. For some currencies you might need to add 4 more digits. For government systems, you'll need even more – 100s of trillions USD today. That's 15 digits of integer value, or up to 19 in other currencies.
For presentation to people, most applications will round huge values to the nearest thousand, million, or billion. I haven't seen rounding to the nearest trillion USD, but I won't be surprised if it happens. :) However, internally every financial application I've ever seen keeps track of every last digit. For USD, that means you'd need a total of 17 significant digits to represent government-scale values. Knowing that government will only get ever more bloated, adding a “pad” of 3 or 4 digits is probably a minimum requirement – so at least 20 or 21 significant digits, maybe a few more if you want to sleep better at night.
An aside here: I once worked on a stock trading application that was USD only, and had 10 significant digits (including cents). That meant the biggest value it could represent was $99,999,999.99. Surely that should have been safe for stock trades, right? Well, it wasn't, as they found out in a spectacular fashion one day. A trader placed a buy order for a blue-chip stock where the total value of the order was about $400 million. A very nice order, indeed! But a price that big couldn't be represented in the system. Worse, the program didn't detect the overflow, and instead returned a nonsense value for the calculation (number of shares times price per share) – and that nonsense value was only a few thousand dollars! The order was transmitted to the market correctly, and actually was executed – but the accounting for it was totally messed up. It took several engineers several days to analyze the logs for each of the thousands of small trades that made up the big order, so that we could get the customer's books in order. The company ended up making that trade for free to get back in the good graces of its customer. The CEO told us to fix that problem, and pronto. Of course he thought the fix would be trivial – just change a couple lines of code and it would all be fixed. It was actually very far from trivial to fix that problem after the fact. The assumption about number of significant digits turns out to have been subtly and broadly spread throughout the entire application. We were fixing related bugs a year later...
Another complication is specific to representing fractional values. If you've digested the challenges above, you may be saying to yourself “Floating point! Use floating point!” For any mainstream programming language I'm aware of, that means floating point compliant with IEEE-754. More specifically, it means the binary32 (“single precision”) or binary64 (“double precision”) variants of IEEE-754, which are the ones implemented in most hardware floating point units and in most programming language libraries. As the variant names imply, they are implemented in binary. The bits to the right of the decimal point have values of 1/2, 1/4, 1/8, and so on (instead of the 1/10, 1/100, 1/1000, and so on we're used to with decimal representations). There is a consequence to this choice of binary vs. decimal base that many programmers are not aware of, and it's a consequence of special import when representing money. It's worth taking some time to understand.
First a brief refresher. In any numeric base, a fraction in radix form is equivalent to a fractional form. For instance, in decimal 0.446 is equivalent to 446/1000. Similarly, in binary 0.11011 is equivalent to 11011/100000. Here's the part that may come as a surprise to you: in any given base, fractional values can only be represented precisely in radix form if the denominator of the fraction is a power of the base or of factors of the base. That's a mouthful, so here are some examples to make it clearer:
To illustrate how much of a problem this is, consider all the numbers between 0.00 and 0.01 in increments of 0.01 (base 10). Here's a complete list of those that can be exactly represented in binary floating point: 0.00, 0.25, 0.50, 0.75. Yup, that's it – just four of the one hundred possible numbers. All the rest are approximations. And approximations ain't too good for representing monetary values. If I have $5.42, I want $5.42, not $5.419999997!
By the way, I've been assuming in here that monetary values are all expressed in base 10. That's not actually strictly true. I know from my days working on foreign exchange software that there is at least one currency out there that is not. Fortunately, though, it's a minor currency (the Mauritanian ouguiya) and
even that one is base 5, so its fractional values can be represented exactly with base 10 fractions.
If you made it this far, perhaps you'll accept that a general-purpose representation of money needs these attributes:
I was the CTO for a company building a stock trading application where someone prior to me had made the decision to use BigDecimal to represent monetary values. We saw lots of issues resulting from this choice, but two of them recurred enough to call them a problem pattern.
One was the performance issue I mentioned earlier. There were certain places in the application where we did a fair amount of arithmetic. None of it was fancy or difficult, but even operations like comparing two values, if done enough, could occupy the CPU for a significant amount of time. I was able to do an interesting experiment there, as the monetary value representation was nicely isolated in a class. We profiled a production server and discovered that a little over 40% of our CPU consumption was occurring inside BigDecimal. That was by far the single largest load on the processor. The only way we could get more throughput from that server was to get rid of BigDecimal (or rewrite it for higher performance).
The other was a little subtler: the unlimited precision. We were forever finding places where a multiply or divide operation created results with very large precision because the programmer forgot to properly round. Once such a number was created, all the arithmetic operations that depended on it (and there might be millions of them) got bogged down using this large number – and then they often created even more giant precision numbers. Those didn't just create performance problems, either – they consumed so much memory that on several occasions they actually took the server down! These were surprisingly difficult to track down, too, because our application involved multiple inter-networked servers...
This is the first of what will be a series of posts about the challenges of representing and manipulating monetary values in computer programs. Most of this is applicable to programs in any computer language, but some will be specific to Java.
Wrong.
The currency identification really is easy. There's a widely accepted standard (ISO 4217) that specifies a three-character code for every currency in the world. It's used in every multi-currency application I've seen in the past 15 years or so. For instance, United States dollars have the code “USD”. But the amount of that currency isn't just a simple number.
The first little complication is that monetary amounts aren't always integers. You might have $5 (a nice integer value), but you might also have $5.27. Different currencies around the world conventionally have anything from 0 to 4 decimal places for a “usual” amount of money – but they will occasionally have even more. For instance, the price in USD for a gram of gold is often specified with three or even four decimal places, like $40.625. Other commodities or financial instruments may have even more decimal places. At a job I had in the early 2000s I ran into a case where prices in USD were specified to seven decimal places (that's 1/100,000th of a cent!). The takeaway here is that the number of decimal places needed isn't infinite, but it's also not as small as you might think. I'd feel pretty safe with 8 decimal places for USD, or 10 for any currency...
The next complication is that monetary amounts can be quite large. Corporations these days deal with amounts as large as 100s of billions USD, or 12 digits of integer value. For some currencies you might need to add 4 more digits. For government systems, you'll need even more – 100s of trillions USD today. That's 15 digits of integer value, or up to 19 in other currencies.
For presentation to people, most applications will round huge values to the nearest thousand, million, or billion. I haven't seen rounding to the nearest trillion USD, but I won't be surprised if it happens. :) However, internally every financial application I've ever seen keeps track of every last digit. For USD, that means you'd need a total of 17 significant digits to represent government-scale values. Knowing that government will only get ever more bloated, adding a “pad” of 3 or 4 digits is probably a minimum requirement – so at least 20 or 21 significant digits, maybe a few more if you want to sleep better at night.
An aside here: I once worked on a stock trading application that was USD only, and had 10 significant digits (including cents). That meant the biggest value it could represent was $99,999,999.99. Surely that should have been safe for stock trades, right? Well, it wasn't, as they found out in a spectacular fashion one day. A trader placed a buy order for a blue-chip stock where the total value of the order was about $400 million. A very nice order, indeed! But a price that big couldn't be represented in the system. Worse, the program didn't detect the overflow, and instead returned a nonsense value for the calculation (number of shares times price per share) – and that nonsense value was only a few thousand dollars! The order was transmitted to the market correctly, and actually was executed – but the accounting for it was totally messed up. It took several engineers several days to analyze the logs for each of the thousands of small trades that made up the big order, so that we could get the customer's books in order. The company ended up making that trade for free to get back in the good graces of its customer. The CEO told us to fix that problem, and pronto. Of course he thought the fix would be trivial – just change a couple lines of code and it would all be fixed. It was actually very far from trivial to fix that problem after the fact. The assumption about number of significant digits turns out to have been subtly and broadly spread throughout the entire application. We were fixing related bugs a year later...
Another complication is specific to representing fractional values. If you've digested the challenges above, you may be saying to yourself “Floating point! Use floating point!” For any mainstream programming language I'm aware of, that means floating point compliant with IEEE-754. More specifically, it means the binary32 (“single precision”) or binary64 (“double precision”) variants of IEEE-754, which are the ones implemented in most hardware floating point units and in most programming language libraries. As the variant names imply, they are implemented in binary. The bits to the right of the decimal point have values of 1/2, 1/4, 1/8, and so on (instead of the 1/10, 1/100, 1/1000, and so on we're used to with decimal representations). There is a consequence to this choice of binary vs. decimal base that many programmers are not aware of, and it's a consequence of special import when representing money. It's worth taking some time to understand.
First a brief refresher. In any numeric base, a fraction in radix form is equivalent to a fractional form. For instance, in decimal 0.446 is equivalent to 446/1000. Similarly, in binary 0.11011 is equivalent to 11011/100000. Here's the part that may come as a surprise to you: in any given base, fractional values can only be represented precisely in radix form if the denominator of the fraction is a power of the base or of factors of the base. That's a mouthful, so here are some examples to make it clearer:
- 382/625 base 10 is 0.6112 exactly, because 625 is 5^4, and 5 is a factor of 10.
- 101/1000 base 2 is 0.101 exactly, because 1000 is 2^4.
- 1/3 base 10 is 0.333..., because 3 is not 10 or a factor of 10.
- 1/1010 base 2 is 0.0001100110011... because 1010 (10 base 10) is not 2 or a factor of 2.
To illustrate how much of a problem this is, consider all the numbers between 0.00 and 0.01 in increments of 0.01 (base 10). Here's a complete list of those that can be exactly represented in binary floating point: 0.00, 0.25, 0.50, 0.75. Yup, that's it – just four of the one hundred possible numbers. All the rest are approximations. And approximations ain't too good for representing monetary values. If I have $5.42, I want $5.42, not $5.419999997!
By the way, I've been assuming in here that monetary values are all expressed in base 10. That's not actually strictly true. I know from my days working on foreign exchange software that there is at least one currency out there that is not. Fortunately, though, it's a minor currency (the Mauritanian ouguiya) and
even that one is base 5, so its fractional values can be represented exactly with base 10 fractions.
If you made it this far, perhaps you'll accept that a general-purpose representation of money needs these attributes:
- Fractions must be represented in decimal format, not binary.
- At least 20 significant decimal digits including the normal number of fractional digits needed for any given currency, and more if possible.
- Depending on the currency, up to 10 digits to the right of the decimal point.
I was the CTO for a company building a stock trading application where someone prior to me had made the decision to use BigDecimal to represent monetary values. We saw lots of issues resulting from this choice, but two of them recurred enough to call them a problem pattern.
One was the performance issue I mentioned earlier. There were certain places in the application where we did a fair amount of arithmetic. None of it was fancy or difficult, but even operations like comparing two values, if done enough, could occupy the CPU for a significant amount of time. I was able to do an interesting experiment there, as the monetary value representation was nicely isolated in a class. We profiled a production server and discovered that a little over 40% of our CPU consumption was occurring inside BigDecimal. That was by far the single largest load on the processor. The only way we could get more throughput from that server was to get rid of BigDecimal (or rewrite it for higher performance).
The other was a little subtler: the unlimited precision. We were forever finding places where a multiply or divide operation created results with very large precision because the programmer forgot to properly round. Once such a number was created, all the arithmetic operations that depended on it (and there might be millions of them) got bogged down using this large number – and then they often created even more giant precision numbers. Those didn't just create performance problems, either – they consumed so much memory that on several occasions they actually took the server down! These were surprisingly difficult to track down, too, because our application involved multiple inter-networked servers...
This is the first of what will be a series of posts about the challenges of representing and manipulating monetary values in computer programs. Most of this is applicable to programs in any computer language, but some will be specific to Java.
Paradise ponders, barely controlled chaos edition...
Paradise ponders, barely controlled chaos edition... Yesterday was another crazy day around here, mainly because of the contractors. It's just amazing how many little decisions need to be made for what you might think would be simple projects. Another such decision got added to the list this morning: our mud room cabinetry is about two weeks from completion, so it's time to pick the handles. Debbie's got that one. :)
In between all the other things going on, I did manage to get a bit of work done on the sun room stairs project – and hopefully more today. As in, there is at least a small chance that I could actually have them completely installed today. Tomorrow is probably more likely. :) Anyway, yesterday I fabricated the mounts that will tie the stairs into the wall. There are two of these, made from short pieces of redwood 4x4. Each of them will have two toggle bolts to hold them onto the wall. That wall used to be an exterior wall, so it's sheathed with 1/2" OSB, with 1/2" drywall on top of it. Then in the center of these 4x4 sections, there's a hole for a 1/2" stainless steel bolt that will connect to the stair's ribs. The mount wasn't simply 3 holes drilled, though. The holes for the toggle bolts are 3/4" diameter except for the 1" closest to the wall, where it's just 1/4". That lets me get the heads of the toggle bolts closer to the wall, so I can use shorter toggle bolts. Then the hole for the stainless steel bolt is 1/2" all the way through, except for the 1/4" away from the stair ribs. Those are 7/8" diameter, big enough to allow me to partially sink a nut in there, and glue it. Here's a couple of photos of the setup I used to drill all these holes (a total of 12 holes with 4 drill bits).
A limitation of not being religious is that you don't know when the religious holidays are. I went to check the stock market today, and was surprised to see yesterday's graph still up. At first I thought there was something wrong with the Google finance site. Later I was checking my calendar, and discovered that today is Good Friday. I had no idea!
In between all the other things going on, I did manage to get a bit of work done on the sun room stairs project – and hopefully more today. As in, there is at least a small chance that I could actually have them completely installed today. Tomorrow is probably more likely. :) Anyway, yesterday I fabricated the mounts that will tie the stairs into the wall. There are two of these, made from short pieces of redwood 4x4. Each of them will have two toggle bolts to hold them onto the wall. That wall used to be an exterior wall, so it's sheathed with 1/2" OSB, with 1/2" drywall on top of it. Then in the center of these 4x4 sections, there's a hole for a 1/2" stainless steel bolt that will connect to the stair's ribs. The mount wasn't simply 3 holes drilled, though. The holes for the toggle bolts are 3/4" diameter except for the 1" closest to the wall, where it's just 1/4". That lets me get the heads of the toggle bolts closer to the wall, so I can use shorter toggle bolts. Then the hole for the stainless steel bolt is 1/2" all the way through, except for the 1/4" away from the stair ribs. Those are 7/8" diameter, big enough to allow me to partially sink a nut in there, and glue it. Here's a couple of photos of the setup I used to drill all these holes (a total of 12 holes with 4 drill bits).
A limitation of not being religious is that you don't know when the religious holidays are. I went to check the stock market today, and was surprised to see yesterday's graph still up. At first I thought there was something wrong with the Google finance site. Later I was checking my calendar, and discovered that today is Good Friday. I had no idea!
Thursday, April 13, 2017
An old mortgage...
My favorite part of this mortgage is the legal description of the property. I've transcribed the beginning of it below:
All that certain farm and premises, situate lying and being in the township of West Windsor, in the County of Mercer and the State of New Jersey, consisting of one tract of land, bounded and described as follows, to wit: Beginning at a stone standing in the middle of the public road leading from Edinburgh to Trenton and running down the same North seventy seven and three quarters degrees East, twenty two chains and seven links to another stone in the middle of said road thence by land of Theodore Tindall and John Applegate South six degrees East, twenty one chains and forty three links to a stone thence still by the said Applegate South twelve and a half degrees West,...All those references to old-fashioned surveying, in a document less than 100 years old! Chains and links refer to a now-obsolete method surveyors used to measure distances. I can't imagine New Jersey still relies on stones in the middle of the street, but “monuments” demarcating surveying “sections” are still the norm, at least in the western U.S. Often these are concrete blocks, deeply sunk, with a round brass medallion sunk into the top. We have one of these on our property in Utah. The angles specified in quarters of a degree (1/1440th of a complete circle) are reflective of the limitations in precision of the surveyor's equipment of the day. More accurate equipment was available, but was so very expensive that it wasn't used for ordinary surveying.
I know this property, something I didn't expect when I picked up the mortgage. It was fairly close to the farm I grew up on, along what is today called Edinburg Road. I believe it was either on or close to the property that the present-day Mercer County Technical Schools are on. When I was a child, Edward's home was on it. I don't actually remember Edward – perhaps he had already died by then – but I do remember his wife and two of their daughters. I have a memory of being at their house with those daughters, playing with their kids, and especially enjoying their backyard pond (literally – this was dug out of the dirt). This would have been sometime around 1956 or 1957. I don't remember any of my siblings there, and a little oddly I remember my dad picking me up in our family's old '48 Dodge. Normally my mom would have been doing that.
Wednesday, April 12, 2017
Paradise ponders, chaos edition... Man, what a day!
Paradise ponders, chaos edition... Man, what a day! Four workers from Leading Edge (the company we use for our heating and air conditioning, and a few other things) showed up to start work on an accumulated list of projects here. First up was replacing our old water heater. It was on its last legs, and the capacity was less than we needed. It's been yanked, and in doing so they discovered the thing was nearly half full of sediment. Sheesh! Now we have a pair of brand new 50 gallon water heaters, plumbed in parallel. That gives us 100 gallons of nice hot water anytime we want it.
The workers also started on correcting a problem the home's previous owner created: not enough air returns to the furnace. This was causing us all sorts of problems. Basically the size of our furnace dictates that there be at least five returns in the house (returns are the grilles that return cold air to the furnace to be reheated; they're generally located near the floor). Our house only had two, and even those were undersized. Why did we only have two? Turns out the previous owner decided to cover up four returns! Argh! Today the workers installed two new returns; tomorrow they're going to install a third and enlarge the two existing ones. They're also going to install a large surface area filter on the furnace to reduce the resistance to air motion in the system. We're hoping that finally ends our series of heating problems.
After they've finished with the furnace, they're going to plumb our diesel and gasoline tanks to the “filling station” we started last year. We've talked with our mason, and he's started the project of putting rock on that. With any luck at all, in a couple of months we'll have our own little filling station here.
Meanwhile, the sprinkler workers were back at it today. They're still making trenches for the 6" water line and risers they're moving out of our back yard. Once that's done, it's grade correction, then sprinkler installation, followed by sod installation. They're going to be here a while.
Both of these groups of workers needed quite a bit of input from me today. I was constantly being pulled in one direction or another by these guys. All necessary stuff – I'm not complaining – but not at all conducive to getting anything done on my list of things to do!
But wait, there's more!
Debbie (who's doing fantastically well) really wanted some chicken noodle soup. To satisfy her requirements for a low sodium diet, that means we have to make it – and right at the moment, we means me! :) So I ran to the grocery store this morning, then came home and cleaned and chopped celery, onions, carrots, and roast chicken; sauteed the veggies, spiced up some unsalted chicken broth, dumped in the veggies and chicken, tossed in some fresh fettuccine to make some homemade chicken noodle soup. It tasted good to me, and got Debbie's enthusiastic approval. Meanwhile, the water in the house is turned off completely (because they were installing the water heaters), so I couldn't clean anything in the kitchen – not even my hands. I used paper towels and Pine-Sol to wash up with. :) Later in the day, when the water was restored, I managed to clean up the disaster area formerly knows as our kitchen.
But wait, there's still more!
Partway through my chicken soup preparations, I saw Miki (our oldest dog) vomiting. That made it three days in a row that he'd done so. Time to be worried. Debbie made a vet appointment, and right after I finished making the soup I headed to the vet with poor little Miki. We were worried about bloat, something that had happened to Mo'i (another of our field spaniels) nearly ten years ago. The vet quickly eliminated both bloat and a different kind of blockage. Good. But he suspects pancreatitis. He took blood and urine for testing, and we'll know for sure by Friday if that's what it is. The vet thinks it's equally likely that this is just some temporary GI disturbance, and will go away all on its own in a few days. If it is pancreatitis, then Miki will be going on a special low-fat diet for a while, and getting regular doses of anti-nausea and metronidazole (an antibiotic used to treat stomach infections, amongst other things). Other than the vomiting he seems to be just fine, except that he's gained almost ten pounds over the past year (from 41 to 50). There may be a diet in his future. :)
For some reason I'm fairly tired this evening. :)
The workers also started on correcting a problem the home's previous owner created: not enough air returns to the furnace. This was causing us all sorts of problems. Basically the size of our furnace dictates that there be at least five returns in the house (returns are the grilles that return cold air to the furnace to be reheated; they're generally located near the floor). Our house only had two, and even those were undersized. Why did we only have two? Turns out the previous owner decided to cover up four returns! Argh! Today the workers installed two new returns; tomorrow they're going to install a third and enlarge the two existing ones. They're also going to install a large surface area filter on the furnace to reduce the resistance to air motion in the system. We're hoping that finally ends our series of heating problems.
After they've finished with the furnace, they're going to plumb our diesel and gasoline tanks to the “filling station” we started last year. We've talked with our mason, and he's started the project of putting rock on that. With any luck at all, in a couple of months we'll have our own little filling station here.
Meanwhile, the sprinkler workers were back at it today. They're still making trenches for the 6" water line and risers they're moving out of our back yard. Once that's done, it's grade correction, then sprinkler installation, followed by sod installation. They're going to be here a while.
Both of these groups of workers needed quite a bit of input from me today. I was constantly being pulled in one direction or another by these guys. All necessary stuff – I'm not complaining – but not at all conducive to getting anything done on my list of things to do!
But wait, there's more!
Debbie (who's doing fantastically well) really wanted some chicken noodle soup. To satisfy her requirements for a low sodium diet, that means we have to make it – and right at the moment, we means me! :) So I ran to the grocery store this morning, then came home and cleaned and chopped celery, onions, carrots, and roast chicken; sauteed the veggies, spiced up some unsalted chicken broth, dumped in the veggies and chicken, tossed in some fresh fettuccine to make some homemade chicken noodle soup. It tasted good to me, and got Debbie's enthusiastic approval. Meanwhile, the water in the house is turned off completely (because they were installing the water heaters), so I couldn't clean anything in the kitchen – not even my hands. I used paper towels and Pine-Sol to wash up with. :) Later in the day, when the water was restored, I managed to clean up the disaster area formerly knows as our kitchen.
But wait, there's still more!
Partway through my chicken soup preparations, I saw Miki (our oldest dog) vomiting. That made it three days in a row that he'd done so. Time to be worried. Debbie made a vet appointment, and right after I finished making the soup I headed to the vet with poor little Miki. We were worried about bloat, something that had happened to Mo'i (another of our field spaniels) nearly ten years ago. The vet quickly eliminated both bloat and a different kind of blockage. Good. But he suspects pancreatitis. He took blood and urine for testing, and we'll know for sure by Friday if that's what it is. The vet thinks it's equally likely that this is just some temporary GI disturbance, and will go away all on its own in a few days. If it is pancreatitis, then Miki will be going on a special low-fat diet for a while, and getting regular doses of anti-nausea and metronidazole (an antibiotic used to treat stomach infections, amongst other things). Other than the vomiting he seems to be just fine, except that he's gained almost ten pounds over the past year (from 41 to 50). There may be a diet in his future. :)
For some reason I'm fairly tired this evening. :)
Tuesday, April 11, 2017
Paradise ponders, wifely hardware removal, impossible niceness, the return of the Internet, and great big smiles edition...
Paradise ponders, wifely hardware removal, impossible niceness, the return of the Internet, and great big smiles edition... Today Debbie had a planned surgery to remove the hardware that was inserted last year to hold her knee together while it healed. The surgery was a relatively minor one; she spent less than 45 minutes in the operating room. It's hard to imagine how it could possibly have gone any better. The surgeon reported zero complications, he saw that her bones were healing very nicely, and he found exactly what he suspected has been causing Debbie's pain when she exercises: two “protuberances” poking into a tendon, one from the hardware and one from a bony growth. He pulled all the hardware out, filled in some of the screw holes (the ones near her tibia plateau), and filed off the bony growth. She's now all nice and smooth under that troublesome tendon. She's also lighter by the amount of the hardware removed (the photo at left) – more than I was expecting! That fancy plate is quite heavy all by itself. The photo at right is a not-too-wonderful duplicate of the X-rays her surgeon took during the surgery. The upper right one shows the injector he used to squirt cement into the screw holes.
For this surgery we were at Cache Valley Hospital instead of the Intermountain Regional Hospital we usually go to. This was at the surgeon's request; he prefers their operating facilities. We've been there just once before, for an emergency room visit shortly after we moved here. We were impressed by the ER staff at the time, but the insurance we got for the following year didn't have Cache Valley hospital as part of their network, so we didn't go there. Our insurance this year includes both hospitals, so we accepted the surgeon's preference.
If you've been reading this blog for a while, you likely remember how impressed we were with the staff at Intermountain Regional Hospital. Like most other people here, they were friendly and warm – and competent. Well, Cache Valley Hospital, I have to say, managed to top even them. There are so many little ways they made this visit easier; I could be here listing them for hours. I'll just mention two that jumped out at me. In the outpatient surgery intake room, the patient's chair was an overstuffed recliner – extremely comfortable, and very comforting for Debbie. In both the intake area and in the recovery area, there are people whose entire job consists of running around to all the patients there to make sure they're ok. Drinks, snacks, blankets, pain meds, a cheerful conversation – whatever is needed to make the patients less anxious and more comfortable, that person was right there every 60 seconds or so to make sure it happened. This is so incredibly different than our experiences in San Diego that I am having trouble finding the words to describe them. Our entire experience there today was a series of pleasant surprises. Have you ever had a hospital experience like that? I certainly hadn't!
There's one other thing about the hospital that I can't fail to mention. When we scheduled the surgery last week, most of the collection of information was handled over the phone (very convenient, that was!). Shortly after we finished with that, I got a call from someone in accounting in the hospital. They'd noted that the entire cost of this surgery would be less (much less, actually) than our insurance deductible – so the entire cost would be out-of-pocket for us. The hospital made us an offer: if we'd prepay the cost at their fixed rate, they'd give us a 10% discount on their cash rate (lower than the rate they bill insurance companies). Now I'm sure that makes good business sense for them, as they'd get their money right away and without risk. But obviously, since we have the money available, it makes good sense for us, too. I happily took that offer – but I'm delighted that the offer was voluntarily extended. Someone's on the ball there, identifying a nice win/win for both the hospital and its customers.
On the way home from the hospital today, I asked Debbie if she'd like to pick up something to eat. She asked for soup (today is beef soup day!) from Los Primos. We stopped there and Debbie waited in the car while I ran in to pick up that soup. I placed my takeout order with a young man who's served us several times before. He asked why weren't eating in the restaurant, so I told him about Debbie's surgery. That brought a young woman (who has also served us many times) over to join the conversation. When I got to the part about Debbie requesting the soup, there were two of the biggest, brightest smiles I've ever seen – and I swear the room got considerably brighter. The two of them scurried off to the kitchen to make sure I left with the best selection of beef and vegetables in the soup. So sweet!
Yesterday our Comcast technician (a fellow named Tony) came out to our house, exactly as promised (even well within the window they'd said). Tony was bright, well-equipped with test gear, and had his brain fully engaged during his entire time here. He actually listened to what I had to tell him, and upon hearing the symptoms agreed that this couldn't be a cable modem problem. He didn't waste any time getting to a short list of three likely sources of the problem: a bad cable distribution amplifier, a bad connector in the interior wiring, or a bad underground service cable. That list is in his order of likelihood. We tested the amp first, and it was fine (and we had a great signal there, so that also eliminated the third possibility). We located the cable leading from the amp to the office where we have the modem located, and a nifty little tester he had quickly told us there was an intermittent problem in that cable. It turned out to be the connector behind the wall plate in Debbie's office. In five minutes Tony had that replaced. Then he noted that it was a crappy connector (I'm sure installed by our house's previous owner), so he checked all the other connectors in that cable. Two of them were similar crappy connectors, so he replaced them too. Then we logged into my cable modem and checked signal levels: rock solid, and 6db stronger than I'd ever seen. So we did a speed test, and it was about 50% higher for download and 100% higher for upload (compared with previous tests I'd done). The visit was at no cost to me. I can't think of any way Tony could have done a better job. If Comcast manages to keep this up (and assuming the same sort of thing is happening everywhere), they just might manage to turn around their worst-company-in-the-universe reputation!
Now I'm off to go coddle Debbie a bit... :)
For this surgery we were at Cache Valley Hospital instead of the Intermountain Regional Hospital we usually go to. This was at the surgeon's request; he prefers their operating facilities. We've been there just once before, for an emergency room visit shortly after we moved here. We were impressed by the ER staff at the time, but the insurance we got for the following year didn't have Cache Valley hospital as part of their network, so we didn't go there. Our insurance this year includes both hospitals, so we accepted the surgeon's preference.
If you've been reading this blog for a while, you likely remember how impressed we were with the staff at Intermountain Regional Hospital. Like most other people here, they were friendly and warm – and competent. Well, Cache Valley Hospital, I have to say, managed to top even them. There are so many little ways they made this visit easier; I could be here listing them for hours. I'll just mention two that jumped out at me. In the outpatient surgery intake room, the patient's chair was an overstuffed recliner – extremely comfortable, and very comforting for Debbie. In both the intake area and in the recovery area, there are people whose entire job consists of running around to all the patients there to make sure they're ok. Drinks, snacks, blankets, pain meds, a cheerful conversation – whatever is needed to make the patients less anxious and more comfortable, that person was right there every 60 seconds or so to make sure it happened. This is so incredibly different than our experiences in San Diego that I am having trouble finding the words to describe them. Our entire experience there today was a series of pleasant surprises. Have you ever had a hospital experience like that? I certainly hadn't!
There's one other thing about the hospital that I can't fail to mention. When we scheduled the surgery last week, most of the collection of information was handled over the phone (very convenient, that was!). Shortly after we finished with that, I got a call from someone in accounting in the hospital. They'd noted that the entire cost of this surgery would be less (much less, actually) than our insurance deductible – so the entire cost would be out-of-pocket for us. The hospital made us an offer: if we'd prepay the cost at their fixed rate, they'd give us a 10% discount on their cash rate (lower than the rate they bill insurance companies). Now I'm sure that makes good business sense for them, as they'd get their money right away and without risk. But obviously, since we have the money available, it makes good sense for us, too. I happily took that offer – but I'm delighted that the offer was voluntarily extended. Someone's on the ball there, identifying a nice win/win for both the hospital and its customers.
On the way home from the hospital today, I asked Debbie if she'd like to pick up something to eat. She asked for soup (today is beef soup day!) from Los Primos. We stopped there and Debbie waited in the car while I ran in to pick up that soup. I placed my takeout order with a young man who's served us several times before. He asked why weren't eating in the restaurant, so I told him about Debbie's surgery. That brought a young woman (who has also served us many times) over to join the conversation. When I got to the part about Debbie requesting the soup, there were two of the biggest, brightest smiles I've ever seen – and I swear the room got considerably brighter. The two of them scurried off to the kitchen to make sure I left with the best selection of beef and vegetables in the soup. So sweet!
Yesterday our Comcast technician (a fellow named Tony) came out to our house, exactly as promised (even well within the window they'd said). Tony was bright, well-equipped with test gear, and had his brain fully engaged during his entire time here. He actually listened to what I had to tell him, and upon hearing the symptoms agreed that this couldn't be a cable modem problem. He didn't waste any time getting to a short list of three likely sources of the problem: a bad cable distribution amplifier, a bad connector in the interior wiring, or a bad underground service cable. That list is in his order of likelihood. We tested the amp first, and it was fine (and we had a great signal there, so that also eliminated the third possibility). We located the cable leading from the amp to the office where we have the modem located, and a nifty little tester he had quickly told us there was an intermittent problem in that cable. It turned out to be the connector behind the wall plate in Debbie's office. In five minutes Tony had that replaced. Then he noted that it was a crappy connector (I'm sure installed by our house's previous owner), so he checked all the other connectors in that cable. Two of them were similar crappy connectors, so he replaced them too. Then we logged into my cable modem and checked signal levels: rock solid, and 6db stronger than I'd ever seen. So we did a speed test, and it was about 50% higher for download and 100% higher for upload (compared with previous tests I'd done). The visit was at no cost to me. I can't think of any way Tony could have done a better job. If Comcast manages to keep this up (and assuming the same sort of thing is happening everywhere), they just might manage to turn around their worst-company-in-the-universe reputation!
Now I'm off to go coddle Debbie a bit... :)
Monday, April 10, 2017
Paradise ponders, leaping Sandhill cranes, snowy machines, and acrylic finish edition...
We had problems all day yesterday with our cable provider (which is actually just Internet connection for us). It was up and down 35 times according to the logs my server keeps. That's by far the worst experience we've had yet. I called Comcast (our ISP), who is notorious for their bad customer service. My experience was ... not bad, actually. I got through their automated system and to an actual technician in under a minute. The technician actually listened to what I described of the symptoms, tried only sensible things (instead of the rote power cycle, reboot the router, etc.), and quickly arrived at the same conclusion that I did: because the problem is isolated to us, there must be a connectivity problem between the junction where our line branches off, and our house. That's Comcast's responsibility, so they need to get a technician out here to check it out. That technician is scheduled for today at 2 pm – less than 24 hours after I called. Either I got really lucky, or Comcast is actually working on their “worst company in the world” image...
Naturally, this morning our Internet is working perfectly. :)
Yesterday, as planned, I finished putting the fourth coat of acrylic (water-based polyurethane) on the stairs I'm building for our sun room. This morning that last coat was nice and dry, so I flipped the stairs over and put the first coat of acrylic on the bottom. The more I use this modern finish, the more impressed I am with it. It goes on easily with a brush, and the self-leveling is so good that I'm not sure I could tell brushed from sprayed. There's little odor, and even that is not foul at all. It dries quite quickly, so much so that you have to work fairly fast so you don't ruffle some partly-dried surface. You only have to wait two hours (though I waited three just to be sure) between coats; it's dry enough at that point to sand. Cleanup is ridiculously easy: warm water to rinse out the bulk of the paint from the brush, a little soap and warm water to get the rest, and in the end the brush is just like new. Pretty close to ideal!
Sunday, April 9, 2017
Paradise ponders, cold snowy morning and polyurethane edition...
I just got done putting the first coat of polyurethane on the sun room stairs I'm building. I'm going to put four coats on the top, sanding between them. It takes three hours to dry thoroughly, so it will take me all day today to get the top done. Then tomorrow I'll flip it and do the bottom – probably just three coats on the visible sections and two on the hidden. That means most of tomorrow will be needed, too – and then Tuesday I'll install them...
Saturday, April 8, 2017
A real-life variable-length encoding challenge...
A real-life variable-length encoding challenge... I've been poking around the idea of building a “money” class for Java. This would be based on a purpose-made decimal floating point class, with better performance than BigDecimal and with features aimed at use in monetary calculations. That decimal floating point class will need a way to encode the decimal exponent of the number held.
There are several interesting questions when it comes to the necessary range of that exponent. For US dollars, a range of 10^-5 (thousandths of cents) to 10^14 (hundreds of trillions of dollars) would seem to be sufficient for just about any financial purpose. That might not be true for other currencies, though the range of 20 orders of magnitude is probably pretty close to the mark in any currency. For the rest of this discussion, I'll assume 20 orders of magnitude is what we need.
How should we encode that exponent? All the existing floating point representations I'm aware of do that with a biased binary number. For this case, they'd use a 5 bit number (the smallest number of bits that can hold 20 states) with a bias of 5 (meaning that you subtract 5 from the binary number to get the actual exponent). So, for example, the exponent 3 would be represented by the binary number 01000 (or 8 decimal), as 8 - 5 = 3. This scheme is very simple, allows asymmetry between the positive and negative ranges, and uses a fixed-length encoding. If there is a requirement for fixed-length encoding, I don't know any way to beat this method. But ... those 5 bits could hold 32 states, and we only need 20. We could expand the range possible to store (that's the usual solution), but for the case we're interested in those extra states would be wasted. What we'd really like would be a way to use less bits – and if two things are true, that's entirely possible.
First, it must be allowable (and useful!) to use a variable-length encoding. I the case I'm working, that is allowable, and it's also very useful (in particular, when storing large lists of numbers, like a bookkeeping journal). Second, the distribution of the numbers to be encoded must be uneven in a reasonably predictable way. Intuitively it seems clear that monetary amounts in a typical bookkeeping system are distributed unevenly. It's less clear how predictable that distribution is, but again, intuitively, it seems like at least the shape of the distribution curve should be similar for any size organization.
I happen to have a sample bookkeeping journal leftover from a past work engagement: real data, with about 40,000 numeric entries. With just a little work, I came up with the exponent distribution at right (click to embiggen). Three quarters of all the entries are either in the tens of dollars or hundreds of dollars range! The frequencies fall off very quickly above and below those two exponents. That's pretty darned uneven, which means we most definitely have an opportunity for a variable-length encoding.
Those frequencies are expressed in a normalized manner, so they all sum to 1. That means we can use a simple formula to calculate the average encoding length for any encoding we want to test.
The first one I tried has two lengths selected by the leading bit. The exponent would be encoded as either 0xx (for exponents 0..3) or 1xxxx (for exponents -5..-1 and 4..14). The average bit length for this encoding on the data I have would be 0.975 * 3 + 0.025 * 5 = 3.05. That's a pretty good payoff for such an encoding! Note that in no case does the length exceed 5 bits, which is what we'd have with fixed-length encoding. Also note that the first form has 4 states and the second 16 states, for a total of 20 states – exactly what we need. There are no “wasted” states.
The second one I tried has two lengths, with a second (longer) encoding selected by an escape code. The short code would be just two bits long, with 0..2 representing exponents 0..2, and 3 being the escape indicating a second 5 bit code following (so a total of 7 bits). This second code would represent exponents -10..-1 and 3..24 – a range considerably larger than what we actually need. The average bit length for this encoding on the data I have would be 0.915 * 2 + 0.085 * 7 = 2.425. That's less than half the fixed-length encoding of 5 bits. However, this encoding has a longer maximum length (7 bits vs. 5 bits) and a bunch of “wasted” states.
I haven't yet decided what encoding I'll actually use, though I'm pretty sure I'll be using some variable-length scheme. Also, stealing an idea from Gustafson, I'm going to make the parameters (including length and encoding details) configurable, so no matter what I supply as default the end user can still configure his way out of any bind I create. I would like the defaults to be well-chosen, though. There are several things I need to investigate a bit more, especially (a) the differences between different currencies, and (b) the differences between kinds of financial applications and the size of the organization using them. All I really have to go on there is my intuition, derived from several decades of experience. I'd rather have hard data. :) Don't know where I'm going to find that, though!
There are several interesting questions when it comes to the necessary range of that exponent. For US dollars, a range of 10^-5 (thousandths of cents) to 10^14 (hundreds of trillions of dollars) would seem to be sufficient for just about any financial purpose. That might not be true for other currencies, though the range of 20 orders of magnitude is probably pretty close to the mark in any currency. For the rest of this discussion, I'll assume 20 orders of magnitude is what we need.
How should we encode that exponent? All the existing floating point representations I'm aware of do that with a biased binary number. For this case, they'd use a 5 bit number (the smallest number of bits that can hold 20 states) with a bias of 5 (meaning that you subtract 5 from the binary number to get the actual exponent). So, for example, the exponent 3 would be represented by the binary number 01000 (or 8 decimal), as 8 - 5 = 3. This scheme is very simple, allows asymmetry between the positive and negative ranges, and uses a fixed-length encoding. If there is a requirement for fixed-length encoding, I don't know any way to beat this method. But ... those 5 bits could hold 32 states, and we only need 20. We could expand the range possible to store (that's the usual solution), but for the case we're interested in those extra states would be wasted. What we'd really like would be a way to use less bits – and if two things are true, that's entirely possible.
First, it must be allowable (and useful!) to use a variable-length encoding. I the case I'm working, that is allowable, and it's also very useful (in particular, when storing large lists of numbers, like a bookkeeping journal). Second, the distribution of the numbers to be encoded must be uneven in a reasonably predictable way. Intuitively it seems clear that monetary amounts in a typical bookkeeping system are distributed unevenly. It's less clear how predictable that distribution is, but again, intuitively, it seems like at least the shape of the distribution curve should be similar for any size organization.
Those frequencies are expressed in a normalized manner, so they all sum to 1. That means we can use a simple formula to calculate the average encoding length for any encoding we want to test.
The first one I tried has two lengths selected by the leading bit. The exponent would be encoded as either 0xx (for exponents 0..3) or 1xxxx (for exponents -5..-1 and 4..14). The average bit length for this encoding on the data I have would be 0.975 * 3 + 0.025 * 5 = 3.05. That's a pretty good payoff for such an encoding! Note that in no case does the length exceed 5 bits, which is what we'd have with fixed-length encoding. Also note that the first form has 4 states and the second 16 states, for a total of 20 states – exactly what we need. There are no “wasted” states.
The second one I tried has two lengths, with a second (longer) encoding selected by an escape code. The short code would be just two bits long, with 0..2 representing exponents 0..2, and 3 being the escape indicating a second 5 bit code following (so a total of 7 bits). This second code would represent exponents -10..-1 and 3..24 – a range considerably larger than what we actually need. The average bit length for this encoding on the data I have would be 0.915 * 2 + 0.085 * 7 = 2.425. That's less than half the fixed-length encoding of 5 bits. However, this encoding has a longer maximum length (7 bits vs. 5 bits) and a bunch of “wasted” states.
I haven't yet decided what encoding I'll actually use, though I'm pretty sure I'll be using some variable-length scheme. Also, stealing an idea from Gustafson, I'm going to make the parameters (including length and encoding details) configurable, so no matter what I supply as default the end user can still configure his way out of any bind I create. I would like the defaults to be well-chosen, though. There are several things I need to investigate a bit more, especially (a) the differences between different currencies, and (b) the differences between kinds of financial applications and the size of the organization using them. All I really have to go on there is my intuition, derived from several decades of experience. I'd rather have hard data. :) Don't know where I'm going to find that, though!
Paradise ponders, slippery mud, wet wood, and sushi edition...
Paradise ponders, slippery mud, wet wood, and sushi edition... I was able to spend most of day yesterday (and some this morning, too!) working on my stairs project. In the first photo below, you see the results of the first round of sanding off the black (using 100 grit and the orbital sander). The large landing has one round of sanding done (second photo is a closeup), while the smaller step is unsanded. Quite a difference! I ended up doing two rounds at 100 grit and a final round at 220 grit, and in the end I was pretty happy with how it looked. The black color stayed in the dents as I'd originally intended, but it also was infused in some other areas in a way that resembles how wood weathers. All good! Then I started applying stain: two coats yesterday, another this morning. The result (still wet) is in the third photo. I think I'm going to stop at three, as it looks pretty good. The stain has the side-effect of making the blackened areas a bit indistinct, again more like wood actually weathers. Once these stairs are in the sun room, the little bit of red still visible will gradually fade, and the white parts will yellow slightly – both of these effects will, I think, improve the appearance. So far I've only stained the top; later today I'll flip it over and start the process on the bottom. By tomorrow I believe I'll be applying the matte polyurethane...
We went out for our lunch yesterday to a local favorite: Black Pearl. I had sushi (my seafood nanudo roll at right); Debbie had their house pad thai. Both were excellent! We were both stuffed by the time we rolled out of there, and Debbie had some leftovers, too. Not me. :) On the way there we had an odd thing happen with the Model X. From appearances, the GPS stopped reporting the car's position to the navigation system. The car's symbol was stuck in one place on the map. Nothing I did made any difference at all. After we ate and started home, the position was still stuck – but only for a couple of blocks. Then it suddenly started working again, and has been since then. I've no idea what might have caused that. The Tesla forums have a couple of mentions of the same problem, with the exact same outcome, so I'm not too worried about it.
Last night and early this morning we got slammed with a big rainstorm. The snapshot at right shows the total storm accumulation; we got just over a half inch. That wouldn't be so bad except that our soils were already super-saturated with water – so nearly all the water from this storm is either sitting on the surface or turning the first inch or so of soil into a slurry. Some areas to our west and southwest got over 2" in this storm. These are areas that are normally very dry, so this will likely cause a burst of plant growth – and wildflowers! We have another storm approaching, too – starting around noon today we're expecting rain for 12 hours or so, with another half inch or a bit more in precipitation. All this water is going to make our sprinkler project quite a bit harder. I suspect Mark T. will be waiting a few days for things to dry out at least a little. After today we've got 10 days with little chance of rain in the forecast, with highs in the upper 50s or low 60s, light winds, and lots of sun. A few days of that and we'll dry out a bit, I hope! The farmers around here who didn't plant this week are going to have to wait another week or so before their tractors can make it through the fields...
Subscribe to:
Posts (Atom)