Monday, May 23, 2005

Time

I work for a company in a regulated industry (securities trading). Some of the regulations require certain of the company's computers to keep verifiably accurate time, to within plus or minus three seconds of the official national time reference, kept at the National Institute of Standards and Technology (NIST) Laboratories, in Boulder, Colorado. This is the F1 Cesium Fountain Atomic Clock, shown in the photo at right.

When I first ran into this requirement a few years ago, I thought this would be a simple matter of implementing the Network Time Protocol (NTP). This is a very well-known and widely deployed protocol for the specific purpose of allowing a computer's clock to be precisely synchronized to time standard on the Internet. These "Stratum 1" Internet time servers are themselves synchronized directly to the NIST time standard.

I ran into a showstopper hitch with this approach: the "verifiable" part of the regulatory requirement. The trouble is that NTP works by communicating with the time servers over the Internet, and the time it takes messages to travel back-and-forth over the Internet is far from certain. Here's a mind experiment that illustrates the problem:

Imagine that you want to set your kitchen clock accurately, but the only time reference available is your neighbor's clock, which you know is accurate. Then imagine that the only way you can communicate with your neighbor is by talking. So to set your clock, you organize 12 of your friends to form a line between your kitchen and your neighbor's house, where each person is within shouting distance of the person next to him in line. Now you're ready: your neighbor shouts to the first person in the line when it's exactly twelve noon, and that person relays it to the next one, and so on, until finally you hear the shout from the last of your twelve friends in line — and at exactly that moment you set your kitchen clock to twelve noon.

Now how accurate do you suppose your clock is? It's going to be off by whatever the delay was; perhaps 20 or 30 seconds in this case. But you have an idea: you'll practice this relaying ahead of time, and time the delay, so you know what offset to subtract to set your kitchen clock correctly! Well, that's basically what NTP does. But it doesn't always work correctly. For instance, suppose one of your friends sneezed exactly when the adjacent friend shouted, so that the shout had to be repeated? Oops — guess that one will take a little longer than usual! Also, how consistent do you suppose that shouting is? Not very — you'll find that the time varies considerably from attempt to attempt. No matter how you slice it, if you set your kitchen clock by the method just described, there's no way you can be positively, absolutely certain that your clock is correctly set. In other words, it's not verifiable.

It turns out there's a perfectly conventional way to solve this problem: you buy your own stratum 1 time server, and put it on your own network. This eliminates the uncertainties of the Internet, and leaves you vulnerable only to the vagaries of your own network. In most corporate IT environments (and even in many homes), these "local area" networks (LANs) are implemented in a way (called "fully switched") that essentially eliminates any uncertainties in the message delivery time. In this situation, the computers that need to have their clocks accurately set can synchronize to this local stratum 1 time server, and this will satisfy (easily) the regulatory requirements such as those my company faces.

One problem with this approach: it's very expensive. The cheapest stratum 1 time servers work by listening to cell phone signals (which are synchronized to NIST's national time reference) or by listening to GPS satellites (which have atomic clocks on board that are in turn synchronized to NIST's national time reference). These servers cost about $5,000 to $10,000 each — and a company like mine really needs to have two of them, just in case one happens to break.

At the time my company was facing this problem, we really didn't want to spend that kind of money on this problem. So we looked for alternative solutions, and found that it's possible to purchase "time receivers" that listen to the aforementioned cell phone signals, GPS satellites, or the WWVB radio station maintained by NIST for the specific purpose of transmitting time information. For a number of reasons, including price, we chose a WWVB receiver. We bought two of these, and connected them to two computers running the Linux operating system. Then we modified the source code for NTP (this is reasonably well documented, and it's open source) to match the WWVB receiver, and installed the modified and recompiled NTP. Voila! We had our own stratum one time servers, for a lot less money than the commercial equipment.

Now, being the geek that I am, I just had to have something even better (and cheaper!) at home. It took me a couple of years for technology to catch up with what I wanted, but it finally has. Garmin recently came out with a new device called the GPS 18 LVC. This little darlin' is about the size and shape of a couple of drink coasters piled on top of each other; a little disk with a wire attached to it. It's a complete GPS sensor, but the part I care about (for these purposes) is that it has (a) an NMEA-compatible interface, and (b) a pulse-per-second (PPS) output. "NMEA" is the National Marine Electronics Association, and those folks long ago created a standard protocol ("language") for GPS units. The standard NTP source code distribution contains an NMEA driver, so this compatibility makes it easy to interface the GPS 18 with NTP. The bottom line: the GPS 18 LVC has everything you need to create a spectacularly good stratum 1 NTP server. And it costs about $80!

So I bought a couple of these (to make a redundant time standard). I've designed some simple circuitry (RS-232 line drivers) to allow me to put the GPS receivers up on my roof, and yet still have a stable signal travel down 100 feet or so of wire. I will be implementing NTP with the standard NMEA driver (modified if needed for the GPS 18's needs), and I'll be using the PPS output to implement the PPS driver in NTP. In the end, I'll have a stratum 1 NTP server synchronized to one millisecond or better with the NIST national time reference. The total cost of this project, including the dual GPS 18 LVCs, is less than $250 (using existing Linux-based computers as the NTP servers).

If you're interested in the design, I'd be happy to send you a Visio of the schematic. Just drop me a line! And once I've built it and got it running with NTP, I'd also be happy to share the NTP driver sources and configurations...

Arabs for common sense

From Rantings of a Sandmonkey:

Look, we as arabs/muslims/whatever can't just play the victim, decry that we are opressed and persecuted, and yet be OK with venemous hate like this be spewed and spread throughout our society. We can't chant "Death to America" on one hand and wonder why they won't be more on our side. While the jews were holding out vigils, and stating that " we are all americans today", The Palestinians danced on the streets and handed out sweets and baklawas when 9/11 happend, and then they openly wonderd why the US was on the side of Israel. LOL. If you, dear reader, are one of those people, let me tell you the story of my friend Irina.

My friend Irina - russian jewish US immigrant and current US customs agent - was all about the peace process and giving the palestinians equal rights with the rest of her family when a suicide bomber blew himself up in 2002 and killed her cousin, who was a peace activist. Since that Day her and her whole Labour-voting family turned against the palestinian peace process and became overnight Likud supporters, because- as she told me- they realized that day "that the palestinians don't differentiate between pro- and anti-peace jews; they are all jews to them and therefore they should all die." She used to get really upset and cry whenever she would hear about palestinian houses getting demolished or palestinian bystanders dying or getting injured in crossfire, and now, when she hears about stuff like that, she says that the one thing that crosses her mind is "better them then one of my family".

Read the whole thing.

Good news from Iraq

Mr. Chrenkoff does it again, in part 28 of his excellent "Good News from Iraq series. A teaser:

American airmen are also currently training the reactivated Iraqi air force:

Pilots and enlisted Airmen from the 777th Expeditionary Airlift Squadron at Little Rock Air Force Base, Ark., are here working hand-in-hand with pilots and crewmembers who served in Saddam Hussein’s air force before Operation Iraqi Freedom.

By the time the program is over, which we estimate to be May 2006, they will be a fully functional squadron,” said Maj. Roger Redwood, operations flight commander for the 23rd Advisory Support Team that is part of the 777th EAS. They are training airmen at Iraq’s 23rd Squadron.

While many of the new recruits have ample flight experience, their exposure to English has been more limited, officials said.

Most of the officers can communicate pretty well, but they have a hard time understanding the radio calls from air traffic control,” Major Redwood said.

They know the airplanes. They know the systems. They can do it all in Arabic, but we require them to do it in English, because if they are going to fly worldwide, they will need to be able to do it in English,” Major Redwood said. The Iraqi airmen are certainly not lacking in determination and dedication, the major said.

“These guys are all true patriots. They want to help their country,” Major Redwood said. “All of them were higher ranking in the Saddam era, so they took a pay cut. A lot of these guys were colonels and now they are majors. The guys coming in now are all captains, and they used to be majors and lieutenant colonels.”

APOD

APOD brings us...

What causes small waves in Saturn's rings? Observations of rings bordering the Keeler gap in Saturn's rings showed unusual waves. Such waves were first noticed last July and are shown above in clear detail. The picture is a digitally foreshortened image mosaic taken earlier this month by the robot Cassini spacecraft now orbiting Saturn. The rings, made of many small particles, were somehow not orbiting Saturn in their usual manner. Close inspection of the image shows the reason - a small moon is orbiting in the Keeler gap. The previously unknown moon is estimated to span about seven kilometers and appears to have the same brightness as nearby ring particles. The gravity of the small moon likely perturbs the orbits of ring particles that come near it, causing them to shimmer back and forth after the moon passes. Since inner particles orbit more quickly than outer particles, only the leading particles of the inner rings and the trailing particles of the outer rings show the wave effect.

Click on the picture for a larger view.

Quote for the day

This is the epitaph I want on my tomb: 'Here lies one of the most intelligent animals who ever appeared on the face of the earth.'

   Benito Mussolini