Today's lecture: ========================== Clock synchronization technology The Huygens algorithm Technical aspects of stock exchanges The White Rabbit algorithm Clock synchronization: ========================= Getting different physical locations to agree on their notion of time Requires us to synchronize both clock offsets and clock frequencies on different clocks Complicated by uncertainity in propagation delay between two different locations: Fundamental results in distributed algorithms that relate clock sync difficulty to this uncertainty: Lundelius and Lynch, 1984 Why is clock synchronization relevant to finance: ========================= Needed to measure one-way delays accurately to know latency from market participants to exchanges One-way delays: Delay from point A to point B Requires us to subtract receive time at point B from transmit time at point A But can't carry out such subtraction unless the two times are synchronized. How to perform clock synchronization: ========================= Many many approaches over the years Main idea: Take a timestamp at location A (tA) Send it to location B. Take a timestamp at location B (tB) --> Naively, apply tB - tA to correct A's clock to be consistent for B (at least instantaneously) --> But we are omitting the fact that tB and tA were not taken at the same time. --> In particular, the one-way delay between A and B has elapsed between the times at which A and B were taken. --> But what is this one-way delay? --> Don't we need clock synchronization to get the one-way delay in the first place? This seems circular. Estimating one-way delay in the absence of clock synchronization: ========================== Many ways of doing this: one approach --> Simplest way: measure round-trip time from A to B back to A. --> Divide by 2 to get estimate of one-way delay. --> Roughly how NTP works. Can we do better? How NTP actually works. ======================== --> Take a bunch of RTT samples. --> Take the minimum of all of these samples to throw out those with queueing delays. --> Then take the average of these samples. --> Divide by two to get more precise one-way delay estimate. --> Use this for clock synchronization --> This is how your laptops/phones, etc. synchronize to Internet time. Can we do even better? ======================== What if one-way delay isn't actually RTT divided by 2? Happens when the forward and backward paths have different levels of queueing, different Internet paths, etc. Can track "dwell times" (queueing delays) in switches and subtract these out. Can apply NTP like techniques between switches themselves where the path is more controlled (i.e., it's a cable, rather than a sequence of Internet routers) Can use physical layer probes to keep clocks synchronized to each other (DTP) The WhiteRabbit system at CERN takes these ideas to a logical extreme: ps-level precision with customized hardware on top of Ethernet switches WhiteRabbit has found users outside of CERN, notably in the finance world. Motivation for Huygens: ======================= Can we get high accuracy without requiring customized hardware for clock synchronization? Huygen's main difference: a software algorithm to synchronize servers, without requiring assistance from switches. Huygen's main ideas: ======================== Coded probes Support Vector Machines Network effect Coded probes: ======================== Send back-to-back packets with a certain separation. --> If the separation increases considerably at the receiver, it means likely one of them hit a queue and the other didn't. --> If the separation decreases considerably at the receiver, it means likely both of them hit a bottleneck link, were stuck behind many packets and then came out rapidly one after the other. --> If the separation is maintained, decent chance that they both encountered a low delay from sender to receiver --> Use these probes. --> New use of probes: previous uses involved their use in available bandwidth estimation Support vector machines: ======================= Classical technique for supervised learning Before deep learning took over Allows us to draw a separating line between points labeled +1 and -1. How SVMs apply here: ====================== Compute lower bounds on clock discrepancy Compute upper bounds on clock discrepancy The paragraph above Figure 4 shows how to compute these upper and lower bounds. Plot all these points as a function of one of the NIC's time. Apply SVM library on this data to find separating line. Remarks: ====================== Do the SVM filtering after throwing out bad probes --> Figure 8 shows the benefits of probe filtering Do this over short time scales of about 2 seconds --> Figure 7 shows what happens over longer time scales, where non-linearity creeps in --> Non-linearity result of clock frequency changes due to temperature Interpreting the SVM lines: ============================ Slope of the line reflects drift in frequencies between clocks A and B Intercept measures clock offset between the two clocks Figure 5 shows examples of clock drifts Network effect: ============================ If clock A is off from clock B by x, and clock B is off from clock C by y, then clock A must be off from clock C by x+y due to the transitive property If they are not off by x+y, it means our previous determinations of x and y were wrong and need to be corrected. Idea in network effect: find by how much the transitive property is violated and apportion this across all the affected nodes many ways to do this: Huygens uses the min-norm solution More the nodes in the system, more opportunities to find triple (or quadruples or higher) to test the transitive property And the more the violation of the transitive property, the more the opportunities to fix the clock sync errors Other things that Huygens benefits from: ============================= Symmetric topologies: Distance from A to B in network hops is the same as distance from B to A. True in a datacenter, but untrue in the wide area in general. Huygens evaluation: ============================ Use two locations that are known to be controlled by the same clock. Pretend that they have different clocks and try to correct their clocks. Use this to determine how well clock synchronization works. CloudEx: ========================== Applying synchronized clocks to stock exchanges Fairness in stock exchanges is everything Unlike the trading perspective, where low latency is everything How to achieve fairness in exchanges: ========================= One way is to measure out cable lengths down to the cm or inch and ensure all participant to exchange lengths are the same But that's impossible to do when you move to the cloud. Use synchronized clocks with delay compensation as an alternative. Two flows in exchanges: ============================ Inbound: market order submission, cancelations, etc. Outbound: market data submission Examples of real inbound and outbound protocols: NASDAQ's ITCH (outbound) and OUCH (inbound) Many more examples here: https://en.wikipedia.org/wiki/List_of_electronic_trading_protocols Interesting difference from existing protocols: drops are OK, it's on the market participants to re-request important market data But how much delay to add? ======================= Tune this based on observed fairness on the inbound and outbound side If unfairness is too high, increase delay parameter, else decrease it. How do you lower latency in CloudEx? ====================== Hedge: send in orders through two different gateways. Matching engine picks the earliest one that reaches the matching engine. White Rabbit: ====================== Extreme clock synchronization techniques PTP+DTP ideas combined together: 1. Estimate dwell time at each network element. 2. Use physical layer techniques to synchronize clock frequencies Some extreme use cases like distributed oscilloscopes.