This is also a story connecting a chain of policy to a single technical decision. This story is from April, 2016. It comes from an engagement by the United States Digital Service (USDS) headquarters and the Defense Digital Service (DDS). Similar examples are legion.
The Air Force has a troubled technology project. A large part of the project is a suite of software. One of the components in that suite is a scientific calculation written in a language called C++. The calculation relies on time-sensitive data, collected by measurement stations. If the data is older than two minutes, the calculation can’t use it. The measurement stations are distributed, delivering their data over a network. Several other components in the software suite also consume this data.
The measurement stations send data using something called multicast user datagram protocol (UDP). Multicast UDP is a basic feature of the internet protocol (IP), since 1989. It provides a way for many listeners to receive the same data from the network. It scales in number of endpoints, central processing unit (CPU) time per byte, and message overhead like IP does (very well). Multicast is part of the internet code in almost every operating system in the world, so it costs no money. Every server operating system includes tools to check and debug multicast UDP. C++ programs can usually receive multicast UDP by default.
This would be fast and reliable, and depend only on the operating system. That is not how it works, though.
There is an entire software pipeline between the data on the network and the scientific code. First, a piece of software subscribes to the multicast UDP messages. It reads the data as it arrives, and checks the cryptographic signatures on each datagram. Then it re-codes the data into a format called XMLSEC, with another cryptographic signature. XMLSEC has a lot of overhead, so this step uses more memory and CPU time. This step also makes the message larger. The software then uses a protocol called SOAP to put this XMLSEC message onto something called an Enterprise Service Bus (ESB). XMLSEC, ESBs, and SOAP are complicated, come in lots of incompatible versions, and cost a lot of money. An ESB, for example, is actually an entire software suite itself. Checking and debugging them also needs special tools. Those tools don’t exist by default on any operating systems.
The ESB eventually delivers the new, XMLSEC message with the monitoring data to yet another piece of software. This software uses SOAP to retrieve the XMLSEC message from the ESB, copies it into memory, and parses out the original data. This step uses a lot of CPU time and memory. After this step, though, the data once again appears in memory just as it did when it first arrived in a multicast UDP packet. Finally, a last piece of software copies the data into a shared segment of memory where the scientific code can access it.
When this document was written, one of the major problems with the multi-billion dollar project was the scientific calculation failing because data didn’t arrive in time. Debugging these timeouts was consuming lots of engineering resources and huge amounts of time. Debugging went slowly because the production systems couldn’t include the software necessary to troubleshoot the SOAP endpoints and the ESB.
Why did the contractor put all this inappropriate technology into this system? The author asked the contractor.
The contractor said, “The customer made us do it!”
The author looked, and this was true. The Air Force wrote the request for proposal (RFP) in a way so that they would only consider solutions including an ESB. Why did the Air Force want all this complex, costly technology built into the system? The author asked the Air Force.
The Air Force said, “The Department of Defense (DoD) made us do it!”
The author looked, and this was true. The DoD issued guidance1 as recently as 2010 that requires all technology acquisitions to be “Service Oriented Architectures,”2 that comply with the “Information Enterprise Architecture.”3 Why did the DoD require this complex, dependency-multiplying technology in all their systems? The author asked the DoD.
The DoD said, “The Office of Management and Budget (OMB) made us do it!”
The author sighed, and felt sad. This confirmed his belief that we are the problem. USDS headquarters was part of OMB! Then the author looked, and it was true. As recently as 2008, OMB issued guidance and requirements that all federal IT projects have a “Service Oriented Architecture (SOA).”4 The rules for SOA require an ESB. Why did OMB doom every federal IT project to ESBs, and an architecture model proven risky, costly, and prone to failure? The author asked OMB.
OMB said, “The Chief Information Officers (CIOs) made us do it!”
The author nodded, feeling defeated, and looked. It was true. As far back as 1999, the Federal CIOs Council had deployed the Federal Enterprise Architecture (FEA)5. FEA requires federal technology solutions of any kind to include an ESB, by way of SOA guidance. Why would the CIOs ensure that all federal technology projects be broken? The author asked the CIOs.
The CIOs said, “Congress made us do it!”
The author felt tired, then looked, and it was true. There are many more pieces to this. The most important two are the 1993 Government Performance and Reform Act6, and the 1996 Clinger-Cohen Act7. Together, they mandate that the Office of the President issue a plan for all the technology. In hindsight, they doomed federal IT projects to massive, predictable failures for at least 20 years.
The author was too tired to ask Congress why they did this. The author also suspected this story has no ending, because he believes that there is no bottom. Resolve to remember this, dear reader: almost all the outcomes generated by technical policy will be unintended consequences.
DoD Reference Architecture Description: http://dodcio.defense.gov/Portals/0/Documents/DIEA/Ref_Archi_Description_Final_v1_18Jun10.pdf↩︎
Department of Defense Net-Centric Services Strategy: http://dodcio.defense.gov/Portals/0/documents/DoD_NetCentricServicesStrategy.pdf↩︎
Department of Defense Information Enterprise Architecture: http://dodcio.defense.gov/Portals/0/Documents/DIEA/DoD_IEA_v1_1_27May09.pdf↩︎
Federal Service Oriented Architecture: https://cio.gov/wp-content/uploads/downloads/2013/03/PGFSOA_v1-1.pdf↩︎
Federal Enterprise Architecture: https://en.wikipedia.org/wiki/Federal_enterprise_architecture↩︎
Government Performance Results Act of 1993: https://www.whitehouse.gov/omb/mgmt-gpra/gplaw2m↩︎
Clinger–Cohen Act: https://en.wikipedia.org/wiki/Clinger%E2%80%93Cohen_Act↩︎