The Cost of Endless Magic — Theme Parks and Technical Debt
How the language of software development can describe the functioning of a theme park.
On a day in early 1995, I was at Disneyland while two important events were happening. First, Ron Dominguez — former vice president of Disneyland who had, at that time, been the park’s longest serving employee and upon who’s family farm parts of the park were built (look up “The Dominguez Palm”) — was having his window on Main Street dedicated and second, there was a preview of the soon to open Indiana Jones and the Temple of the Forbidden Eye attraction.
I didn’t know that either of these things were happening when I arrived but was delighted to see the dedication and to see George Lucas wandering around the front of the Temple queue and (ohmygoshohmygoshohmygosh!) to actually be bumped into by Roy E. Disney.
“Excuse me,” he bestowed.
“Sorry,” I acknowledged and leaped out of his way.
It felt like we had a real connection.
Anyway, a lot of us stood behind the rope barrier for quite a while hoping that we would get let into the soft opening of the ride after the press and celebrities had passed through. We didn’t, which is a shame because that might’ve been the closest to the complete, original, ride that I would ever have experienced. A notoriously complicated attraction, elements began to fall away — some of which would never be repaired — almost immediately.
Now, I want to be clear, if you go on Indy today — and it just re-opened after a prolonged refurbishment — you will have a wonderful experience. Some things about the experience are actually better than what you would have experienced in 1995, but it is the first example that I know of where the park had a ride with elements that had gone offline and an intentional decision would be made to never repair or replace them. This leads to what I’d like to discuss here: the concept of “technical debt” in theme parks.
Tech debt, as those of us in the technical architecture field call it, is usually defined as a flaw in some sort of technology caused by budgetary or time constraints which will require a certain amount of technical expertise to deal with in the future. Originally the term was coined to explain the need for “refactoring” software code: taking out a troublesome piece of code and rewriting it to reduce its negative impact on the use product. The term has taken on a life of its own, however, as it proves to be such a useful metaphor for quite a number of technical and engineering projects.
Here I’m going to use it to describe technologies surrounding theme parks where the impacts of the debt are particularly acute and the idea of refactoring must be carefully weighed out in order to balance the benefits against the potential customer disappointment of a key attraction or feature being missing from an expensive — potentially once in a lifetime — park experience.
An organization can deal with tech debt in a few ways. It can continually make minor compensations to keep the technology hobbling along, or it can try and simply ignore the element that is the problem and these choices can result in what is often referred to as “human spackle” where constant human interaction is required for something that should be automated. Think of it as interest on the debt that you must always pay without the principle ever decreasing. The organization can also deal with the debt by choosing to replace the element of the technology that is causing the problem with a version that no longer contains the debt; this is traditional refactoring. Finally, it can scrap an element or the entire product. The plusses and minuses of each of these decisions are fairly evident. If you human-spackle the problem, you have committed unplanned resources to the issue in perpetuity. If you refactor the issue, there will be long-term gains, but fairly severe short-term losses as you have to bring the attraction/show/interactive application offline (usually) to fix it. Finally, if you scrap the element, you risk diminishing the value of what you’ve built and if you scrap the product itself, you have thrown that value away entirely.
Okay, so that’s all a little dry, and I promise we’ll get back to all the cool theme park stuff shortly, but here’s the secret about tech debt — it’s not always bad. In fact, in some industries that work on the bleeding edge of technology, it is almost always inevitable to have a certain amount of it to make the new things you’re trying to make; and the tech debt incurred can even often teach us about the nature of those things. If I make some sort of never before seen particle beam and every time I set it off my desk is covered in a kind of copper dust, I’ve learned something interesting outside of my goal. I do have to clean up my desk EVERY DAMN TIME, but I’ve invented something new and learned an interesting thing about its nature. Tech debt can grow your understanding in the same way that a misplaced musical note can spur brilliant improvisation.
The classic definition of technical debt is that it is incurred due to a lack of funding or time, but the reality is that every project has a lack of funding or time at some level in a finite universe within a capitalistic monetary system; so embracing your technical debt while, at the same time, understanding that you will have to deal with it in some way, is often wise; which brings us back to Indy.
Now Indiana Jones and the Temple of the Forbidden Eye has a raft of effects that don’t work/only kind of work/have been turned off. We’re never going to see the ice fall from the chamber ceiling again. The diamond tile in the queue is never going to cause the rocks overhead to move and threaten the passers-by. We’ll only ever go through the center door in the Chamber of Destiny from now on. The less said about the “rat branch” the better. But who cares? As an audience, we have reached a rapprochement between the Disney technical and maintenance staff and what we think is acceptably fun. To be fair, it’s more than acceptably fun. It’s wonderful.
Disney has chosen a mixture of solutions to deal with Indy’s tech debt: they continually fix some elements (The blow darts and rats have been, at best, sporadic.), let others fade away, and have completely replaced others (I hear there’s a new giant cobra!); and here’s what is important to keep in mind, Disney’s technical resources — skilled staff and money — are also finite; so it will be up to them to decide how much human spackle they can attach to the ride to keep it up to the standard they’ve chosen.
But, for now, know that Indy leverages its tech debt in many different ways, some of which include human intervention on a somewhat regular basis. (If you’ve ever had a walk off of an attraction, you have seen human spackle in action. In fact, you have BECOME human spackle.)
Let’s roll back a bit and look at some earlier examples of technical debt and how the Disney company dealt with it. These two examples show how two decisions, both made in 1966, handled the problem in different ways given the resources at hand.
Now, before we step into this first attraction I’d like to remind you that if you feel off balance or nauseous, you can just look down at the floor and hold the handrail… Please make your way into the darkened theater where the show is about to begin. The film — well, films really — will be projected just slightly above the handrails in fabulous Circarama — a 360-degree image completely surrounding you!
Circarama/Circle-Vision 360 (1955–1997) was an opening day attraction at Disneyland that was primarily engineered by famed animator and Mickey Mouse co-creator Ubbe (Ub) Iwerks. Iwerks took eleven 16mm cameras and mounted them on a metal disk using an elaborate chain system to keep them all in sync. The disk could then, in turn, be mounted on any number of vehicles — cars, trains, helicopters, etc. — to film the panoramic images that would be projected in the attraction.
Ub’s son, Don Iwerks, who had previously worked on the Disney feature 20,000 Leagues Under the Sea (1954) was sent with the film crew to help with the technical hurdles of keeping the unusual camera rig running and in sync. Don would continue to partner with his father through many permutations of this attraction.
In terms of technical debt, there was little involved in the filming process of the Circarama (soon to be re-named Circle-Vision 360) movies. Once they were done and cut, they simply had to reprint a negative until Imagineers decided that they wanted to make a new film, which happened so infrequently as to hardly count towards any sort of technical inefficiency. However, the projection of the film was another matter entirely.
The Circarama films ran continuously anywhere from eight to eighteen hours every day, up to seven days a week. A circular theater with eleven screens allowed projectors placed between the gaps between them to project across the room onto an opposite screen which would, when they all ran at the same time, produce the 360-degree effect. The films were all loops, feeding from a central hole in the reel, going through the projector, and winding back onto the outside of the reel they had started from. If the film broke, which Don Iwerks estimated happened about every thousand hours of play — roughly every 100 days or so for each of the eleven films being projected in tandem — not only would the film in question have to be replaced, but the whole attraction would have to be stopped and all eleven films resynched. Now one outage every 100 days or so doesn’t sound that bad until we consider that not all eleven films are going to break at once. So they will begin to stagger. Soon you will have a scattering of breakages over the course of that 100 days. And that is where the human spackle again steps in.
For projectionists to have to constantly be in place and monitor an attraction that is built to be continuous and occasionally have to pull all eleven projectors off-line when only one piece of film has broken is a troubling amount of technical debt. Not only is the park staff paying fairly constant interest on that debt with their monitoring and repair duties, but film prints are having to be struck regularly off of negatives to replace the broken film. (In a standard theater scenario, a broken film could be spliced back together without replacing the whole movie, but if you lost a frame — inevitable with splicing — every twenty-four runs of the film would put it one second out of sync with the rest of the projectors, which would be noticeable fairly quickly.) So we have stacked up several charges of technical debt for this early attraction.
By 1966, Walt Disney was about to make some big changes to Tomorrowland and Circarama was going to need to get its act together in order to stick around. Walt held a meeting with Ub and Don Iwerks to see if they could somehow improve the performance of the attraction. He was asking, without this term having been invented yet, his technicians to reduce his technical debt. Ub and Don and their team came up with a few brilliant solutions to refactor the attraction and severely reduce the debt.
First, they determined that the jury-rigged, but purchased off the shelf, 16mm projectors that had run the attraction for over a decade, were not serving them well. The sound for the film was produced externally from the projectors (otherwise there would be a cacophony of eleven soundtracks each potentially milliseconds off) and much of the internal machinery of a modern projector is built specifically to make sure that the embedded soundtrack on the film reaches the sound head in just the right position at just the right tension. Custom building new 35mm projectors, they had many of the guiding wheels and sprockets left out. Without this extra tension and friction, the film would gain some longevity. The team also got rid of the self-feeding film spools, instead letting the stock flow into large cabinets with guides leading it into gentle loops which removed both the stress of the tight wrapping around the reel as well as the friction of being pressed up against the rest of the film.
These changes as well as the robustness of the wider film which also allowed for a broader circle with fewer projectors — from eleven to nine — improved the performance of the attraction dramatically.
“… the film went from maybe one thousand runs [ before wearing out]… up to ten thousand or fifteen thousand.” — Don Iwerks quoted in The Imagineering Story, Leslie Iwerks
This example of refactoring an application (or attraction) to dramatically reduce its technical debt is almost classic in execution and speaks to the genius of these two men and their team. Debt was identified and reduced through innovative refactoring.
For a second example, with a significantly different outcome… Introducing Disneyland’s Flying Saucers (1961–1966)!
Designed by Imagineering legend Bob Gurr, the Flying Saucers were built, in part, to bolster the fairly anemic Tomorrowland of Disneyland’s first decade. Best described as something like bumper cars on a giant air hockey table, the Saucers had a lot of promise. Indeed, when they built the prototype, it worked great. By the time they were assembling the attraction in Disneyland, however, problems began cropping up.
First, the plenum (the area under the attraction within which the air pressure had to build up) was over twice as deep as the one they had built for the prototype, meaning that it was much more difficult to keep the pressure at the necessary intensity to keep the saucers afloat. Second, they had not tested thoroughly with lighter-weight people who, they discovered, would find themselves being popped up and down in place without the weight to cause a significant enough lean in their Saucer to shift the air pressure underneath. That pressure shift was necessary to cause the nearby floor valves to open and effect motion.
And, finally, there was the effect that Bob Gurr called “undamped divergent oscillation.” This was when an unpredictable collection of floor valves would all be opened at the same time causing a massive “boom!” and loss of pressure to the entire attraction. Every rider’s saucer would drop suddenly to the floor.
Surprisingly, this slow loading, not entirely effective, attraction with massive amounts of human spackle — the loading and unloading was a Cast Member ordeal following an air pressure drop, and technicians were also needed to build the pressure in the plenum back up — was kept open for almost five years. The cost-benefit analysis must’ve included the fact that Tomorrowland, at that point, had very little in it and to remove the attraction would’ve left the land near its hub-side entrance uncomfortably sparse. With the soon-to-come renovation of Tomorrowland in 1967, however, adding Adventures Through Inner Space, The Wedway PeopleMover, and The Carousel of Progress, that sparsity became significantly less critical and it became time to assess the impact of this attraction’s technical debt.
So, what was the debt incurred by this project?
- The plenum was too deep and made the functioning of the attraction erratic.
- The body sizes of certain guests — particularly smaller, younger, ones — were going to cause a poor experience for that rider.
- Unpredictable problems often come with entirely new technology: nothing like this attraction existed in the world before this did and that often begets technical debt, such as Gurr’s divergent oscillation problem which was a consistent issue. (We see this form of debt a lot at Disney parks and it may be the BEST reason to incur it: doing something new!)
Fixing this attraction was never in the cards as, to some extent, they couldn’t really isolate the conditions that would cause the undamped divergent oscillation. It was too unpredictable. Also, the plenum being too deep would’ve likely required gutting the entire attraction to rebuild its sub-structure. And the continuing interest payments were too high due to the almost constant human intervention for the time in which it operated. The novelty of a five-year-old attraction that didn’t work very well in a land about to be filled with new cutting-edge stuff was never going to warrant the extreme amount of refactoring to pay off the tech debt, and thus it was scrapped.
Weirdly, Disney tried to build another variant of this ride almost fifty years later with Luigi’s Flying Tires (2012–2015) in Disney California Adventure. It also didn’t really work.
Two attractions evaluated in the same year with different decisions made to deal with the technical debt that they incurred gives us a wonderful model by which we can use modern tech terms to try and gain some insight into how decisions might’ve been made about the parks in the early days.
Now let’s translate that thinking to a more modern version of the parks, particularly Disneyland. What has changed since these examples above in 1966? Pressure has increased on a lot of fronts: the parks being open seven days a week, tremendously higher attendance paying tremendously higher prices to be there, shareholder demands for profits, competition. If paying off interest on technical debt or taking an attraction offline to refactor elements of it used to be like repairing the engine of a moving train, now it is like a tightrope walking between two jet fighters.
In 2019, pre-COVID, Disneyland was welcoming an average of, very roughly, 50,000 guests per day; some days less, others much more. By comparison in 1966 Disneyland was welcoming less than half of that, on average, per day. (Please note that Disney does not release official daily counts and so much of this depends on what information the company DOES release measured up against sites that work with park statistics.) The park has grown in size and attractions over the years which works to mitigate the effects of the crowd increase, to some extent, but this puts a heavy burden on the operability of these attractions to keep swallowing people.
In other words, if Indiana Jones and the Temple of the Forbidden Eye eats 2400 riders per hour and it is offline, those 2400 folks need something else to do; so they will increase the crowd elsewhere in the park. 2400 people cast adrift is not an insignificant proportion of your 50,000-per-day average; so it is noticeable. (I’m suggesting broad strokes here. Obviously, there are people in queues for extended periods. The attractions are often only a few minutes long. For far more in-depth studies of crowd mechanics, check out touringplans.com. That’s their bread and butter and nobody does it better.)
This is all to say that taking an attraction offline for refactoring is a major decision, and this is beyond the simple concerns of a refurbishment which may be baked in as a safety necessity. This is something more. On-going technical debt can also be a major problem if an attraction keeps breaking down or intermittently pausing to pay some interest that looks like an inefficient loading scheme or an unplanned malfunction requiring a reset. And there’s one other piece of interest on the debt that we’re circling around here which we should also acknowledge, that of customer satisfaction.
This is where the current, potentially concerning, state of technical debt comes into play.
Acknowledging that it is a good thing to incur the debt when in service of a bold new idea leading to an ambitious project and that there are, now, market forces that demand something new and cutting-edge every few years, it is important to understand that the debt incurred still has to be paid in one way or another, no matter how noble the reason for its accumulation. If Disney doesn’t have enough resources in money, labor, expertise and time to, at least, cover the interest on the debt, they will do what all of us do when we begin to default: start to fall apart.
Here is something I experienced at Disneyland recently: the newest attraction at the park Mickey and Minnie’s Runaway Railway (about 2400 possible riders per hour) went down. This is absolutely normal and understandable for an attraction this new. It will go up and down all the time. At the same time the most popular attraction in the park, Rise of the Resistance (about 1900 potential riders per hour), also went down. It is the most sophisticated and complex theme park attraction of all time: again, understandable. Now the park has displaced some 4300 guests per hour on a moderately busy day, say about 60,000 people, due to technical debt and resources are being employed not to fix the piece that’s breaking — that would be paying off the principle of the debt and involve a refactoring and even more downtime — but to reset the attractions in order to run them as long as they can before the fundamental issue rises again and the attraction must again be reset — paying off constantly accumulating interest. Resources will need to continually be allocated to these attractions just to maintain those payments.
A quick note: Rise of the Resistance has had some refactoring done along the way with the cannon motion towards the end of the attraction having been, seemingly permanently, disabled. Also, both Rise and Runaway Railway have factored in resilience to the impacts of their debts. Both have “B modes” which can substitute for certain more temperamental effects and Runaway Railway has brilliantly added extended character effects, such as Mickey whistling after his song ends or Daisy taking some extra deep breaths before her dance lesson starts, to compensate for ride hiccups or delays.
During these outages Indy was also down for an, I’m pleased to say, large-scale refactoring about which I’ve heard great reports. Nonetheless, that’s another 2400 people that would normally have a place to be, flowing elsewhere in the park.
We’re up to around 6700 people per hour that are not in attractions. Given our wobbly estimates, it’s more than ten percent of that day’s total attendance at the park. (Again, I am aware that there are elements that make this less cut and dry than my description — on a 60,000-person attendance day, the guests aren’t all there at the same time, for one thing — but, as I said, broad strokes.)
Now we run into what I might call “a canary in the coal mine:” a sign that the technical debt might be getting unmanageable. The single highest capacity attraction at Disneyland, the absolutely classic Pirates of the Caribbean was — you guessed it — also down for much of the day. This attraction has steadily eaten about 3400 riders per hour since opening in 1967. It is a remarkable piece of work that only rarely has issues serious enough to bring the attraction offline. It was up and down multiple times that day. Why is this so significant? Well, having no inside knowledge of what’s going on with any of these attractions, this could just be a coincidence, but let’s hold onto the idea that we have multiple, complex, rides with a lot of technical debt that are securing a certain level of resource to each in order to keep the interest from getting out of hand, and that those resources might be starting to become spread a little thin. Could it be that there simply isn’t enough technical staff resources to go around? If substantial portions of your maintenance staff and budget are being concentrated on a few complex attractions, do other attractions suffer? At what level has the park incurred too much technical debt: perhaps when a normally reliable attraction is not having its interest paid?
And maybe it’s not just attractions, we could also add to the debt elements like Genie and Genie+ as well as Individual Lightning Lanes, Virtual Queues, and phone-based interactive experiences. All of this adds to the debt burden that the park must, in one way or another, assign resources to maintain or assume the burden involved in refactoring.
This is a new world of high-tech engineering and illusion. As we’ve seen, the old ’60s model had already become outmoded due to the intensity and expectations of the park’s guests and by the ’90s different strategies had begun to come into play. Now with technology advancing faster than anyone could imagine thirty years ago, much less sixty years ago, it may be time for Disney to re-evaluate again how they deal with technical debt. Perhaps it's a matter of money or perhaps it’s a new philosophy of how refactoring decisions are made. Perhaps it’s also a new strategy for setting guest expectations. So many elements can factor into a coherent architectural strategy for dealing with your new and legacy technical debt.
What I don’t want to do here is to give the impression that I’m complaining about the park for having outages. Quite the contrary, I am deeply impressed that Disney continues to swing for the fences, even if it means some struggles on the implementation side. Walt Disney, notoriously, often made foolish financial mistakes whenever he was trying to create something new. Notably, he deeply indebted himself and his company to recording mogul/con-man Pat Powers in order to get sound put on Steamboat Willie. It took the company years to get out from under those terms, but it clearly paid off in sheer progress and we wouldn’t be talking about this company almost a hundred years later if he hadn’t. It’s the nature of this company to take risks and they should be commended every time they do so, even if they have to pay some technical debt on the back end.
Managing one’s technical debt is a difficult, and often costly, process. Resources, skills, and money: they all have to be balanced. It’s critical, though, that it gets dealt with in some way. As we saw with the examples from 1966, this is not a modern phenomenon; the technical terms may be, but the ideas go as far back as there has been the courage to invent and make new things.
And that brings us back to Indy: a complex, elaborate, technically sophisticated attempt to simulate a movie serial from the 1930s. Some effects are gone and some new ones have been added. The experience you will have now will be different than it was in the ’90s because the technical debt incurred those three decades ago has been partially paid and, by adding brand new elements, new debts may have been leveraged. But ultimately, for all this, the experience you will have there… will still be wonderful.
Credit where credit is due: Sam Gennaway’s The Disneyland Story was an invaluable source of information about the Disneyland Flying Saucers attraction as was Leslie Iwerks’ The Imagineering Story for her father and grandfather’s work on Circarama/Circle-Vision 360.
I consulted TouringPlans.com for attraction capacity information and any figures which are correct are entirely due to them and any that are incorrect are the best guesses I made on my own.
Be sure to follow Boardwalk Times for more coverage of The Walt Disney Company.
Support Boardwalk Times on Ko-Fi.
If you like this article. Subscribe to our free newsletter Boardwalk Times: Stories from the Seashore, a monthly dose of Disney and its businesses.
Kelly McCubbin is a columnist for Boardwalk Times
