Data Science • Published on November 2, 2020
- Infrastructure: We added a TypeScript compiler to the build process. This allowed TypeScript files to exist in the codebase.
- Education: We explained the benefits of the new language to engineers and we encouraged new files to be written in TypeScript.
After step three, with all new code being written in TypeScript, migrating files became a prerequisite for any major refactoring change. To expedite the conversion process and to split the work across engineers, an informal meeting was proposed to leadership. These “meetings” would address the technical debt in Shipotle in an incremental, collaborative, and enjoyable way. And thus, TypeScript Conversion Parties were born!
The parties turned out to be hugely successful. Engineers were passionate about cleaning up the codebase and a new Slack channel was born: #typescript-conversion. Anyone could post their pull request for a review or ask questions about confusing chunks of code. With this steady effort, we began to chip away at the monolithic repository. We also started highlighting bugs, documenting best practices, and creating a giant spreadsheet to track our progress. The best part was that everyone found different reasons to attend. New employees participated because the changes were low risk and they wanted to familiarize themselves with the codebase as well as TypeScript (possibly a new language for them). Others enjoyed the opportunity to stay connected with people outside of their team. Above all else, people were motivated by the mission and the free food. We provided pizza and beverages at every party, and the parties always took place later in the workday.
Eventually, though, participation diminished and enthusiasm dwindled. People were busy with their feature work and the events were optional. Many files were also avoided for being too large, too unfamiliar, too important to mess up, or a combination of all three. Despite Shipotle being shared across many different teams and dozens of engineers, it was difficult for people to volunteer their time towards tedious tech debt cleanup. After a year, it was clear that our pace was insufficient (we were only converting a few files each week), and we would need to come up with a new plan.
Seeking other solutions
The Shipotle Infrastructure team began to discuss how to complete the migration. Any-typing (the process of assigning ‘any’ to all variables missing a type) could quickly unlock some of the linting benefits. However, this technique would fail to identify bugs and improve developer confidence. Another approach contemplated was using automatic conversion tools. This option fell short because, as mentioned earlier, they needed supervision and nontrivial refinements. Lastly, advanced manual typing would resolve even the most complicated issues, but this method required significant effort from our developers. The following table roughly summarizes our findings from the different approaches considered:
The final push
After many discussions with engineers working in Shipotle, one engineer from the Infrastructure team analyzed how long it would take to convert a handful of files with basic typing (i.e. adding types when obvious and applying ‘any’ everywhere else). By extrapolating the time it took to convert several files to the total number of files in the codebase, it was estimated that a three-person team could manually migrate the remaining 40,000 lines of production code in approximately three weeks. The final proposal fell somewhere between any-typing and advanced typing, and included the following characteristics:
- Add simple and complex types when easily identifiable
- Discover and document type mismatches or bugs
- Use the type ‘any’ for those issues and add a TODO-FIX comment
In the end, the final conversion push took four weeks instead of three but otherwise was a complete success. All of the production code was now in TypeScript! Looking back, we learned a few things along the way.
- Partial conversions are inadequate. Sometimes a final push is required to reach the desirable state of completion and consistency.
- Using tools for automatically converting types seems obvious. However, the tools will likely need oversight, adjustments, and significant investment (especially for inferring complex types).
- Limit the scope of the project. Migrating does not mean resolving all bugs. Label some issues rather than further delay the mission.
- Love Problems, Not Solutions (one of our company core values). To achieve success, focus on the goal rather than the process, without compromising on quality.
Finally, this endeavor could not have finished without the patience and persistence of everyone involved. Every time work stumbled and lost momentum, someone else was there to pick up the torch. Now that the migration work is complete, we can ship code at a faster speed, in greater confidence, and best of all, with fewer bugs. The seemingly insurmountable task was in fact possible and we hope this story inspires you to take on your own difficult migrations. Thank you for reading and good luck!