From Seed to Harvest: A Story of How an Open Source Project Evolved

I was thinking lately about conferences and it reminded me of two which hold fond memories for me: my first KubeCon/CloudnativeCon and my first Helm summit. As I reminisced, I realized then that these two conferences struck a cord with me because of my journey starting out with Helm 3 at KubeCon/CloudnativeCon NA 2018 (Seattle) and the release nearing completion at the Helm summit 2019 (Amsterdam). Helm 3 was released in time for KubeCon/CloudnativeCon NA 2019 and it made me wonder how things evolved in such a relative short space of time. Much has changed from Seattle 2018, Helm 3 has been available since November 2019 and Helm 2 will go EOL soon in November 2020. Users have started to use Helm 3 and have moved over from Helm 2 in stages that suit them. When I think about that Thursday evening in Seattle on the last day of the conference, where a large crowd turned out yearning for Helm 3 availablity. One question from the audience stands out among all the excitement, “How will one migrate from Helm 2 to Helm 3?”

Well, this is the story of how this question led to a small but key Helm open source project called Helm 2to3 Plugin.

The Background

When I joined the Helm community in Autumn 2018, Helm 3 was in early development. I joined with the “chop wood, carry water” mentality and with the goal of helping get Helm 3 released. Helm 3 was a major release which centered on the chief aim of removing the Tiller server. The Helm 2 architecture included an in-cluster server (Tiller) that interacted with the Helm client, and interfaced with the Kubernetes API server. This architecture made sense when Helm 2 was released in 2016. Kubernetes developed rapidly in the intervening years and by 2018 Tiller was no longer required. The community called for a more “simplier and secure” tool, and Helm maintainers and contributors obliged.

You are probably wondering at this stage, where is the open source project in all this? Hold on, we’re getting to it next.

The Seed is Planted

At KubeCon CloudnativeCon NA 2018 in Seattle, Matt Butcher and Adam Reese (2 of the co-creators of Helm) presented a Helm Deep Dive session. This session was very interesting because it allocated a substantial block of time at the end for Q&A with the audience. It was during the Q&A that Henry Nash (a Helm contributor) raised a really good question/point about the “impact at upgrade” for exisitng Helm 2 users. This led to a really interesting and interactive discussion between users, contributors and maintainers about migrating to Helm 3. As it was unfolding, Michelle Noorali (Helm maintainer) jumped on this and created an issue (live on screen in front of attendees) for a helm 2 to helm 3 upgrade story. Josh Dolitsky (Helm maintainer) meanwhile germinated a really cool idea of a “Helm 2 to 3 plugin” and before he knew it (and to much laughter) the issue was assigned to him! The power of open source was alive for all to see.

The seed had been planted at the conference and thanks to Michelle for raising the issue as it gave a place to continue the conversation. In the days that followed, a number of contributors and maintainers jumped on board and started adding ideas and use cases. It is worth recognising the input from Henry Nash, Rimas Mocevicius, Adnan Abdulhussein and Matthew Fisher who fleshed out a good starting point for migration. Josh Dolitsky also dropped a really good UX design for the Helm plugin concept that he had raised at the session.

The Project Journey Begins

The conference displayed the appetite in the community for Helm 3. It was therefore all shoulder to the wheel on driving it forward towards release in the coming months. As a consequence, things started to get in the way and migration ended up on the back burner.

I subsequently got more involved in the project and became a maintainer in April 2019. This was a very busy period with pushing and reviewing PRs, issue triaging and other day to day project work. As contributions were steadily increasing for Helm 3, the community tried to maintain the Helm 2 code base and at the same time bring Helm 3 to fruition. Even though I was busy on other things, I kept the migration feature always at the back of my mind as something needing completion before release. I had experince with past product upgrades and knew the importance of a good migration path for users.

Around April 2019, I had a conversation with Henry Nash about migration. He had a lot of experince with product upgrades also, and was wondering how the migration feature was progressing as he too was distracted with other work. I raised it at the next Helm dev weekly meeting, offering to investigate the feature. I got the “green light” from folks to go look at it and started by documenting my investigations. I initially naively thought that this could be covered by documentation only.

The investigations went on in the background during May and June as we tried to finish Helm 3 features and stabilize the code base enough for Helm 3 Alpha 1. By June it started to become more apparent to me that due to changes in the underlying architecture and plumbing from Helm 2 to Helm 3, documentation only wouldn’t be sufficient and a tool would also be required as had been mentioned at the Seattle conference. Scripting would not solve this one! It was around this time that I raised an issue in the Helm core project to cover this. The feedback from a dev weekly meeting at this time was to create a proposal with a PoC so that contributors could review it.

I worked on a proposal on a part-time basis and it was available at the start of August for review. There was some really helpful feedback on the PoC from Matthew Fisher, Rimas Mocevicius and Taylor Thomas which helped simplify the plugin by calling v3 storage directly and using Maor Friedman Helm v2 utility functions.

Following the feedback of the PoC, I used it as a base and created a Helm plugin which I pushed to GitHub. The little open source project was now available to help users migrate from their Helm 2 environments.

The Project Gains Traction

Once the repo was available, it started to gain traction quickly. Helm v3.0.0-alpha.2 was released at the end of July, Beta 1 release was due in September and the Helm Summit 2019 in Amsterdam was also that month. It proved good timing that the migration tool was available for users to try out at that time.

One of the users who became an early adopter of the plugin was Rimas Mocevicius. Rimas (who was a co-creator of Helm but now was more on the user side) started to test the tool from the offset in his environments. His goal was to be able to upgrade Helm in his company as soon as Helm 3 was released. He understood the importance of migration for his company and others.

I soon realized that the tool should belong in the Helm organization as it is a key component for existing users moving to Helm 3. I raised this with the Helm organization to move the repo from my GitHub account and it was put under consideration. I hoped to discuss this further with other Helm maintainers at the Helm Summit in early September.

The day before the summit, I met Rimas who was very excited about the plugin. By then, he had started to contribute to the project and wanted to promote it at the summit to grow people’s awarness. He had added build automation to the project and was also looking to move the project under the Helm org. I agreed that we would discuss it with other maintainers the next day.

We discussed it on the first day of the summit with two of the maintainers (Matt Butcher and Matthew Fisher). There was an agreement made that it should move into the org. It was suggested to contact Matt Farina (a Helm org maintainer who couldn’t make it to the summit) to help facilitate the move. One small issue of different timezones first though! Once Matt was online, Rimas grabbed me while I was trying to enjoy a beer after day 1 of the summit. We recovened to a back room at the summit and worked with Matt to get it moved into the Helm org. It was all a bit frantic at the time but it was worth it in the end as we wanted to make it available during the summit.

The final piece of the jigsaw was to officially launch it and make it widely known to users. I announced the plugin availability and Rimas announced a how to migrate blog.

The Harvest Moon

I suppose you really know if a tool is being used, when bugs and features are being raised. And so it was over the next few months, people contributed from the community. One feature that displays the interaction with users was a request to support cleanup of a specific release. This was something we hadn’t realized that people wanted. We thought they wanted to do the conversion of the releases and then cleanup fully once happy. This request shows the value of the different use cases of a tool/product that user input provides.

The plugin has proved useful to users as they migrate from Helm 2 to Helm 3, since Helm 3 was released in November 2019. The tool will eventually pass its “use by” date, once all migrations are complete. It can then be retired gracefully.

Conclusion

It was not the goal to create an open source project when the question was asked about migration in Seattle. It was about helping people move onto Helm 3 and it is good that the tool is there to help.

Writing this piece however has made me acutely aware that the journey was also a really interesting part of this process. It showed how people from all walks of life, working for different companies or independently, can work together to help people. It is about contributions and this is the true value of open source communities.

It may have started as a feature request in Seattle in December 2018 and it may have taken till Amsterdam September 2019 before it was realized, but it happened. Thanks to all who contributed and apologies to anyone who I may have forgotten.