Releasing a new version of an open-source library is an exhilarating experience. You ship new features, bug fixes, improved documentation, etc., to serve your users and also the mission of your library. Being one of the maintainers of the 🧨 Diffusers library, I am no exception to this.
Once a release is finalized, it’s usually published on a software repository for the distribution programming language you’re using. Diffusers is a Python library, so PyPI is our publishing platform.
In this post, I share some of what I learned from trying to streamline the entire process of releasing a new version of the library and then publishing it. If you have similar responsibilities at your workplace or for your personal projects, this post might be helpful for you.
An example release workflow (manual)
Before we proceed to the other sections of the post, it will be helpful to have a schematic of what constitutes a release. Note that this workflow will vary from library to library, but some principles will still apply. I will take the workflow we follow for Diffusers as an example.
The steps are well laid out in setup.py and can be found here. Broadly, these are:
- Prepare the release branch and cut it out from the
main. - Run any test on the release branch and wait for them to pass. Fix any failures if needed.
- Tag the release branch and push the tag.
- Build the package source and wheel.
- Upload the package distribution to the Test PyPI server and run any tests.
- Finally, upload to the actual PyPI server.
We identified that steps 1-3 will always require a bit of human intervention and cannot be automated much (props if that’s not the case for you). But steps 3-6 can indeed be automated. These steps require more attention, too:
- When building the package distribution, one must delete the previous one before starting the build. Otherwise, it can have unintended consequences.
- Managing the credentials for the Test PyPI and PyPI servers.
- Running any tests after publishing them on the Test PyPI server.
These steps would be better automated in your library’s Continuous Integration suite, greatly reducing the mental burden.
Semi-automating the release workflow
Once we identified the above findings, we prepared a GitHub Actions workflow that gets triggered after a release is tagged and the tag is pushed. Additionally, we configured the workflow to be manually triggerable in case any intervention was needed.
This workflow takes the following steps:
- Find out the release branch so that it can be checked out for the sequential steps.
- Steps 3-6 as outlined in the above section.
It’s worth noting that the trigger for this kind of workflow should be figured out to suit what’s best for the given project. In the case of Diffusers, we realized that release steps that come after pushing the release tags can be largely automated. Hence, we went with that trigger.
The workflow file is available here. When successfully executed, it appears like so:
You can find the complete details about the action run here.
The initial workflow was missing a dependency that was needed to run the import tests after Test PyPI publishing. This was fixed in this PR. So, please double-check any dependency that might be needed to run the tests after your package has been published on the Test PyPI server.
The workflow doesn’t make use of any pre-built actions (such as pypa/gh-action-pypi-publish@v1.6.4) for publishing on PyPI. Instead, we decided to just follow what we’d do manually, i.e., use twine to manage the process. If you’re looking to use such an action, this can be a handy reference.
Publishing the release notes and communications
The next step in the release process involves publishing the release notes on your repository and tagging it. Once a release is published, team members usually communicate about it internally within an organization and also more broadly with their communities through social media channels.
On the Diffusers team, we take release notes pretty seriously (example notes). This is why we intentionally keep the process of writing the notes and publishing them purely manual. Once a release is published on the repository, a workflow gets automatically triggered to communicate about it to an internal Slack channel. Successful execution of this workflow makes a bot automatically post the message below to a particular Slack channel:
This workflow can be found here.
Both the above steps were introduced in Diffusers through this PR. I recommend readers to go through it if they want to incorporate similar changes in their projects.
Considerations
I played with the workflows rigorously on a dummy repository before introducing them in Diffusers. This is optional but highly recommended to confidently land similar changes in your actual projects.
We used incoming webhooks on Slack so that the bot could post messages. If you’re configuring something similar, this official tutorial can be quite useful.