Using GitHub Actions to manage CI/CD for Empire

We’ve been using GitHub actions for Empire and Starkiller for quite some time now. It’s been a significant productivity boost for our releases because we manage multiple versions of the two projects. When doing the releases manually, there were a lot of steps and a lot of room for human error.

As we are currently going through some major changes to the Empire server in 5.0, I needed to revisit our release flow and branching strategy, which in turn caused a big change to our workflows. This post will talk about how we use GitHub Actions for our builds and releases.

Pull Request CI 

We run quite a few things on pull requests to validate the changes and give us confidence in the new changes.

The first step in the workflow is linting the code. We run black, which checks formatting, and isort which checks the import sorting. Both tools are configured in the repository as pre-commit hooks, so they automatically run when a developer commits code. At some point, we will add in flake8, but first, we need to make all the old code flake8 compliant. 

If the pull request passes the code linting, we then run a series of tests. We have a pytest suite, which in 5.0 contains about 220 integration and unit tests to ensure all the API endpoints and some of the internal functionality hasn’t regressed. We run this test suite against Python 3.8, 3.9, and 3.10. On each of those Python versions, we run it once using SQLite and another time using MySQL (the new default in 5.0). 

Some pull request workflows are conditional because we don’t want to take up too much CI time on every pull request. If the pull request is a release (the branch is prefixed with release/), then we also run our test suite within the Empire docker image to ensure it still builds with everything it needs. We then use a tool called container-structure-test to check that container contains certain things like the expected version of Python, .NET, and PowerShell. It also checks for the existence of specific directories, such as the Invoke-Obfuscation directory. 

If the pull request modifies the install script, we run another set of container-structure-tests. These take a while (~45 minutes), so we don’t want to run them too often. It uses docker-compose to create docker images for each of our supported operating systems using the install script. It then uses container-structure-test to see if it properly installed the OS packages and Empire.

Release Flow 

Now the more complicated part… The release process. 

We maintain 3 “main” branches across two GitHub repositories. Why? We put out one version of Empire for Sponsors and one for Kali each from the private “Sponsors” repository. We put out a version of Empire to the public in the public Empire repository. The public release is also gated by a 30-day Kali/Sponsor exclusivity. For the most part, these different versions only contain subtle differences. But with Empire 5.0 packaging Starkiller as a submodule, we are guaranteed that each of those releases will at the least diverge there. 

Looking at options for a release process, we want it to be as automated as possible, as simple as possible, and allow for those diverging changes in the “main” branches without needing to manually cherry-pick everything. This calls for some above-average complexity in the release process. 

Commits flow from private-main to main, kali-main, and sponsors-main via git merge. A cherry-pick is used if there is ever a need to merge things back to private-main and to the others. Most code changes go to private-main and flow down to each of the releases.  

Let’s look at the path of a feature. A feature should be merged to a different branch depending on where we want it to end up. 

1. Only in Kali – We merge the branch to kali-main 
2. Only in Sponsors – We merge the branch to sponsors-main 
3. Only in Public – We merge the branch to the public repo’s main branch 
4. All 3 (The most likely scenario) – We merge to the public repo’s main branch and cherry-pick to private-main, or if one has access, we merge to the private repo’s private-main. From private-main, it gets merged to kali-main, sponsors-main, and main. 

Release CI/CD 

We have a few different things that happen during the release workflows. I will gloss over these things in the automation section, so I want to outline them here. 

When a release is prepared, we must do a few things. 

1. Update the changelog with the new version 
2. Update pyproject.toml with the new version 
3. Update the python code with the new version 
4. Update the Starkiller submodule (for >=5.0) 
5. Open a pull request with these changes 

We need to do a few things when that pull request gets merged. 

1. Cut a git tag and GitHub release 
2. Copy the release notes from the changelog into the GitHub release 
3. Release a docker image (if in the public repo) 

Release Automation 

We utilize a combination of manual trigger workflows and automatic workflows. 

Kali and Sponsors release

1. Merge private-main into kali-main and sponsors-main 

This is a sync that can be run at any time. We run this as much as possible to keep kali and sponsors in line with the changes in private-main, which is where most changes are merged. 

2. Run the private-release-start workflow 

This bumps the changelog, version, etc., described above and creates a pull request from release/x.y.z to private-main. That pull request runs the full test suite described in the Pull Request CI section. 

3. Someone hits merge 

Once all the CI is passing, and the changelog has been updated, someone hits merge. This triggers the private-release-tag workflow automatically, which will generate a tag and release for vx.y.z-private. This will come in handy later when we do our public release. 

4. Run the sponsors/kali release workflow 

This will branch from private-main and set the Starkiller version for each kali and sponsors (if necessary). It then opens a release/x.y.z-kali and release/x.y.z-sponsors pull request to each of them so the test suite can run. 

This can sometimes produce merge conflicts, especially in the CHANGELOG.md file. If that occurs, we can fix it in the release pull request. 

5. Someone hits merge 

When these two pull requests get merged, they automatically trigger the sponsor-kali-tag workflow, which generates tags and releases for vx.y.z-kali and vx.y.z-sponsors. 

Public Release

30 days have passed, and we’re ready to make the code public. 

1. Run the public release workflow 

This will merge the supplied tag from the sponsors repo into a release/x.y.z branch in the public repo and open a pull request that runs all the pull request CI. 

2. Someone hits merge 

The merge triggers an automatic workflow that generates a tag and release for vx.y.z . It will also kick off our docker build and release workflow.

What comes next?

These GitHub action workflows can be seen in the Empire repo in the github/directory. Starkiller’s release follows a similar workflow.

We have some ideas for potential enhancements written the in the release documentation to automate things further. Those docs also contain more detailed runbooks for doing the releases.

Empire 5.0 is around the corner. It is currently available via Kali’s experimental repo or through our GitHub sponsors repo. Expect a blog post in the coming month with more details.