Sidekiq Pod takes a long time to complete Init
Summary
The sidekiq pod needs to have the ability to scale up and down quickly (opinion). There are many ways for which we can configure the sidekiq definition for a GitLab install. One of the more interesting concepts is the ability to deploy a single sidekiq deployment PER QUEUE. Without getting into the complexity this creates, let's discuss the problems with the way the current sidekiq Pod setup prevents this from being successful and also impacts overall scale of sidekiq, regardless of deployment methods. The Pod has a dependencies
init container. This container performs a check related to the database schema. This procedure takes a really long time and creates contention around scaling quickly. Due to the failure to scale quickly, this leaves things in the queue to be delayed. This script takes roughly 50 seconds on an environment that is barely utilized. I ponder if this container should be removed from the sidekiq Pod and is instead ran as part of the migrations or upgrade Pod that runs.
With GitLab.com, we can always safely assume the database will be at the appropriate schema version as this is a step in our deployment model that occurs prior to any services being updated. Due to this, this init container does not provide value, and only slows down the ability to quickly scale sidekiq. Does this container provide value for other helm installations? Probably yes, but I don't believe it should be run prior to sidekiq starting.
We need this script to run prior to an upgrade of GitLab, is there a better place for it than the sidekiq Pod, perhaps the upgrade_check_hook
of helm?
Script in question: https://gitlab.com/gitlab-org/build/CNG/-/blob/master/gitlab-rails/scripts/wait-for-deps
Expected behavior
Sidekiq should start quickly, less than 50 seconds. Enough time for the configurations to be built and the sidekiq process started. Checking database schemas seems like something of a dependency that would prevent ANY pod from starting on a GitLab installation.
Versions
- Chart:
81eba953d0639970e82b0569913e6c814d49d710
- Platform:
- Cloud: GKE
- Kubernetes: (
kubectl version
)- Client:
1.14.7
- Server:
1.14.7
- Client:
- Helm: (
helm version
)- Client:
2.14.2
- Server:
n/a
- Client:
Relevant logs
Timestamps below supplemented from our logging mechanism.
% k logs gitlab-sidekiq-export-966444c8-gb778 dependencies
# Dec 18, 2019 @ 13:43:24.016
+ /scripts/set-config /var/opt/gitlab/templates /srv/gitlab/config
Begin parsing .erb files from /var/opt/gitlab/templates
Writing /srv/gitlab/config/resque.yml
Writing /srv/gitlab/config/sidekiq_queues.yml
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/database.yml
Copying other config files found in /var/opt/gitlab/templates
Copying smtp_settings.rb into /srv/gitlab/config
+ exec /scripts/wait-for-deps
Checking database connection and schema version
WARNING: This version of GitLab depends on gitlab-shell 10.2.0, but you're running Unknown. Please update gitlab-shell.
Database Schema - current: 20191122135327, codebase: 20191122135327
# Dec 18, 2019 @ 13:44:11.048