Русский
Русский
English
Статистика
Реклама

Как восстановить Sentry после неудачного обновления

Всем привет. Я хочу рассказать о том, как проходило восстановление Sentry после неудачного обновления.

Что же такое Sentry?

Это система полного отслеживания ошибок с открытым исходным кодом, которая поддерживает широкий спектр серверных, браузерных, настольных и родных мобильных языков и сред, включая PHP, Node. js, Python, Ruby, C #, Java, Go, React, Angular, Vue, JavaScript и другие.

Немного о том, как просто и непринуждённо всё поломалось.

Жили мы на версии 9.0.0 и пришло время обновиться. Пощупав web интерфейс 10.0.0, принял решение обновиться на неё. Это была самая большая ошибка!

Обновление прошло в штатном режиме. Сначала переход на 9.1.2 затем, на 10.0.0 (иначе нельзя). Далее обнаружил, что не запускается worker. Причиной этому добавление новых приложений и зависимостей.

Попытки собрать приложения и засунуть их в k8s не увенчались успехом.

Было принято решение откатиться назад. Но тут меня ждало разочарование. После миграции на 10 версию, часть данных переезжает в clickhouse и удаляется из Postgres.

После 3 часов работ и привлечения DBA эксперта данные были восстановлены до рабочего состояния.

Как?

Сделали рядом инстанс PG, закинули туда схему от версии 9.0.0 и начали потабличное восстановление. Конечно, без ошибок и напильника тут не обошлось.

На этом история не заканчивается. Пришло время всё же обновиться на версию 9.1.2.

Собрал образ, запускаю:

sentry upgrade

и видим вот такое чудесное сообщение:

 ! I'm not trusting myself; either fix this yourself by fiddling ! with the south_migrationhistory table, or pass --delete-ghost-migrations ! to South to have it delete ALL of these records (this may not be good).Exception in thread Thread-1 (most likely raised during interpreter shutdown)

После долгих поисков, проб и ошибок было найдено решение.

Необходимо восстановить схему данных текущей версии.

Для этого клонируем репозиторий:

git clone https://github.com/getsentry/onpremise.git

Переходим в ветку 9.1.2, так как ветки 9.0.0 уже нет.

cd onpremise && git checkout tags/9.1.2

Далее правим образ в Dockerfile:

#cat Dockerfile ARG SENTRY_IMAGEFROM ${SENTRY_IMAGE:-sentry:9.0.0}-onbuild
Затем правим compose file

docker-compose.yml
# NOTE: This docker-compose.yml is meant to be just an example of how# you could accomplish this on your own. It is not intended to work in# all use-cases and must be adapted to fit your needs. This is merely# a guideline.# See docs.getsentry.com/on-premise/server/ for full# instructionsversion: '3.4'x-defaults: &defaults  restart: unless-stopped  build:    context: .  depends_on:    - redis   # - postgres    - memcached    - smtp  env_file: .env  environment:    SENTRY_MEMCACHED_HOST: memcached    SENTRY_REDIS_HOST: redis    SENTRY_POSTGRES_HOST: 'sentry.cl.ats'    SENTRY_DB_USER: 'sentryDB'    SENTRY_DB_NAME: 'sentry_user'    SENTRY_DB_PASSWORD: 'sentry_passwd'    SENTRY_POSTGRES_PORT: 5432    SENTRY_EMAIL_HOST: smtp  volumes:    - sentry-data:/var/lib/sentry/filesservices:  smtp:    restart: unless-stopped    image: tianon/exim4  memcached:    restart: unless-stopped    image: memcached:1.5-alpine  redis:    restart: unless-stopped    image: redis:3.2-alpine  #postgres:  #  restart: unless-stopped  #  image: postgres:9.5  #  environment:  #    POSTGRES_HOST_AUTH_METHOD: 'trust'  #  volumes:  #    - sentry-postgres:/var/lib/postgresql/data  web:    <<: *defaults    ports:      - '9000:9000'  cron:    <<: *defaults    command: run cron  worker:    <<: *defaults    command: run workervolumes:    sentry-data:      external: true    #sentry-postgres:    #  external: true


Немного пояснений. Так как у меня есть отдельный инстанс базы данных, то создание через compose пропускаю, также не нужен volume sentry-postgres.

После данных манипуляций запускаем сборку:

./install.sh 

В процессе может выскочить ошибка, о которой я упомянул выше. Не обращаем внимания и после того, как процесс закончится, запускаем следующую команду:

docker-compose run --rm web sentry django migrate --delete-ghost-migrations

Примерный вывод
Creating network "onpremise_default" with the default driverCreating onpremise_memcached_1 ... doneCreating onpremise_smtp_1      ... doneCreating onpremise_redis_1     ... done21:30:26 [INFO] sentry.plugins.github: apps-not-configuredRunning migrations for sentry:- Nothing to migrate. - Loading initial data for sentry.Installed 0 object(s) from 0 fixture(s)Running migrations for nodestore:- Nothing to migrate. - Loading initial data for nodestore.Installed 0 object(s) from 0 fixture(s)Running migrations for search:- Nothing to migrate. - Loading initial data for search.Installed 0 object(s) from 0 fixture(s)Running migrations for social_auth:- Nothing to migrate. - Loading initial data for social_auth.Installed 0 object(s) from 0 fixture(s)Running migrations for tagstore:- Nothing to migrate. - Loading initial data for tagstore.Installed 0 object(s) from 0 fixture(s)Running migrations for jira_ac:- Nothing to migrate. - Loading initial data for jira_ac.Installed 0 object(s) from 0 fixture(s)Running migrations for hipchat_ac:- Nothing to migrate. - Loading initial data for hipchat_ac.Installed 0 object(s) from 0 fixture(s)


Далее можно запускать upgrade.

docker-compose run --rm web sentry upgrade

Stdout
Starting onpremise_smtp_1      ... doneStarting onpremise_memcached_1 ... doneStarting onpremise_redis_1     ... done21:30:48 [INFO] sentry.plugins.github: apps-not-configuredSyncing...Creating tables ...Installing custom SQL ...Installing indexes ...Installed 0 object(s) from 0 fixture(s)Synced: > django.contrib.admin > django.contrib.auth > django.contrib.contenttypes > django.contrib.messages > django.contrib.sessions > django.contrib.sites > django.contrib.staticfiles > crispy_forms > debug_toolbar > raven.contrib.django.raven_compat > rest_framework > sentry.plugins.sentry_interface_types > sentry.plugins.sentry_mail > sentry.plugins.sentry_urls > sentry.plugins.sentry_useragents > sentry.plugins.sentry_webhooks > sudo > south > sentry_plugins.slackNot synced (use migrations): - sentry - sentry.nodestore - sentry.search - social_auth - sentry.tagstore - sentry_plugins.jira_ac - sentry_plugins.hipchat_ac(use ./manage.py migrate to migrate these)Running migrations for sentry:- Nothing to migrate. - Loading initial data for sentry.Installed 0 object(s) from 0 fixture(s)Running migrations for nodestore:- Nothing to migrate. - Loading initial data for nodestore.Installed 0 object(s) from 0 fixture(s)Running migrations for search:- Nothing to migrate. - Loading initial data for search.Installed 0 object(s) from 0 fixture(s)Running migrations for social_auth:- Nothing to migrate. - Loading initial data for social_auth.Installed 0 object(s) from 0 fixture(s)Running migrations for tagstore:- Nothing to migrate. - Loading initial data for tagstore.Installed 0 object(s) from 0 fixture(s)Running migrations for jira_ac:- Nothing to migrate. - Loading initial data for jira_ac.Installed 0 object(s) from 0 fixture(s)Running migrations for hipchat_ac:- Nothing to migrate. - Loading initial data for hipchat_ac.Installed 0 object(s) from 0 fixture(s)Creating missing DSNsCorrecting Group.num_comments counter


Итак, схема восстановлена. Можно обновиться на 9.1.2.

Для этого в Dockerfile меняем версию на 9.1.2:

ARG SENTRY_IMAGEFROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild
И запускаем install.sh

./install.sh

Если всё сделано верно, то процесс завершится без ошибок. Вывод должен быть примерно такой:

Stdout
# ./install.sh Checking minimum requirements...Creating volumes for persistent storage...Created sentry-data..env already exists, skipped creation.Building and tagging Docker images...smtp uses an image, skippingmemcached uses an image, skippingredis uses an image, skippingBuilding webStep 1/2 : ARG SENTRY_IMAGEStep 2/2 : FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild# Executing 4 build triggers ---> Running in 6c97f9fcaf63DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-supportRemoving intermediate container 6c97f9fcaf63 ---> Running in 9e0f65ee3af6Removing intermediate container 9e0f65ee3af6 ---> Running in 09754c44319cRemoving intermediate container 09754c44319c ---> a12fa31c2675Successfully built a12fa31c2675Successfully tagged onpremise_web:latestBuilding cronStep 1/2 : ARG SENTRY_IMAGEStep 2/2 : FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild# Executing 4 build triggers ---> Using cache ---> Using cache ---> Using cache ---> Using cache ---> a12fa31c2675Successfully built a12fa31c2675Successfully tagged onpremise_cron:latestBuilding workerStep 1/2 : ARG SENTRY_IMAGEStep 2/2 : FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild# Executing 4 build triggers ---> Using cache ---> Using cache ---> Using cache ---> Using cache ---> a12fa31c2675Successfully built a12fa31c2675Successfully tagged onpremise_worker:latestDocker images built.Generating secret key...Secret key written to .envSetting up database...Starting onpremise_smtp_1      ... doneStarting onpremise_redis_1     ... doneStarting onpremise_memcached_1 ... done21:35:07 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.21:35:10 [INFO] sentry.plugins.github: apps-not-configuredSyncing...Creating tables ...Installing custom SQL ...Installing indexes ...Installed 0 object(s) from 0 fixture(s)Migrating...Running migrations for sentry: - Migrating forwards to 0472_auto__add_field_sentryapp_author. > sentry:0424_auto__add_field_integration_status > sentry:0425_auto__add_index_pullrequest_organization_id_merge_commit_sha > sentry:0425_remove_invalid_github_idps > sentry:0426_auto__add_sentryappinstallation__add_sentryapp__add_field_user_is_sent > sentry:0427_auto__add_eventattachment__add_unique_eventattachment_project_id_event > sentry:0428_auto__add_index_eventattachment_project_id_date_added > sentry:0429_auto__add_integrationexternalproject__add_unique_integrationexternalpr > sentry:0430_auto__add_field_organizationintegration_status > sentry:0431_auto__add_field_externalissue_metadata > sentry:0432_auto__add_field_relay_is_internal > sentry:0432_auto__add_index_userreport_date_added__add_index_eventattachment_date_ > sentry:0433_auto__add_field_relay_is_internal__add_field_userip_country_code__add_ > sentry:0434_auto__add_discoversavedqueryproject__add_unique_discoversavedqueryproj > sentry:0435_auto__add_field_discoversavedquery_created_by > sentry:0436_rename_projectdsymfile_to_projectdebugfile > sentry:0437_auto__add_field_sentryapp_status > sentry:0438_auto__add_index_sentryapp_status__chg_field_sentryapp_proxy_user__chg_ > sentry:0439_auto__chg_field_sentryapp_owner > sentry:0440_auto__del_unique_projectdebugfile_project_debug_id__add_index_projectd > sentry:0441_auto__add_field_projectdebugfile_data > sentry:0442_auto__add_projectcficachefile__add_unique_projectcficachefile_project_ > sentry:0443_auto__add_field_organizationmember_token_expires_at > sentry:0443_auto__del_dsymapp__del_unique_dsymapp_project_platform_app_id__del_ver > sentry:0444_auto__add_sentryappavatar__add_field_sentryapp_redirect_url__add_field > sentry:0445_auto__add_promptsactivity__add_unique_promptsactivity_user_feature_org > sentry:0446_auto__add_index_promptsactivity_project_id > sentry:0447_auto__del_field_promptsactivity_organization__add_field_promptsactivit > sentry:0448_auto__add_field_sentryapp_is_alertable > sentry:0449_auto__chg_field_release_owner > sentry:0450_auto__del_grouphashtombstone__del_unique_grouphashtombstone_project_ha > sentry:0451_auto__del_field_projectbookmark_project_id__add_field_projectbookmark_ > sentry:0452_auto__add_field_sentryapp_events > sentry:0452_auto__del_field_releaseenvironment_organization_id__del_field_releasee > sentry:0453_auto__add_index_releasefile_release_name > sentry:0454_resolve_duplicate_0452 > sentry:0455_auto__add_field_groupenvironment_first_seen > sentry:0456_auto__add_dashboard__add_unique_dashboard_organization_title__add_widg > sentry:0457_auto__add_field_savedsearch_is_global__chg_field_savedsearch_project__ > sentry:0457_auto__add_monitorcheckin__add_monitor__add_index_monitor_type_next_che > sentry:0458_global_searches_data_migrationSaved Searchs: 100% |########################################################################################################################################################################| Time: 0:00:07 > sentry:0459_global_searches_unique_constraint > sentry:0460_auto__add_field_servicehook_organization_id > sentry:0461_event_attachment_indexes > sentry:0462_auto__add_servicehookproject > sentry:0462_releaseenvironment_project_id > sentry:0463_backfill_service_hook_project > sentry:0464_auto__add_sentryappcomponent__add_field_sentryapp_schema > sentry:0464_groupenvironment_foreignkeys > sentry:0465_sync > sentry:0466_auto__add_platformexternalissue__add_unique_platformexternalissue_grou > sentry:0467_backfill_integration_status > sentry:0468_auto__add_field_projectdebugfile_code_id__add_index_projectdebugfile_p > sentry:0468_recent_search > sentry:0469_fix_state > sentry:0470_org_saved_search > sentry:0471_global_saved_search_types > sentry:0472_auto__add_field_sentryapp_authorThe following content types are stale and need to be deleted:    sentry | dsymobject    sentry | dsymapp    sentry | useridentity    sentry | dsymsymbol    sentry | dsymsdk    sentry | globaldsymfile    sentry | versiondsymfile    sentry | projectdsymfile    sentry | grouphashtombstone    sentry | minidumpfile    sentry | dsymbundleAny objects related to these content types by a foreign key will alsobe deleted. Are you sure you want to delete these content types?If you're unsure, answer 'no'.    Type 'yes' to continue, or 'no' to cancel: yesRunning migrations for sentry.nodestore:- Nothing to migrate.Running migrations for sentry.search:- Nothing to migrate.Running migrations for social_auth:- Nothing to migrate.Running migrations for sentry.tagstore:- Nothing to migrate.Running migrations for sentry_plugins.hipchat_ac:- Nothing to migrate.Running migrations for sentry_plugins.jira_ac:- Nothing to migrate.Synced: > django.contrib.admin > django.contrib.auth > django.contrib.contenttypes > django.contrib.messages > django.contrib.sessions > django.contrib.sites > django.contrib.staticfiles > crispy_forms > debug_toolbar > rest_framework > sentry.plugins.sentry_interface_types > sentry.plugins.sentry_mail > sentry.plugins.sentry_urls > sentry.plugins.sentry_useragents > sentry.plugins.sentry_webhooks > sudo > southMigrated: - sentry - sentry.nodestore - sentry.search - social_auth - sentry.tagstore - sentry_plugins.hipchat_ac - sentry_plugins.jira_acCreating missing DSNsCorrecting Group.num_comments counterCleaning up...----------------You're all done! Run the following command get Sentry running:  docker-compose up -d


Всё, версия 9.1.2 установлена.

Выводы.

Это всем известные прописные истины. Не спешите, делайте резервные копии и проверяйте всё несколько раз.

Благодарю за внимание!
Источник: habr.com
К списку статей
Опубликовано: 23.07.2020 00:16:58
0

Сейчас читают

Комментариев (0)
Имя
Электронная почта

Devops

Тестирование веб-сервисов

Sentry

Backup

Категории

Последние комментарии

© 2006-2020, personeltest.ru