Build and Deploy
Contents[hide] |
Overview
This page details out the jobs required to be run as part of the upgrade from Sunbird and Vidaydaan release 4.3.0 to release 4.4.0. Use the following table to understand the jobs that need to be executed in order to successfully complete the upgrade. Any jenkins job configuration or pre-requisites mentioned under manual configuration section needs to be done first before running any of the mentioned jobs. The order of the jobs should also be run as shown below. They can be run in parallel to speed up the execution.
Variables for Sunbird
| Variable Name | Service Name | Comments |
|---|---|---|
| sunbird_default_file_size | Player | Update in Core/common.yml The default value is 150 |
| Add new ACL to Vidyadaan consumer | OnboardConsumers | Add under VDN consumer grouprefreshTokenCreate |
| uci_api_admin_token | transformer, uci-api | Update in Core/Common.yml The value can be generated using https://passwordsgenerator.net/ |
| hasura_graphql_admin_secret | transformer, uci-api | Update in Core/Common.yml The value can be generated using https://passwordsgenerator.net/ |
| fusionauth_service_admin_key | transformer, uci-api | Update in Core/Common.yml The value can be generated using https://passwordsgenerator.net/ |
| uci_encryption_key_base64 | transformer, uci-api | Update in Core/Common.yml The value can be generated using https://passwordsgenerator.net/ |
| uci_encryption_key_string | transformer, uci-api | Update in Core/Common.yml The value can be generated using https://passwordsgenerator.net/ |
| uci_odk_user | okd | Update in Core/Common.yml administrator |
| uci_odk_password | okd | Update in Core/Common.yml The value can be generated using https://passwordsgenerator.net/ |
| uci_kafka_topic_create | UCI | Update in Core/Common.yml true |
| uci_go_to_start_char | transformer | Update in Core/Common.yml 0 |
| uci_one_level_up_char | transformer | Update in Core/Common.yml 99 |
Build and Deploy for Sunbird
| Service to be Build | Build Tag | Service to Deploy | Deploy Tag | Comments |
|---|---|---|---|---|
| Build/Plugins/ContentPlayer | release-4.4.0_RC1 | Deploy/Plugins/ContentPlayer | release-4.4.0_RC4 | |
| Build/Plugins/ContentPlugins | release-4.4.0_RC5 | Deploy/Plugins/ContentPlugins | release-4.4.0_RC4 | |
| Deploy/Core/KafkaSetup | release-4.4.0_RC4 | |||
| Deploy/Kubernetes/Keycloak | release-4.4.0_RC4 | |||
| Provision/Core/ESMapping | release-4.4.0_RC4 | |||
| Build/Mobile/Sunbird-IONIC-Mobile-App | release-4.4.0_RC22 | |||
| Build/Core/OfflineInstaller | release-4.4.0_RC32 | Deploy/Core/OfflineInstaller | release-4.4.0_RC4 | |
| Build/Core/Cassandra | release-4.4.0_RC2 | Deploy/Kubernetes/Cassandra | release-4.4.0_RC4 | |
| Build/Kubernetes/Assessment | release-4.4.0_RC3 | Deploy/Kubernetes/Assessment | release-4.4.0_RC5 | |
| Deploy/Kubernetes/Content | release-4.4.0_RC3 | Deploy/Kubernetes/Content | release-4.4.0_RC5 | |
| Build/Kubernetes/DiscussionsMW | release-4.4.0_RC4 | Deploy/Kubernetes/DiscussionsMW | release-4.4.0_RC4 | |
| Build/Kubernetes/Groups | release-4.4.0_RC4 | Deploy/Kubernetes/Groups | release-4.4.0_RC4 | |
| Build/Kubernetes/Learner | release-4.4.0_RC11 | Deploy/Kubernetes/Learner | release-4.4.0_RC4 | |
| Build/Kubernetes/Nodebb | release-4.4.0_RC3 nodebb_version: v1.16.0 |
Deploy/Kubernetes/Nodebb | release-4.4.0_RC4 | |
| Build/Kubernetes/Notification | release-4.4.0_RC5 | Deploy/Kubernetes/Notification | release-4.4.0_RC4 | |
| Build/Kubernetes/Player | release-4.4.0_RC32 | Deploy/Kubernetes/Player | release-4.4.0_RC5 | |
| Build/Kubernetes/Search | release-4.4.0_RC2 | Deploy/Kubernetes/Search | release-4.4.0_RC5 | |
| Build/Kubernetes/Taxonomy | release-4.4.0_RC2 | Deploy/Kubernetes/Taxonomy | release-4.4.0_RC5 | |
| Deploy/managed-learn/ml-analytics-service | release-4.4.0_RC4 ml_analytics_version:release-4.4.0_RC1 |
|||
| Build/managed-learn/ml-core-service | release-4.4.0_RC8 | Deploy/managed-learn/ml-core-service | release-4.4.0_RC4 | |
| Build/managed-learn/ml-projects-service | release-4.4.0_RC9 | Deploy/managed-learn/ml-projects-service | release-4.4.0_RC4 | |
| Build/managed-learn/ml-reports-service | release-4.4.0_RC4 | Deploy/managed-learn/ml-reports-service | release-4.4.0_RC4 | |
| Build/managed-learn/ml-survey-service | release-4.4.0_RC4 | Deploy/managed-learn/ml-survey-service | release-4.4.0_RC4 | |
| Build/UCI/Utils | release-4.4.0_RC1 | From here some build jobs needs to be run in sequencesequence-1 |
||
| Build/UCI/message-rosa | release-4.4.0_RC1 | sequence-2 |
||
| Build/UCI/dao | release-4.4.0_RC1 | sequence-3 |
||
| Build/UCI/Adapter | release-4.4.0_RC1 | sequence-4 |
||
| Build/UCI/Inbound | release-4.4.0_RC1 | Deploy/UCI/Inbound | release-4.4.0_RC4 | |
| Build/UCI/odk | release-4.4.0_RC1 | Deploy/UCI/odk | release-4.4.0_RC4 | |
| Build/UCI/Outbound | release-4.4.0_RC1 | Deploy/UCI/Outbound | release-4.4.0_RC4 | |
| Build/UCI/Orchestrator | release-4.4.0_RC1 | Deploy/UCI/Orchestrator | release-4.4.0_RC4 | |
| Build/UCI/Transformer | release-4.4.0_RC1 | Deploy/UCI/Transformer | release-4.4.0_RC4 | |
| Build/UCI/UCI | release-4.4.0_RC1 | Deploy/UCI/UCI | release-4.4.0_RC4 | |
| Deploy/UCI/fusionauth | release-4.4.0_RC4 | |||
| Deploy/UCI/gql | release-4.4.0_RC4 | |||
| Deploy/Kubernetes/LoggingFileBeatsVM | release-4.4.0_RC4 | |||
| Deploy/Kubernetes/Logging | release-4.4.0_RC4 | |||
| Deploy/Kubernetes/OnboardAPIs | release-4.4.0_RC4 | |||
| Deploy/Kubernetes/OnboardConsumers | release-4.4.0_RC4 | |||
| OpsAdministration/Core/GraylogMongoImport | release-4.4.0_RC4 | |||
| Build/KnowledgePlatform/FlinkJobs | release-4.4.0_RC3 | Deploy/KnowledgePlatform/FlinkJobs | release-4.4.0_RC5 | Deploy all flink jobs |
| Build/KnowledgePlatform/Learning | release-4.4.0_RC5 | Deploy/KnowledgePlatform/Learning | release-4.4.0_RC5 | |
| Build/KnowledgePlatform/Yarn | release-4.4.0_RC5 | Deploy/KnowledgePlatform/Yarn | release-4.4.0_RC5 | |
| BuildDataPipeline/AnalyticsCore | release-4.4.0_RC2 | Deploy/DataPipeline/AnalyticsCore | release-4.4.0_RC2 | |
| Build/DataPipeline/CoreDataProducts | release-4.4.0_RC1 | Deploy/DataPipeline/CoreDataProducts | release-4.4.0_RC2 | |
| Build/DataPipeline/EdDataProducts | release-4.4.0_RC3 | Deploy/DataPipeline/EdDataProducts | release-4.4.0_RC2 |
Manual Configurations for Sunbird
| Manual Step | Instruction |
|---|---|
| User Feed Migration | Run manually with the help of documentaion in spark machine SB-26190 : UserFeed Migration Script |
| Adding UCI folder in private repo | Create UCI folder under specific env and create symlinks from core folder |
| Create jenkins jobs related to UCI | Create Build, ArtifactUpload and Deploy jobs for UCI services from below Github Link |
| Create postgres DB’s for UCI services | Change username based on envcreate database "uci-botdb"; GRANT ALL PRIVILEGES ON DATABASE "uci-botdb" to sunbirdstaging; GRANT ALL PRIVILEGES ON DATABASE "uci-botdb" to "sunbirdstaging@sunbirdstaging-postgres"; create database "uci-formsdb"; GRANT ALL PRIVILEGES ON DATABASE "uci-formsdb" to sunbirdstaging; GRANT ALL PRIVILEGES ON DATABASE "uci-formsdb" to "sunbirdstaging@sunbirdstaging-postgres"; create database "uci-fusionauth"; GRANT ALL PRIVILEGES ON DATABASE "uci-fusionauth" to sunbirdstaging; GRANT ALL PRIVILEGES ON DATABASE "uci-fusionauth" to "sunbirdstaging@sunbirdstaging-postgres"; create database "uci-odk-aggregate"; GRANT ALL PRIVILEGES ON DATABASE "uci-odk-aggregate" to sunbirdstaging; GRANT ALL PRIVILEGES ON DATABASE "uci-odk-aggregate" to "sunbirdstaging@sunbirdstaging-postgres"; |
| Create cassandra keyspace for UCI | Use below command to create keyspace CREATE KEYSPACE _uci_store |
| Settingup FusionAuth after UCI service deployments | Setting up Fusionauth and adding API Key kubectl get services --namespace=<env> | grep fusionauth-frontend Go to the Public IP above http://<IP>:9011/admin/setup-wizard Setup Fusionauth with the following fields. First Name - UCI-Staging Last Name - Sunbird Email - <respective email> Pass: generate using https://passwordsgenerator.net/ Go to the http://<IP>:9011/admin/api-key/add and add the API Key - by creating a new API Key. |
update existing variable in core/common.yml, do not add new variable enable_tenant_config |
“*” : It will allow for all tenants current value is "OFF" we are disabling it in the current release so use OFF value in preprod and prod |
Truncate portal.sessions cassandra table |
TRUNCATE portal.sessions; Remove the backup / snapshot folder on every node after truncate Example: rm -rf /var/lib/cassandra/data/portal/sessions-9811ae90915711eb85b423bdb45bf1b8/snapshots/truncated-1635236404732-sessions |
| Cassandra data migration from userProfileType to userProfileTypes | Follow steps mentioned in below link SB-26822 |
| ES reindexing for user is required to be updated | Jenkins job : Provision/job/dev/job/Core/job/ESMapping/ provide indices_name : userv3 In es_files options choose : indices, mappings, pipelines Follow mentioned steps SB-26823 |
| Move ESMapping job from OpsAdministration folder to Provision Folder | Github Link |
| Update ESMapping jenkins job config from this PR | Github Link |
| System Setting update to support new sms DLT templateId as per this tkt SB-27415 | Execute command mentioned in Link |
| SB-26424 : Data migration for existing users - Consent status to be updated for sso users with multiple externalids” | Execute steps mentioned in SB-26424 |
| Add new option for Logging VM job | Check below PR https://github.com/project-sunbird/sunbird-devops/pull/2996/files |
| Upgrade postgres to version 11 in staging | Follow below steps for upgrade https://project-sunbird.atlassian.net/wiki/spaces/DevOps/pages/3008462870/Generic+Postgres+Upgrade+Document |
Note: The below jobs are applicable only if you are running Vidyadaan infrastructure
Variables for Vidyadaan
| Variable Name | Service Name | Comments |
|---|---|---|
| dock_default_file_size | sourcing portal | Default Value: 150 |
| dock_default_video_size | sourcing portal | Default Value: 15000 |
| Add new ACL to Vidyadaan consumer | refreshTokenCreate | This has to be done on sunbird environment |
| sunbird_api_auth_token | Grant refersh token API access to bearer mapped for this variable (https:///auth/v1/refresh/token) |
|
| use_sunbird_kong_token | Default Value is true, which disables the feature. Please make it true to enable new auth token structure. Ref |
|
| sunbird_domain_name | sourcing portal | eg: staging.sunbirded.org |
Build and Deploy for Vidayadaan
| Service to be Build | Build Tag | Service to Deploy | Deploy Tag | Comments |
|---|---|---|---|---|
| Build/Plugins/Content-plugins | release-4.4.0_RC3 | Deploy/Plugins/Content-plugins | release-4.4.0-vdn | |
| Deploy/Dock/Kubernetes/OnboardConsumers | release-4.4.0-vdn | |||
| Deploy/Kubernetes/OnboardAPIs | release-4.4.0-vdn | |||
| Deploy/Kubernetes/UploadSchemas | release-4.4.0-vdnkp_branch_or_tag: release-4.4.0_RC53 |
|||
| Deploy/Kubernetes/Upload_CollectionHierarchy_CSV | release-4.4.0_RC4 | |||
| Deploy/KnowledgePlatform/KafkaSetup | release-4.4.0_RC5 | |||
| Build/Dock/Assessment | release-4.4.0_RC2 | Deploy/Kubernetes/Assessment | release-4.4.0-vdn | |
| Build/Dock/Content | release-4.4.0_RC2 | Deploy/Kubernetes/Content | release-4.4.0-vdn | |
| Build/Dock/Player | release-4.4.0_RC53 | Deploy/Kubernetes/Player | release-4.4.0-vdn | |
| Build/Dock/Program | release-4.4.0_RC5 | Deploy/Kubernetes/Program | release-4.4.0-vdn | |
| Build/Dock/Search | release-4.4.0_RC2 | Deploy/Kubernetes/Search | release-4.4.0-vdn | |
| Build/Dock/Taxonomy | release-4.4.0_RC2 | Deploy/Kubernetes/Taxonomy | release-4.4.0-vdn | |
| Build/Dock/KnowledgePlatform/FlinkJobs | release-4.4.0_RC3 | Deploy/KnowledgePlatform/FlinkJobs | release-4.4.0_RC5 | Deploy all flink jobs |
| Build/Dock/KnowledgePlatform/Learning | release-4.4.0_RC5 | Deploy/KnowledgePlatform/Learning | release-4.4.0_RC5 | |
| Build/Dock/KnowledgePlatform/Yarn | release-4.4.0_RC5 | Deploy/KnowledgePlatform/Yarn | release-4.4.0_RC5 | |
| Deploy/Kubernetes/PostgresqlMigration | release-4.4.0_RC5 |
Manual Configurations for Vidyadaan
| Manual Step | Instruction |
|---|---|
| Create Upload_CollectionHierarchy_CSV jenkins job | PR Link |
| Upgrade postgres to version 11 in vdn staging | Follow below steps Link |
| Please update the Question Paper collection category definition in Dock staging Category Definition details |
Ref: https://project-sunbird.atlassian.net/browse/SB-26908 |
| Category defination updates in VDN | Ref: https://project-sunbird.atlassian.net/wiki/spaces/SingleSource/pages/3014164481/Content+NOT+created+under+a+target+collection |