Prerequisites

Edit | Report

Overview

Before you install Sunbird in a server environment, make sure that you have the required permissions, servers, hardware, software, accounts and so on. Without these aspects in place, you may face delays that can best be avoided.

To efficiently handle Sunbird installation, you need to have:

  • System administrator permissions on Sunbird and all servers
  • Hands-on experience in administering Linux systems
  • Hands-on experience using Docker and Kubernetes to run containerized services

Note: Sunbird is tested on cloud hosted Linux servers (Azure & AWS). Hence, it is recommended that you use Linux servers. Sunbird and its functionality is not tested on Microsoft® operating systems, or on in-premise infrastructure, like rack mounted servers. The Sunbird installer has a known issue on virtual machines such as those created by VirtualBox or VMWare.

Provisioning Servers

Before you start the installation process, ensure that you provision for servers to host applications and set up required accounts and repositories as per details provided.

Application Server Count
Jenkins 4core 16G 250G HDD 1
KP 4core 16G 60G HDD 1
DP 4core 16G 60G HDD 1
DB 4core 16G 60G HDD 1
Yarn 4core 16G 60G HDD 2
Druid 4core 16G 60G HDD 1
Load Balancers - 2 (Optional)

List of Servers with their Ansible Group Name

</tr>
Module Servers Service Ansible Group Name
Build and Deploy Server-1 (Jenkins) Jenkins Master
Databases Server-2 (DB's) Cassandra cassandra-1, lp-cassandra, dp-cassandra, core-cassandra, cassandra-node-1, cassandra-ps, cassandra
Postgres postgresql-master-1, postgresql-slave-1, postgres
Application Elasticsearch es-1, composite-search-cluster, es-ps, core-es-1
Neo4j learning-neo4j-node1
Knowledge Platform Server-3 (KP Services and Kafka) Learning learning1, logstash-ps, learning
Redis redis1, lp-redis, redis, dp-redis, lms-redis
dial dial1
Kafka processing-cluster-kafka, processing-cluster-zookeepers, kafka-ps, kafka-1, ingestion-cluster-kafka
Data Pipeline Server-4 (DP Services) Spark spark
Analytics analytics-api, analytics-ps
Kafka Indexer kafka-indexer
InfluxDB influxdb
Docker Manager and Worker swarm-manager-1, swarm-agent-for-prometheus-1, swarm-agent-for-alertmanager-1, swarm-bootstrap-manager, swarm-node-1 , swarm-agent-dashboard-1, swarm-dashboard-1
Keycloak Keycloak
Logs Elasticsearch log-es-1
Secor secor, secor-ps
Yarn Server-5 (Yarn Master) Yarn Master Slave 1 yarn-master, yarn-slave, yarn-ps
Server-6 (Yarn Slave) Yarn Master Slave 2 yarn-master, yarn-slave, yarn-ps
Druid Server-7 druid servicer druid-postgres,raw-coordinator,raw-overlord,raw-broker,raw-historical,raw-middlemanager,raw-graphite,raw-zookeeper

Infra Requirements

1.k8s Cluster
2.Private GitHub repository to store Ansible hosts and secrets
3.Fully Qualified Domain Name (FQDN) with SSL
4.Azure Storage account
5.Docker hub account
6.A Public IP
7.Security:

  • All ports must be open in internal networks (Azure-Vnet or AWS-VPC) for internal comumnication between the VMs
  • By default, all the outbound ports are allowed for public access from the VM.

Creating the AKS cluster

Note Follow the steps given below to create the Kubernetes cluster in Azure. Refer to the documents provided by respective cloud providers to create the Kubernetes cluster on any other cloud. The AKS cluster and VM’s should be in same vnet. If they are in diffrent vnet, you have to peer the vnets. To peer the vnets the IP address of the two vnets should not overlap.

1.Use the following command to create the AKS cluster: (requires az cli and aks-preview)

      - create service principal and assign contributor role to service principal, get the secrets and client id of service principal. (https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli)
    
    - az aks create --resource-group <resouse-group-name> --node-resource-group <k8s-resource-group-name> --name <cluster name>  --node-count 4 --admin-username deployer --kubernetes-version 1.16.13 --service-principal "<service principal id>" --node-vm-size <vm size> --client-secret "<client id>" --network-plugin azure --ssh-key-value @deployer.pub -l <region> --vm-set-type VirtualMachineScaleSets --vnet-subnet-id /subscriptions/<subscription id>/resourceGroups/<resouse-group-name>/providers/Microsoft.Network/virtualNetworks/<vnet-name>/subnets/<subnet name>

    - command to get kube config file for created cluster:
       az aks get-credentials --resource-group <resource group name> --name <cluster name> --file - > k8s.yaml

## Configuring the Azure storage account

1.Update the CORS rule for the storage account as follows:

      Allowed Origins: *
    Allowed Methods: GET,HEAD,OPTIONS, PUT
    Allowed Headers: Access-Control-Allow-Method,Origin,x-ms-meta-qqfilename,x-ms-blob-type,x-ms-blob-content-type,Content-Type
    Exposed Headers: Access-Control-Allow-Origin,Access-Control-Allow-Methods
    Max Age: 200

2.Disable ‘Secure transfer required’ in storage account configuration

3.Create the following containers in Storage account with public ACL (dial, termsandcondtions, content)

Edit | Report