Kubernetes

MongoDB switch-over from VM to AKS

  • Scalability Constraints:
8 min read
MongoDB switch-over from VM to AKS

What is Mongo DB?MongoDB, a popular open-source NoSQL database management system, distinguishes itself from traditional SQL databases by adopting a flexible approach to data storage. In SQL, databases are structured with tables and a rigid schema, while MongoDB utilizes collections similar to tables. However, MongoDB introduces a dynamic aspect with its document-oriented model, allowing nested data storage. This capability empowers users to establish intricate relationships within a single document, enhancing data retrieval and manipulation efficiency. Unlike SQL’s vertical scaling, MongoDB offers horizontal scaling, making it well-suited for dynamic, scalable, and evolving datasets. The emphasis on performance over strict consistency caters to scenarios where adaptability and speed are paramount, making MongoDB an ideal choice for managing modern, diverse data types.

2. Mongo DB Migration from VM to AKSMongoDB migration to Kubernetes entails several crucial steps to integrate the database seamlessly into the container orchestration environment. First and foremost, the MongoDB instance must be containerized, allowing it to run within a Kubernetes cluster. Deployment is facilitated through **Kubernetes **manifests, specifying the necessary configurations and resource requirements. Persistent storage solutions are configured to ensure data retention and durability across container restarts. Security considerations are addressed by managing secrets and configurations through tools like Vault, safeguarding sensitive information.

Service discovery mechanisms are also implemented to enable effective communication within the Kubernetes cluster. Monitoring and logging strategies are crucial for the migration process to optimize performance and maintain operational visibility. Lastly, scaling strategies are devised to accommodate varying workloads and ensure the efficient utilization of resources, contributing to the overall success of MongoDB integration within a Kubernetes environment.

3. Challenges with VM-based Deployment

  • Scalability Constraints:

The fixed nature of VMs restricts easy scalability. Increasing resources often requires downtime or complex scaling procedures. 

**Impact: **Inability to swiftly adjust resources in response to varying workloads or business demands, leading to potential performance bottlenecks during peak usage.

  • Resource Management Difficulties:

Allocating and managing resources across VMs can be complex. VMs might need to utilize resources more optimally.

**Impact: **

Inefficient resource utilization leads to underutilized or overburdened VMs, impacting performance and cost-efficiency.

  • Deployment Inflexibility:

Deploying updates or changes across VMs can be challenging and time-consuming.

**Impact: **

Slow deployment cycles, increased chances of errors during updates, and difficulties maintaining consistency across the environment.

4. Benefits of Migrating to Kubernetes

  • Enhanced Scalability:

Kubernetes allows for seamless horizontal scaling, enabling resources to be easily added or removed based on demand.

Impact:

MongoDB instances can dynamically adapt to varying workloads, ensuring optimal performance during peak usage without downtime.

  • Improved Resource Utilization:

Kubernetes efficiently manages resources by allocating them precisely where needed, avoiding underutilization or overprovisioning.

**Impact: **

Optimal resource usage minimizes costs and performance, enhancing overall database efficiency.

  • Flexible Deployment and Updates:

Kubernetes facilitates automated deployment and updates, reducing deployment cycles and complexities.

Impact:

Faster deployment processes enable quick iterations and updates, enhancing system agility and change responsiveness.

5. Mongo DB Migration from VM to AKS5.1 Review of Current Infrastructure

The current infrastructure consists of a MongoDB replica set with 3 replicas deployed across 3 VMS.

  • MongoDB version – 5.0
  • DB size – 50GB
  • Network – All VMs deployed under the same subnet in VPC alpha.

5.2 Preparation Stage

  • Networking Setup: Configure VPC peering to ensure connectivity between the VM and Kubernetes clusters. Validate and ensure seamless interconnectivity among all MongoDB pods and associated VMs to guarantee robust communication across the entire environment.
  • **Hostname Entries: **Establish and document hostname entries for all virtual machines (VMs) within each MongoDB pod. 
  • Replication key: Integrate and synchronize the replication key used in MongoDB VM replication with Vault
  • Backup Live Cluster: Before commencing migration, ensure a complete backup of the existing live MongoDB cluster is taken for data security and recovery purposes.
  • **External Traffic Policy: **Modify the external traffic policy to prioritize local traffic within the Kubernetes environment.

5.3 Migration Steps## 5.3.1 Pod Configuration

Set up MongoDB instances with hostnames in the new Kubernetes cluster and expose services as NodePort for external connectivity.

.fusion-syntax-highlighter-14 > .CodeMirror, .fusion-syntax-highlighter-14 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-14 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-14 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }Copy to ClipboardSyntax Highlighterapp: name: mongo code: MONGO notify: alerts@ecloudcontrol.com build: version: 5.0 env: PROD build_file: none output_files: output/*.zip image_template: mongo-5.0

deploy: context: demo/DEV resources: mongo-dev type: statefulset replicas: 3 port:

  • 27017

  • 9216 volumes:

  • claim: mongo-data mount: /appz/data name: mongo-data size: 100Gi mode: ReadWriteOnce matchLabels: false

  • claim: mongo-keys mount: /appz/keys name: mongo-keys size: 1Gi mode: ReadWriteOnce matchLabels: false

  • claim: mongo-backup mount: /appz/backup name: mongo-backup size: 200Gi mode: ReadWriteOnce matchLabels: false

properties: DB_USER_ADMIN: admin DB_USER_ADMIN_PASSWORD: vault: DB_USER_ADMIN_PASSWORD MONGO_USER_PASSWORD: vault: MONGO_USER_PASSWORD AUTH_KEY: vault: AUTH_KEY REPLICATION: 3

MONGODB_HOST: 127.0.0.1 MONGODB_PORT: 27017 MONGO_BACKUP_DAYS: 7 MONGO_BACKUP_TIME: 21 MASTER_ENABLE: 'no' revision: 32### 5.3.2 Cluster Integration Add the new MongoDB pods as slaves to the existing cluster. The replica-set can trigger a leader election in response to adding replica events; hence, we should configure the new members to be excluded from the replica-set leader election during the adding and synchronization stage by lowering the vote and priority for all new instances.

Example

.fusion-syntax-highlighter-15 > .CodeMirror, .fusion-syntax-highlighter-15 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-15 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-15 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }Copy to ClipboardSyntax Highlighterrs.add( { host: "10.0.0.9:31090", priority: 0, votes:0 } )Check the rs.status and wait for db sync to be completed.

Repeat this to add the other 2 instances.

5.3.3 Changing priority

.fusion-syntax-highlighter-16 > .CodeMirror, .fusion-syntax-highlighter-16 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-16 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-16 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }Copy to ClipboardSyntax Highlightercfg = rs.conf(); // Lower the priority of the old members cfg.members[0].priority = 10; cfg.members[1].priority = 0.5; cfg.members[2].priority = 0.5; // Restore the voting rights and priority of the new members cfg.members[3].votes = 1; cfg.members[3].priority = 1; cfg.members[4].votes = 1; cfg.members[4].priority = 1; cfg.members[5].votes = 1; cfg.members[5].priority = 1; // Save the changes rs.reconfig(cfg);### 5.3.4 Remove old secondaries .fusion-syntax-highlighter-17 > .CodeMirror, .fusion-syntax-highlighter-17 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-17 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-17 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }Copy to ClipboardSyntax Highlighterrs.remove("old instance-2") rs.remove("old instance-3")### 5.3.5 Switch primary .fusion-syntax-highlighter-18 > .CodeMirror, .fusion-syntax-highlighter-18 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-18 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-18 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }Copy to ClipboardSyntax Highlightercfg = rs.conf(); cfg.members[0].priority = 0.5; cfg.members[3].priority = 10; rs.config(cfg);### 5.3.6 Remove old primary .fusion-syntax-highlighter-19 > .CodeMirror, .fusion-syntax-highlighter-19 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-19 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-19 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }Copy to ClipboardSyntax Highlighterrs.remove("old instance-1")## 5.3.7 Change hostnames .fusion-syntax-highlighter-20 > .CodeMirror, .fusion-syntax-highlighter-20 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-20 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-20 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }Copy to ClipboardSyntax Highlightercfg = rs.conf() cfg.members[3].host = "mongo-5-0-0.mongo-5-0:27017" cfg.members[4].host = "mongo-5-0-1.mongo-5-0:27017" cfg.members[5].host = "mongo-5-0-2.mongo-5-0:27017" rs.reconfig(cfg)### 7. Challenges faced during Migration

  • After the primary switch, the application had an error ‘NotWritablePrimary’.

  • Replicated the issue locally and revised steps.

  • Scale down app-b to 0

  • Remove Secondary DB VMs from the cluster.

.fusion-syntax-highlighter-21 > .CodeMirror, .fusion-syntax-highlighter-21 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-21 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-21 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }Copy to ClipboardSyntax HighlighterRemove vm-db-1 Remove vm-db-23. Switch primary to k8s mongo-pod.

.fusion-syntax-highlighter-22 > .CodeMirror, .fusion-syntax-highlighter-22 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-22 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-22 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }Copy to ClipboardSyntax HighlighterDesignate mongo-5-0-1 as the new primary instance.4. Remove Primary DB VMs from the cluster.

.fusion-syntax-highlighter-23 > .CodeMirror, .fusion-syntax-highlighter-23 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-23 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-23 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }Copy to ClipboardSyntax HighlighterRemove vm-db-35. Update K8s Mongo secondary pods hostaddress from  NodeIP: NodePort to Podname.svcname

.fusion-syntax-highlighter-24 > .CodeMirror, .fusion-syntax-highlighter-24 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-24 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-24 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }Copy to ClipboardSyntax HighlighterUpdate 10.224.0.x:30xxx to mongo-5-0-0.mongo-5-0 Update 10.224.0.x:30xxx to mongo-5-0-1.mongo-5-0 Update 10.224.0.x:30xxx to mongo-5-0-2.mongo-5-0

  • Check the health of the Mongo replica set by capturing rs.conf and rs.status
  • Deploy ‘app-g’ with configuration pointing to the MongoDB pods by updating the ‘appz.yaml’ file.
  • Perform application testing.
  • Update the ‘app-svc’ service to reference ‘app-g’.
  • Execute a basic performance test using the live domain.*

Conclusion

Migrating MongoDB to a Kubernetes cluster represents a strategic leap toward improved scalability and operational efficiency. This integration streamlines deployment processes and optimizes resource utilization, which is especially beneficial for contemporary, data-intensive applications. By harnessing the dynamic orchestration capabilities of Kubernetes, MongoDB can adapt seamlessly to varying workloads, ensuring a more agile and responsive infrastructure. This synergy between MongoDB’s database strengths and Kubernetes’ flexible orchestration paves the way for a modern, efficient, and scalable solution for managing complex datasets.

ABOUT THE AUTHOR### Francis JoyCloud DevOps Engineer | Cloud Control

Cloud DevOps Engineer with more than 4 years of experience in supporting, automating, and optimizing deployments to hybrid cloud platforms using DevOps processes, tools, CI/CD, Docker containers, and K8s in both Production and Development environments.

CLOUD SOLUTIONS