πŸ“œ
Our Manifesto
🧰
Backup & Disaster Recovery
πŸ‘¨β€πŸ‘©β€πŸ‘§β€πŸ‘¦ Customer Success & Supporty
πŸ‘¨β€πŸ‘©β€πŸ‘§β€πŸ‘¦ Community
Powered By GitBook
Increase Cassandra PVC
Issue: Client's Cassandra PVC (10Gi) is filled which has caused Atlas and related components to go down, need to resize the PVC but since Cassandra is a stateful set we can't edit its size directly.
Solutions: Resize the PVC, delete the stateful set then release a new statefulset with the new size all the data will be retained.

Prerequisites

Kubectle access would be requried Release portal access would be required The product must be on latest release ( β‰₯ v1.69.38-stable)

Step 1: Edit the Cassandra PVC

    1.
    Get all the PVCs for Cassandra as per below command (should be 3 or more) .
1
kubectl get pvc -n atlas | grep cassandra
2
​
3
# Output we are looking for
4
​
5
data-atlas-cassandra-0
6
data-atlas-cassandra-1
7
...
Copied!
    1.
    Increase the [spec.resources.requests.storage](http://spec.resources.requests.storage) to desired value let's say 20Gi (keep a note of the exact value and the unit)
    kubectl edit pvc data-atlas-cassandra-0 -n atlas
cassandra-output
Repeat this step for all the PVCs that we got in the last output so that all the Cassandra PVCs are covered.
    1.
    Verify the changes by describing each of the PVC
    kubectl describe pvc data-atlas-cassandra-0 -n atlas
    repeat this for all the pvcs
    ​
    ​

Step 2: Delete the Cassandra Statefulset

    Get the Statefulsets
1
kubectl get sts -n atlas | grep cassandra
2
# output
3
atlas-cassandra
Copied!
    Get the Statefulsets
1
kubectl get sts -n atlas | grep cassandra
2
# output
3
atlas-cassandra
Copied!

Step 3: Make config changes on the release portal.

    Go to the release portal of Atlan
    Switch to the Config tab
    Scroll down to the Atlas Configuration and Check the Atlas Custom Compute Configs option
snapshotrelease
    Fill in the exact same value in Cassandra Persistent Volume Size with units that were used while editing the PVCs in one of the previous steps.
cassendravolume
    Lastly, scroll down to the end of the page and save your changes.
    Go to the latest release and deploy the changes.
πŸ₯³ The PVC for Cassandra has been successfully updated.

Step 4: Troubleshooting

If things go south try these:
    Check if all Atlas pods are online.
    Check if all the Cassandra pods are online.
1
kubectl get sts -n atlas
2
# output
3
NAME READY AGE
4
atlas 2/2 21d
5
atlas-cassandra 3/3 61m
6
atlas-elasticsearch-master 3/3 21d
7
logstash-logstash 1/1 21d
8
zookeeper 3/3 21d
9
​
10
# the ready state fraction should be 1
Copied!
Note the total size of atlas statefulset
If things get stuck for long time you can try scaling down atlas and scaling it back up again:
1
# scale down
2
kubectl scale statefulsets atlas --replicas=0 -n atlas
3
# confirm it scaled down
4
kubectl get sts atlas -n atlas
5
# scale back up to N the original statefulset size
6
kubectl scale statefulsets atlas --replicas=N -n atlas
Copied!
Last modified 2d ago