HIVE-29492: Add AutoScaling to K8s operator

[ayushtkn]

What changes were proposed in this pull request?

Add auto scaling to Hive Operator

Why are the changes needed?

Better usage & cloud saving.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Manually

Installed Dependencies (ZK, Postgres & Ozone)

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install zookeeper bitnami/zookeeper \
  --set replicaCount=1 --set auth.enabled=false \
  --set image.repository=bitnamilegacy/zookeeper \
  --set image.tag=3.9.3-debian-12-r21 \
  --set global.security.allowInsecureImages=true --wait


helm install postgres bitnami/postgresql \
  --set auth.username=hive --set auth.password=hive123 \
  --set auth.database=metastore --wait


kubectl create secret generic hive-db-secret --from-literal=password=hive123


helm repo add ozone https://apache.github.io/ozone-helm-charts/
helm install ozone ozone/ozone --version 0.2.0 --wait
sleep 50
kubectl exec statefulset/ozone-om -- ozone sh volume create /s3v
kubectl exec statefulset/ozone-om -- ozone sh bucket create /s3v/hive

Started Hive Operator With AutoScaling Enabled (Very Low Thresholds for Testing)

helm install hive ./helm/hive-operator \
  --set cluster.database.type=postgres \
  --set cluster.database.url="jdbc:postgresql://postgres-postgresql:5432/metastore" \
  --set cluster.database.driver="org.postgresql.Driver" \
  --set cluster.database.username=hive \
  --set cluster.database.passwordSecretRef.name=hive-db-secret \
  --set cluster.database.passwordSecretRef.key=password \
  --set cluster.database.driverJarUrl="https://repo1.maven.org/maven2/org/postgresql/postgresql/42.7.5/postgresql-42.7.5.jar" \
  --set cluster.zookeeper.quorum="zookeeper:2181" \
  --set cluster.storage.coreSiteOverrides."fs\.defaultFS"="s3a://hive" \
  --set cluster.storage.coreSiteOverrides."fs\.s3a\.endpoint"="http://ozone-s3g-rest:9878" \
  --set-string cluster.storage.coreSiteOverrides."fs\.s3a\.path\.style\.access"=true \
  --set 'cluster.storage.envVars[0].name=HADOOP_OPTIONAL_TOOLS' \
  --set 'cluster.storage.envVars[0].value=hadoop-aws' \
  --set 'cluster.storage.envVars[1].name=AWS_ACCESS_KEY_ID' \
  --set 'cluster.storage.envVars[1].value=ozone' \
  --set 'cluster.storage.envVars[2].name=AWS_SECRET_ACCESS_KEY' \
  --set 'cluster.storage.envVars[2].value=ozone' \
  --set cluster.hiveServer2.autoscaling.enabled=true \
  --set cluster.metastore.autoscaling.enabled=true \
  --set cluster.llap.autoscaling.enabled=true \
  --set cluster.tezAm.autoscaling.enabled=true \
  --set-string cluster.llap.configOverrides."hive\.llap\.daemon\.task\.scheduler\.wait\.queue\.size"="1" \
  --set cluster.hiveServer2.autoscaling.scaleUpThreshold=1 \
  --set cluster.metastore.autoscaling.scaleUpThreshold=2

Launched Beeline

kubectl exec -it deployment/hive-hiveserver2 -- beeline -u "jdbc:hive2://hive-hiveserver2:10001/;transportMode=http;httpPath=cliservice"

OUTPUTS:

Initial Start -> Only 1 HMS, 1 HS2 (1 == Min Configured)

Hits First Beeline Session -> Tez AM, LLAP Daemons starts (Min 1 configured)

AutoScaling HS2 to 2 & Tez AM(Reduced max threshold)

Tez AM

HS2

Auto Scaling HMS & LLAP to 2

HMS

LLAP (Load reduced by the time, query finished :-( )

Scale Downs (After Cooling Periods)

Scheduled

Done (After waiting for cool down period for specific service)

[sonarqubecloud]

Quality Gate passed

Issues
103 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
4.9% Duplication on New Code

See analysis details on SonarQube Cloud