Solr 10 Upgrade¶

Apache Solr is upgraded from 9.10.1 to 10.0.0. This page covers the prerequisites, what to expect during the data migration, how to monitor it, how to roll back, and how to recover from a failure.

The whole migration process will happen automatically. There are no manual steps required. During migration, older audit records fill in gradually and can be missing for a while until the copy finishes. No data is lost.

Solr is upgraded to 10.0.0 to resolve critical CVEs (Common Vulnerabilities and Exposures) on the Solr 9.10.1 image and its bundled Jetty.

Solr 10 is the default, for both a fresh install and an upgrade. To stay on Solr 9, set in config/custom-vars/vars.solr.yml:

YAML
SOLR_V10_ENABLE: "false"

Throughout this page, <namespace> is your DEPLOYMENT_ENV_NAME.

For the full list of Solr 10 variables and their defaults, see Solr 10 configuration variables.

1. Prerequisites¶

For both fresh installs and upgrades¶

Solr 10 installs with your existing Solr settings. Cluster size, memory, CPU, storage class and volume size all come from the SOLR_K8S_* variables you already use, so there is nothing to set.

The following need attention before you upgrade:

If the Solr ingress is enabled, the external hostname changes. It is derived from the Solr service name, so it moves from solr-service-<namespace>.<domain> to solr10-service-<namespace>.<domain>, and its TLS secret changes from solr-tls-secret to solr10-tls-secret. Update firewall or security-group rules, external monitoring, and any scripts or tools that query Solr directly.

Additional prerequisites for an upgrade¶

Capacity for a second Solr cluster¶

Both clusters run at the same time until the data copy finishes, so plan for another SOLR_K8S_CLUSTER_SIZE Solr pods:

`DEPLOYMENT_SIZE`	Extra CPU per pod	Extra memory per pod
SMALL	1	~1.8 Gi
MEDIUM	4	~9.6 Gi
LARGE	8	~38.4 Gi

A three-pod MEDIUM deployment needs roughly 12 more CPU and 29 Gi more memory available for the duration of the copy.

Sizing Solr 10 independently¶

Solr 10 takes its size from Solr 9 by default, so a deployment that sets nothing gets the cluster config it already runs. Each of these vars overrides Solr 10 alone, leaving the Solr 9 cluster exactly as it is:

Variable	Defaults to
`SOLR10_K8S_CLUSTER_SIZE`	`SOLR_K8S_CLUSTER_SIZE`
`SOLR10_HEAP_MIN_MEMORY_MB` / `SOLR10_HEAP_MAX_MEMORY_MB`	the Solr 9 heap for your `DEPLOYMENT_SIZE`
`SOLR10_K8S_CPU_REQUESTS` / `SOLR10_K8S_CPU_LIMITS`	`SOLR_K8S_CPU_REQUESTS` / `SOLR_K8S_CPU_LIMITS`
`SOLR10_K8S_PVC_STORAGE_SIZE`	`SOLR_K8S_PVC_STORAGE_SIZE`

What happens on upgrade¶

A Solr 10 cluster starts alongside your running Solr 9 cluster.
All collections are created on Solr 10. The installer waits for this, because no Privacera service can start without them.
Every service — Ranger, Portal, Audit Server, Discovery, PolicySync, PEG, and the Spark, Trino and EMR plugins — is repointed at Solr 10 and starts writing there.
A migration job is submitted and the installer finishes without waiting for it. The copy continues in the background.
When the copy is verified, the migration job scales Solr 9 to zero and retains its volumes.

Warning

The installer finishing does not mean the data copy is done. See Monitoring the data migration.

Skipping the data copy¶

With SOLR_MIGRATION_SKIP: "true", Solr 10 starts empty and takes new writes, and Solr 9 is retired immediately. Nothing is deleted — its volumes are retained and it can be scaled back up to read the old documents.

4. Monitoring the data migration¶

The migration job copies every document from Solr 9 to Solr 10. It runs after the installer finishes, so your deployment is already live on Solr 10 while it works.

What you will see in the UI meanwhile¶

This is expected and temporary:

Audits, Alerts, Discovery, Lineage and Scan screens show what has been copied so far, plus everything written since the switchover. Older records appear as the copy progresses.
No data is lost. New records are written to Solr 10 and are unaffected. Historical records stay on Solr 9 until the copy is verified.

Checking progress¶

Bash
kubectl -n <namespace> logs -f job/solr-resource-migration

Progress is reported per batch, with the collection's position in the list and a running percentage:

Text Only
1	`[1/22] ranger_audits: posted 5000 docs 245000/442789 (55%)`

To check whether it is still running:

Bash
kubectl -n <namespace> get job solr-resource-migration

Confirming it finished¶

The log ends with all migrate=yes collections are migrated and count-verified, followed by retired statefulsets/solr. Then:

Bash
kubectl -n <namespace> get statefulset solr solr10

Solr 9 at 0/0 and Solr 10 at full replicas are the finished state. Solr 9 is retired only on a verified copy, so if it is still running, the copy has not been verified yet.

5. Recovering from a failure¶

Every failure path leaves your data intact. Solr 9 is retired only after a verified copy, so a failed migration means Solr 9 is still running with every document.

The installer stopped at `solr-collection-manager`¶

The collections this job creates are a prerequisite, so nothing after it was installed. Your services are still pointed at Solr 9 and serving normally.

Bash
kubectl -n <namespace> logs job/solr-collection-manager
kubectl -n <namespace> get pods -l app=solr10-app

The usual cause is Solr 10 pods that cannot be scheduled. If they are Pending, check the pod events for insufficient CPU or memory, add capacity, and re-run the upgrade.

The migration job failed¶

Read the end of the log, change the setting it points to, then restart the job.

Bash
kubectl -n <namespace> logs job/solr-resource-migration --tail=50

What the log or pod status says	What it means	What to do
`still_absent=N` with `the source grew by N document(s) during the run`	Something is still writing to Solr 9	Find and stop the writer, then restart the job
`still_absent=N` with no growth message	Some documents did not copy	Restart the job — it retries only what is missing
`an id contains the terms separator`	Your document IDs contain a `\\|` character	Set `SOLR_MIGRATION_ID_SEPARATOR` to a character your IDs do not use, then re-run the upgrade
`does not exist on the destination`	Collections were not created	Re-run the upgrade (`./privacera-manager.sh setup && ./pm_with_helm.sh upgrade`) — it recreates the collections and starts a fresh migration job
Pod `OOMKilled`	Not enough memory for a very large collection	Raise `SOLR_MIGRATION_K8S_MEM_LIMITS`, and `SOLR_MIGRATION_SORT_BUFFER` to no more than a quarter of it
Pod evicted for ephemeral storage	Scratch space filled	Raise `SOLR_MIGRATION_K8S_WORK_SIZE_LIMIT`

Restarting the migration¶

The job is safe to restart at any point. It compares document IDs between the two clusters and copies only what is missing, so a restart resumes rather than starting over, and documents your services have already written to Solr 10 are left alone.

Bash
kubectl -n <namespace> delete job solr-resource-migration
kubectl -n <namespace> apply -f ~/privacera/privacera-manager/output/kubernetes/helm/solr-migration/templates/

If you changed any setting, re-run the upgrade instead so the new value is picked up.

6. Rolling back to Solr 9¶

Roll back in two steps:

In config/custom-vars/vars.solr.yml, set:
YAML
1
SOLR_V10_ENABLE: "false"
Re-run the upgrade. Solr 9 comes back up from its retained volumes, every service is repointed at it, and Solr 10 is scaled to zero with its own volumes retained.

If you mirror images into your own registry, keep the bdc8a02 (Solr 9) tag available for as long as you may want to roll back.

Warning

Documents written to Solr 10 after the switchover do not come back with a rollback. They remain in the retained Solr 10 index. The longer you run on Solr 10, the more a rollback costs.

Reading old Solr 9 data without rolling back¶

Scale Solr 9 up by hand. Its documents are untouched and it takes no traffic:

Bash
kubectl -n <namespace> scale statefulset solr --replicas=1

Collection shards and replicas¶

Collection layout is set from Privacera Manager rather than by the application that owns each collection. Geometry cascades from a global value, to a Discovery-collections group, to per-collection overrides. For the variable names, collections and defaults, see Collection geometry.

A fresh install uses these values as set. An upgrade preserves the geometry your Solr 9 collections already have, so the globals do not change existing collections. After the upgrade:

Raising a replica count works. Set the variable and re-run the upgrade. You need at least that many Solr pods for the new replicas to be placed on.
Lowering a replica count is not applied. The installer logs a warning and leaves the collection unchanged.
Changing a shard count is not applied. Shard count is fixed when a collection is created, and changing it needs a reindex. Contact Privacera Support.

Monitoring¶

Solr 10 removed the standalone Prometheus exporter, so the solr-exporter deployment is no longer installed and is scaled to zero along with Solr 9. Solr metrics are served by Solr itself and scraped from the Solr pods at /solr/admin/metrics. Repoint any custom scrape configuration or dashboard that targets solr-exporter.

Prev topic: Upgrading Privacera