--- title: Core troubleshooting slug: enterprise/core-troubleshooting docTags: createdAt: 2026-10-15T14:23:17.276Z --- Start with the affected function and time. Compare the same period in Rancher, Grafana and the Portal. # Choose the first check | Symptom | First view | Next action | | ---------------------------------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------ | | The Portal and Rancher do not load | A session on the Core server. | Follow [Unresponsive Core](docId\:uXVDT1CZkhWjKsE9g6vl6). | | Grafana shows **Page not found** | Rancher. | Follow **Reopen Grafana** below. | | High CPU or memory pressure | **Node Exporter / Nodes**. | Open **Kubernetes / Compute Resources / Namespace (Workloads)** to compare services. | | Missing GPU or other readings | The affected Grafana panel. | Follow **Missing measurements** below. | | Growing processing delay | **Inference services**. | Compare requests, errors and duration with [GPU use](docId\:wqOLXuH2pfp9ig4HyjKG2). | | Delayed events or media | **Kafka Exporter** and event-worker logs. | Check whether consumer lag grows and delivery errors repeat. | | Delayed search results | **innovi-core-indexer** and **ElasticSearch** dashboards. | Inspect indexer errors and physical free disk space. | | Offline camera or Edge | **Portal → Administrator → Physical view**. | Separate one affected camera from an affected Edge as described below. | | Backup or cleanup failure | **Rancher → Workloads → CronJobs**. | Check the latest Job and logs in [Core maintenance](docId\:BxhH1ZaTSVqUYx7mQYT-S). | Consumer lag measures how far message processing trails incoming messages. A growing backlog needs investigation; it does not directly measure delay in seconds. # Unresponsive Core Use these checks when the Portal and Rancher do not load. They need an SSH session on the Core server with `sudo` rights, and they only read state. 1. Check free disk space with `df -h`. 2. Read the running kernel version with `uname -r`. 3. List the pods with `sudo kubectl get pods -A`. 4. Look for pods whose **STATUS** is not `Running` or `Completed`, or whose **READY** count is incomplete, such as `0/1`. A full root or `/mnt/irisplus-data` filesystem can stop services. Compare the kernel version with the one the [installation manual](docId:3xLEiGbYU0j0RQtRZQbmT) specifies. An operating system update can install a different kernel, which can stop the GPU driver from loading. Send the output to Irisity with the [support information](docId\:BxhH1ZaTSVqUYx7mQYT-S). # Reopen Grafana The **Target** link opens Grafana in a new browser tab, which can show **Page not found**. Grafana's own menu still works from this screen. ![](https://api.archbee.com/api/optimize/JzHznijrWxPLJLDUaV5A1/hxB9w2HmybX8YWMxkQdPw_live-2026-10-16-grafana-target-page-not-found.png "Grafana displays Page not found") If Grafana's menu lists **Dashboards**: 1. Select **Dashboards**. 2. Type the dashboard name in the search box. 3. Open the dashboard. If the menu lists only **Home** and **Administration**, sign in first: 1. Select **Administration**. 2. Select **Home** to open the Grafana sign-in form. 3. Sign in with your Grafana account. 4. Return to the Rancher tab with the Services list. 5. Select the Grafana **Target** link again. 6. Select **Dashboards** in Grafana's menu. After step 3, the browser can show the Rancher home page. Steps 4 and 5 bring you back to Grafana. Move between dashboards with Grafana's menu. A reloaded or bookmarked Grafana page opens a Rancher error page. Then start again from the **Target** link. If neither Rancher nor the Portal loads, follow [Unresponsive Core](docId\:uXVDT1CZkhWjKsE9g6vl6). If the **Target** link or Grafana's menu is unavailable, reach out to Irisity. # Missing measurements Use these checks when a dashboard shows **No data** or expected metrics are missing. 1. Check the panel's time range. 2. Check its server, service and model filters. 3. Check the selected data source. 4. Open **Rancher → Monitoring**. 5. Select **Prometheus Targets**, if available. 6. Read the collection error for any affected target marked **Down**. If access is denied, ask your installation administrator for permission. If **Monitoring** or **Prometheus Targets** is unavailable, or its link fails, reach out to Irisity. If collection succeeds but a supplied panel remains empty, reach out to Irisity with its dashboard name, filters, time range and error. # Locate the service and read logs The following names apply to the R2.1 package. Select the namespace before finding a workload. | Function | Namespace | Workload or pod | | --------------------------- | --------- | ----------------------------------------------------------------------------------------- | | Edge analytics connections | `app` | `innovi-core-agent-proxy` | | Edge management connections | `app` | `innovi-core-device-proxy` | | Model processing | `app` | `innovi-inference-service-2`, `innovi-inference-service-2-pai-1`, `innovi-triton-service` | | Events and media | `app` | `innovi-core-event-worker-service` | | Metadata and event indexing | `app` | `innovi-core-indexer` | | Messaging | `data` | Pods `kafka-kafka-0`, `kafka-kafka-1`, `kafka-kafka-2` | | Search storage | `data` | StatefulSets `elasticsearch-es-master`, `elasticsearch-es-data` | A StatefulSet manages service instances that keep their identities and storage. A Kafka consumer group contains service instances that share message processing. In Kafka, `event_service_events` is the event-worker consumer group. `event_indexer` and `metadata_indexer` belong to the indexer. To inspect an affected service: 1. Open **Rancher → Workloads**. 2. Select its namespace. 3. Open its workload. 4. Select an affected pod. 5. Inspect the pod's conditions and events. 6. Open the affected container's **Logs**. 7. Select previous-container logs if the container restarted. Check timestamps and repeated errors before changing or restarting anything. A cumulative restart count alone does not show a current failure. # Offline camera or Edge In **Portal → Administrator → Physical view**, select the affected camera or Edge. Review its connection information and **Health alerts**. - **One camera:** check its network connection, stream source and camera health alerts. - **Several cameras on one Edge:** check the Edge connection, network and local resources. Follow [Edge monitoring](docId\:nYIItacpKTFMoV4VmhEY2). - **Several Edges:** check shared network paths and the Core connection services listed above. Pause load increases while persistent faults remain. Use the [support information](docId\:BxhH1ZaTSVqUYx7mQYT-S) if you need help resolving the fault.