Core monitoring
Rancher and Grafana do not send infrastructure alerts out of the box. Ask your installation administrator how alerts reach the responsible operator.
Open Rancher and Grafana. Use the same Core server and time range in each view.
A Core review moves from the services in Rancher, through the server, GPU and processing readings in Grafana, to the results in the Portal.
flowchart LR
subgraph RANCHER["In Rancher"]
A["Services<br/>ready"]
end
subgraph GRAFANA["In Grafana"]
B["CPU, memory<br/>and disk"]
C["Each GPU"]
D["Processing<br/>activity"]
end
subgraph PORTAL["In the Portal"]
E["Events and media<br/>shown in Monitor"]
end
A --> B --> C --> D --> ERun the Core review
In Rancher, a workload manages the pods that run a service. A pod contains one or more containers. Namespaces group resources, such as app and data.
Open this view | Inspect | Next action |
|---|---|---|
Rancher → Workloads | Ready pods, failed pods and increasing restart counts. | |
Grafana → Node Exporter / Nodes | Measured CPU use and available memory. | |
Node Exporter / Nodes | Free space and growth on the physical filesystem. | |
NVIDIA DCGM Exporter Dashboard | Utilization and used video memory for each GPU. | Read GPU use. |
Inference services | Request activity and processing duration. | |
IRIS+ Portal | Camera health, events and media. | Confirm that a representative event and its media arrived. |
A running pod can still be unready or return application errors.
On the Rancher Cluster Dashboard, CPU Reserved describes requested capacity. It does not measure actual CPU use.
To read server load in Grafana:
- Open Grafana → Dashboards → Node Exporter / Nodes.
- Select the Core server.
- Set a time range that includes typical peak activity.
- Read CPU use, available memory and free disk space.
Load guidance
Keep sustained CPU and each GPU's utilization below 80% during representative peaks. A brief spike alone does not show overload. This guidance leaves headroom but does not guarantee capacity.
Check memory and disk space separately, and keep headroom in both. An average can hide one busy server or GPU.
What the results mean
Result | What to do |
|---|---|
CPU and each GPU stay below 80% at peaks, memory and disk keep headroom, and results arrive | The Core has headroom. Keep your review schedule. |
CPU stays above 80% during peaks | Pause new load, then find the busy service. |
A GPU stays above 80% during peaks | Pause new load. To see which service is busy, use Confirm processing and results. |
A view shows No data | No data is not zero load, so the review is incomplete. Follow Missing measurements. |
Resources look normal, but events or media are missing | Start from Choose the first check. |
Read GPU use
Inference services shows average GPU use. NVIDIA DCGM Exporter Dashboard shows each GPU. Its readings come from NVIDIA Data Center GPU Manager, or DCGM.
Average GPU use
- Open Grafana → Dashboards → Inference services.
- Find Triton GPU and CPU utilization %.
Triton is the model server that runs models on the GPU. The yellow series averages GPU utilization. The green series shows host CPU use. An average can hide one busy GPU.
Each GPU
- Open NVIDIA DCGM Exporter Dashboard.
- Select the server in instance.
- Set the time range.
- Scroll below GPU SM Clocks.
- Read GPU Utilization for each GPU.
- Read GPU Framebuffer Mem Used for each GPU.
- Compare the readings during representative peaks.
Framebuffer memory is the GPU's video memory.
Several services can share one GPU. The legend then repeats that GPU's number, such as GPU 0, once per service. Treat these rows as one physical GPU, not as extra GPUs. Each row gets only some of the readings, so its line can break into dots. Read the rows together.
Loaded models keep their video memory after GPU activity falls. Used memory can therefore stay high while utilization is low.
Investigate unexpected memory growth or memory-allocation errors.
If instance shows None, the dashboard has no GPU readings. Failed collection can also produce zero readings. Before you accept low GPU use, check these points:
- The dashboard lists each GPU installed in the server.
- Inference services shows low processing activity for the same period.
If per-GPU readings are unavailable, the Core capacity check remains incomplete. Follow Missing measurements.
Confirm processing and results
The dashboard has no service selector. Use these rows and panels:
Row or panel | Shows |
|---|---|
Inference service | innovi-inference-service-2, the general inference service. |
Inference service pai 1 | innovi-inference-service-2-pai-1, an optional separate inference service. It shows activity only when your installation routes analytics to it. |
Triton image count per second | Model activity on the shared Triton service. |
Analytic names in the Portal can differ from dashboard legends. To learn which service and model an analytic uses, reach out to Irisity.
- Open Inference services.
- Expand the row for the service you are checking.
- Read its requests per second panel, such as innovi-inference-service-2 requests/second.
- Read its duration panel, such as Inference service request duration ms.
- Compare both with the GPU readings for the same time range.
Replica counts shows replicas and average requests in progress. These are requests a service handles at the same time, not a queue. A replica is one running copy of a service, not a GPU.
Investigate when requests in progress stay high while duration or errors rise.
Duration panel legends do not reliably name models. Use the request-rate and image-count panels to identify active models.
Both inference services can use the shared Triton service. A separate row does not mean another GPU.
Triton inference duration ms does not show the duration of each request. Use Inference service request duration ms for request duration.
Then check Portal → Monitor for an expected event and its required image or clip. Messaging, storage or indexing can delay results after processing.
Find the busy service
Use Kubernetes / Compute Resources / Namespace (Workloads) to compare CPU and memory for each service.
What it does | Services and jobs in Rancher | Main demand | What raises it |
|---|---|---|---|
Edge connections | Agent Proxy, Device Proxy | CPU, memory and network. | Connected Edges, camera-state updates and uploads. |
Model processing | Inference services, Triton | CPU, memory, GPU and video memory. | Enabled analytics, models and processing requests. |
Events and search | Event worker, indexer | CPU, memory, disk and network. | Events, media handling and indexing. |
Data and messaging | Kafka, Elasticsearch, databases | Memory, disk and network. | Stored data, messaging, searches and retention. |
Scheduled work | Report, backup and cleanup jobs | CPU and disk activity. | Scheduled runs and requested reports. |
Event bursts add media, indexing and delivery work. Searches and reports can compete with camera processing.
Camera counts alone do not establish capacity. Stream properties, analytics combinations, scene activity and hardware affect demand.
Core dashboards do not show Edge resources. Use Edge monitoring for Edges.
Add load in batches
- Record the current load and delivered results.
- Add a manageable batch of cameras or analytics.
- Wait until startup and model loading settle.
- Repeat the Core review during representative peak activity.
- Check the expected events and media.
No universal batch size or settling time exists.
Pause additions if CPU or GPU use stays high, processing slows, errors increase or measurements are missing. Resolve the limiting stage first.
If processing cannot keep up, reduce the added load with the controls in the application guide.
When to review
Review daily during the first week of operation. Then review weekly or monthly, as activity and operating needs require. Repeat the review after changes and during expected peaks.
Agree who performs these reviews while no infrastructure alerts are set up.
Include backup and cleanup checks in your maintenance routine.