Percona Monitoring and Management (PMM) includes Grafana , which comes with the alerting feature.
We need to do alerting based on graphs (which don’t use templating).
Click on “Create New” in the Dashboards list to create a basic dashboard for your alerts:
Click on “Add Panel” and select “Graph”:
Click on the panel title of the related panel on the menu sign, and then click on “Panel JSON”.
Now you need to go back to the other browser window, and the dashboard with the graph you want to alert on.
Show the JSON panel for it. In this case, we go to “MySQL Overview” and show the JSON for “MySQL Connections” panel.
Copy the JSON from the “MySQL Connections” panel and paste it into the new panel in the dashboard created for alerting.
Panel is broken because we’re using templating variables in dashboard expressions. None of them are set up in this dashboard.
Expressions won’t work. We must replace the template variables in the formulas with actual hosts, instances, mount points, etc., for we want to alert on .
We need to change $host to the name of the host we want to alert on, and the $interval should align with the data capture interval (here we’ll set it to 5 seconds):
If correctly set up, you should see the graph showing the data.
Save the Dashboard with appropriate name.Finally, we can go to edit the graph. Click on the “Alert” and “Create Alert”.Specify Evaluate Every to create an alert.
This sets up the evaluation interval for the alert rule. Obviously, the more often the alert evaluates the condition, the more quickly you get alerted if something goes wrong (as well as alert conditions).
At this point, you should see the alert working. A little heart sign appears by the graph title, colored green (indicating it active) or red (indicating it is alerting).
Additionally, you will see the red and green vertical lines in the alert history.
These show when this alert gets triggered and when the system went back to normal.