Sometimes you may want to delete some metrics from Prometheus if those metrics are unwanted or you just need to free up some disk space.
Time series in Prometheus can be deleted over administrative HTTP API only (disabled by default).
To enabled it, pass --web.enable-admin-api
flag to Prometheus through start-up script or docker-compose file, depending on installation method.
Cool Tip: Install Prometheus using Docker on Ubuntu and CentOS! Read More →
Delete Time Series Metrics
Use the following syntax to delete all time series metrics that match some label:
$ curl -X POST \ -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={foo="bar"}'
To delete time series metrics that match some job
or instance
, run:
$ curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="node_exporter"}' $ curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="192.168.0.1:9100"}'
To delete all data from Prometheus, run:
$ curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~".+"}'Note that the above API calls don’t delete data immediately.
The actual data still exists on disk and will be cleaned up in future compaction.
To determine when to remove old data, use --storage.tsdb.retention
option e.g. --storage.tsdb.retention='365d'
(by default, Prometheus keeps data for 15 days).
To completely remove the data deleted by delete_series
send clean_tombstones
API call:
$ curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones'
Cool Tip: Install Node Exporter on Ubuntu and CentOS! Read More →
The successful exit status for the both delete_series
and clean_tombstones
is 204
.
Good post, one query regarding above article, if we completely remove time series using below call
curl -X POST -g ‘http://localhost:9090/api/v1/admin/tsdb/clean_tombstones
Does it result in high CPU spike?
Thanks! It really helped me to delete data from old switches.