Prometheus: Delete Time Series Metrics

Sometimes you may want to delete some metrics from Prometheus if those metrics are unwanted or you just need to free up some disk space.

Time series in Prometheus can be deleted over administrative HTTP API only (disabled by default).

To enabled it, pass --web.enable-admin-api flag to Prometheus through start-up script or docker-compose file, depending on installation method.

Cool Tip: Install Prometheus using Docker on Ubuntu and CentOS! Read More →

Delete Time Series Metrics

Use the following syntax to delete all time series metrics that match some label:

$ curl -X POST \
    -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={foo="bar"}'

To delete time series metrics that match some job or instance, run:

$ curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="node_exporter"}'
$ curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance=""}'

To delete all data from Prometheus, run:

$ curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~".+"}'

Note that the above API calls don’t delete data immediately.

The actual data still exists on disk and will be cleaned up in future compaction.

To determine when to remove old data, use --storage.tsdb.retention option e.g. --storage.tsdb.retention='365d' (by default, Prometheus keeps data for 15 days).

To completely remove the data deleted by delete_series send clean_tombstones API call:

$ curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones'

Cool Tip: Install Node Exporter on Ubuntu and CentOS! Read More →

The successful exit status for the both delete_series and clean_tombstones is 204.

2 Replies to “Prometheus: Delete Time Series Metrics”

  1. Mohammed Allauddin says: Reply

    Good post, one query regarding above article, if we completely remove time series using below call
    curl -X POST -g ‘http://localhost:9090/api/v1/admin/tsdb/clean_tombstones
    Does it result in high CPU spike?

  2. Thanks! It really helped me to delete data from old switches.

Leave a Reply