Document histograms and histogram_quantile.

content/docs/concepts/metric_types.md

@@ -24,7 +24,7 @@ tasks completed, errors occurred, etc. Counters should not be used to expose
 current counts of items whose number can also go down, e.g. the number of
 currently running goroutines. Use gauges for this use case.
 
-See the client library usage documentation for counters:
+Client library usage documentation for counters:
 
    * [Go](http://godoc.org/github.com/prometheus/client_golang/prometheus#Counter)
    * [Java](https://github.com/prometheus/client_java/blob/master/client/src/main/java/io/prometheus/client/metrics/Counter.java)
@@ -41,7 +41,7 @@ Gauges are typically used for measured values like temperatures or current
 memory usage, but also "counts" that can go up and down, like the number of
 running goroutines.
 
-See the client library usage documentation for gauges:
+Client library usage documentation for gauges:
 
    * [Go](http://godoc.org/github.com/prometheus/client_golang/prometheus#Gauge)
    * [Java](https://github.com/prometheus/client_java/blob/master/client/src/main/java/io/prometheus/client/metrics/Gauge.java)
@@ -49,26 +49,51 @@ See the client library usage documentation for gauges:
    * [Ruby](https://github.com/prometheus/client_ruby#gauge)
    * [Python](https://github.com/prometheus/client_python#gauge)
 
-## Summaries
+## Histogram
 
-A _summary_ samples observations (usually things like request durations) over
-sliding windows of time and provides instantaneous insight into their
-distributions, frequencies, and sums.
+A _histogram_ samples observations (usually things like request durations or
+response sizes) and counts them in configurable buckets. It also provides a sum
+of all observed values.
+
+A histogram with a base metric name of `<basename>` exposes multiple time series
+during a scrape:
+
+  * cumulative counters for the observation buckets, exposed as `<basename>_bucket{le="<upper inclusive bound>"}`
+  * the **total sum** of all observed values, exposed as `<basename>_sum`
+  * the **count** of events that have been observed, exposed as `<basename>_count` (identical to `<basename>_bucket{le="+Inf"}` above)
+
+Use the [`histogram_quantile()`
+function](/docs/querying/functions/#histogram_quantile()) to calculate quantiles
+from histograms or even aggregations of histograms. A histogram is also suitable
+to calculate an [Apdex score](http://en.wikipedia.org/wiki/Apdex). See [summary
+vs. histogram](/docs/practices/instrumentation/#summary-vs.-histogram) for
+details of histogram usage and differences to [summaries](#summary).
+
+Client library usage documentation for summaries:
+
+   * [Go](http://godoc.org/github.com/prometheus/client_golang/prometheus#Histogram)
+   * [Java](https://github.com/prometheus/client_java/blob/master/simpleclient/src/main/java/io/prometheus/client/Histogram.java) (histograms are only supported by the simple client but not by the legacy client)
+   * [Python](https://github.com/prometheus/client_python#histogram)
+
+## Summary
+
+Similar to a _histogram_, a _summary_ samples observations (usually things like
+request durations and response sizes). While it also provides a total count of
+observation and a sum of all observed values, it calculates configurable
+quantiles over a sliding time window.
 
 A summary with a base metric name of `<basename>` exposes multiple time series
 during a scrape:
 
-  * streaming **quantile values** of observed events, exposed as `<basename>{quantile="<quantile label>"}`
+  * streaming **φ-quantiles** (0 ≤ φ ≤ 1) of observed events, exposed as `<basename>{quantile="<φ>"}`
   * the **total sum** of all observed values, exposed as `<basename>_sum`
   * the **count** of events that have been observed, exposed as `<basename>_count`
 
-This is quite convenient, for if you are interested in tracking latencies of an
-operation in real time, you get three types of information reported for free
-with one metric.
-
-A typical use-case is the observation of request latencies or response sizes.
+See [summary
+vs. histogram](/docs/practices/instrumentation/#summary-vs.-histogram) for
+details of summary usage and differences to [histograms](#histogram).
 
-See the client library usage documentation for summaries:
+Client library usage documentation for summaries:
 
    * [Go](http://godoc.org/github.com/prometheus/client_golang/prometheus#Summary)
    * [Java](https://github.com/prometheus/client_java/blob/master/client/src/main/java/io/prometheus/client/metrics/Summary.java)

content/docs/introduction/roadmap.md

@@ -23,17 +23,6 @@ detailed local views.
 
 GitHub issue: [#9](https://github.com/prometheus/prometheus/issues/9)
 
-**Aggregatable histograms**
-
-The current client-side [summary
-types](/docs/concepts/metric_types/#summaries) do not
-support aggregation of quantiles. For example, it is [statistically
-incorrect](http://latencytipoftheday.blogspot.de/2014/06/latencytipoftheday-you-cant-average.html)
-to average over the 90th percentile latency of multiple monitored instances.
-We plan to implement server-side histograms which will allow for this use case.
-
-GitHub issue: [#480](https://github.com/prometheus/prometheus/issues/480)
-
 **More flexible label matching in binary operations**
 
 [Binary operations](/docs/querying/operators/) between time series vectors

content/docs/practices/instrumentation.md

@@ -144,9 +144,9 @@ gauge for how long the collection took in seconds and another for the number of
 errors encountered.
 
 This is one of the two cases when it is okay to export a duration as a gauge
-rather than a summary, the other being batch job durations. This is because both
-represent information about that particular push/scrape, rather than
-tracking multiple durations over time.
+rather than a summary or a histogram, the other being batch job durations. This
+is because both represent information about that particular push/scrape, rather
+than tracking multiple durations over time.
 
 ## Things to watch out for
 
@@ -191,10 +191,10 @@ processing system.
 If you are unsure, start with no labels and add more
 labels over time as concrete use cases arise.
 
-### Counter vs. gauge vs. summary
+### Counter vs. gauge
 
-It is important to know which of the three main metric types to use for a given
-metric. There is a simple rule of thumb: if the value can go down, it's a gauge.
+To pick between counter and gauge, there is a simple rule of thumb: if
+the value can go down, it's a gauge.
 
 Counters can only go up (and reset, such as when a process restarts). They are
 useful for accumulating the number of events, or the amount of something at
@@ -206,11 +206,55 @@ Gauges can be set, go up, and go down. They are useful for snapshots of state,
 such as in-progress requests, free/total memory, or temperature. You should
 never take a `rate()` of a gauge.
 
-Summaries are similar to having two counters. They track the number of events
-*and* the amount of something for each event, allowing you to calculate the
-average amount per event (useful for latency, for example). In addition,
-summaries can also export quantiles of the amounts, but note that [quantiles are not
-aggregatable](http://latencytipoftheday.blogspot.de/2014/06/latencytipoftheday-you-cant-average.html).
+### Summary vs. histogram
+
+Summaries and histograms are more complex metric types. They both sample
+observations. They track the number of observations *and* the sum of the
+observed values, allowing you to calculate the average observed value (useful
+for latency, for example). Note that the number of observations (showing up in
+Prometheus as a time series with a `_count` suffix) is inherently a counter (as
+described above, it only goes up), while the sum of observations (showing up as
+a time series with a `_sum` suffix) is inherently a gauge (if a negative value
+is observed, it goes down).
+
+The essential difference is that summaries calculate streaming φ-quantiles on
+the client side and expose them, while histograms count observations in buckets
+and expose those counts. Calculation of quantiles from histograms happens on the
+server side using the [`histogram_quantile()`
+function](/docs/querying/functions/#histogram_quantile()).
+
+Both approaches have specific advantages and disadvantages:
+
+|   | Histogram | Summary
+|---|-----------|---------
+| Configuration | Need to configure buckets suitable for the expected range of observed values. | Need to configure φ-quantiles and sliding window, other φ-quantiles and sliding windows cannot be calculated later.
+| Client performance | Observations are very cheap as they only need to increment counters. | Observations are expensive due to the streaming quantile calculation.
+| Server performance | Calculating quantiles is expensive, consider [recording rules](/docs/querying/rules/#recording-rules) as a remedy. | Very low resource needs.
+| Number of time series | Low for Apdex score (see below), very high for accurate quantiles. Each bucket creates a time series. | Low, one time series per configured quantile.
+| Accuracy | Depends on number and layout of buckets. Higher accuracy requires more time series. | Configurable. Higher accuracy requires more client resources but is relatively cheap.
+| Specification of φ-quantile and sliding time window | Ad-hoc in Prometheus expressions. | Preconfigured by the client.
+| Aggregation | Ad-hoc aggregation with [Prometheus expressions](/docs/querying/functions/#histogram_quantile()). | In general [not aggregatable](http://latencytipoftheday.blogspot.de/2014/06/latencytipoftheday-you-cant-average.html).
+
+Note the importance of the last item in the table. Let's say you run a service
+with an SLA to respond to 95% of requests in under 200ms. In that case, you will
+probably collect request durations from every single instance in your fleet, and
+then you want to aggregate everything into an overall 95th percentile. You can
+only do that with histograms, but not with summaries. Aggregating the
+precomputed quantiles from a summary rarely makes sense.
+
+A histogram is suitable to calculate the [Apdex
+score](http://en.wikipedia.org/wiki/Apdex). Configure a bucket with the target
+request duration as upper bound and another bucket with 4 times the request
+duration as upper bound. Example: The target request duration is 250ms. The
+tolerable request duration is 1s. The request duration are collected with a
+histogram called `http_request_duration_seconds`. The following expression
+yields the Apdex score:
+
+```
+(
+  http_request_duration_seconds_bucket{le="0.25"} + http_request_duration_seconds_bucket{le="1"}
+) / 2 / http_request_duration_seconds_count
+```
 
 ### Timestamps, not time since
 

content/docs/querying/functions.md

@@ -28,6 +28,12 @@ the 1-element output vector from the input vector:
 This is useful for alerting on when no time series
 exist for a given metric name and label combination.
 
+## `bottomk()`
+
+`bottomk(k integer, v instant-vector` returns the `k` smallest elements of `v`
+by sample value.
+
+
 ## `ceil()`
 
 `ceil(v instant-vector)` rounds the sample values of all elements in `v` up to
@@ -80,6 +86,55 @@ and value across all series in the input vector.
 `floor(v instant-vector)` rounds the sample values of all elements in `v` down
 to the nearest integer.
 
+## `histogram_quantile()`
+
+`histogram_quantile(φ float, b instant-vector)` calculates the φ-quantile (0 ≤ φ
+≤ 1) from the buckets `b` of a histogram. The samples in `b` are the counts of
+observations in each bucket. Each value must have a label `le` where the label
+value denotes the inclusive upper bound of the bucket. (Samples without such a
+label are ignored.) The [histogram metric
+type](/docs/concepts/metric_types/#histogram) automatically
+provides time series with the `_bucket` suffix and the appropriate labels.
+
+Use the `rate()` function to specify the time window for the quantile
+calculation.
+
+Example: A histogram metric is called `http_request_duration_seconds`. To
+calculate the 90th percentile of request durations over the last 10m, use the
+following expression:
+
+```
+histogram_quantile(0.9, rate(http_request_duration_seconds_bucket[10m]))
+```
+
+The quantile is calculated for each label combination in
+`http_request_duration_seconds`. To aggregate, use the `sum()` aggregator
+outside of the `rate()` function. Since the `le` label is required by
+`histogram_quantile()`, it has to be included in the `by` clause. The following
+expression aggregates quantiles by `job`:
+
+```
+histogram_quantile(0.9, sum(rate(http_request_duration_seconds_bucket[10m])) by (job, le))
+```
+
+To aggregate everything, specify only the `le` label:
+
+```
+histogram_quantile(0.9, sum(rate(http_request_duration_seconds_bucket[10m])) by (le))
+```
+
+The `histogram_quantile()` interpolates quantile values by assuming a linear
+distribution within a bucket. The highest bucket must have an upper bound of
+`+Inf`. (Otherwise, `NaN` is returned.) If a quantile is located in the highest
+bucket, the upper bound of the second highest bucket is returned. A lower limit
+of the lowest bucket is assumed to be 0 if the upper bound of that bucket is
+greater than 0. In that case, linar interpolation is applied within that bucket
+as usual. Otherwise, the upper bound of the lowest bucket is returned for
+quantiles located in the lowest bucket.
+
+If `b` contains fewer than two buckets, `NaN` is returned. For φ < 0, `-Inf` is
+returned. For φ > 1, `+Inf` is returned.
+
 ## `rate()`
 
 `rate(v range-vector)` calculate the per-second average rate of increase of the
@@ -123,6 +178,11 @@ Same as `sort`, but sorts in descending order.
 this does not actually return the current time, but the time at which the
 expression is to be evaluated.
 
+## `topk()`
+
+`topk(k integer, v instant-vector)` returns the `k` largest elements of `v` by
+sample value.
+
 ## `<aggregation>_over_time()`: Aggregating values over time:
 
 The following functions allow aggregating each series of a given range vector
@@ -133,11 +193,3 @@ over time and return an instant vector with per-series aggregation results:
 * `max_over_time(range-vector)`: the maximum value of all points under the specified interval.
 * `sum_over_time(range-vector)`: the sum of all values under the specified interval.
 * `count_over_time(range-vector)`: the count of all values under the specified interval.
-
-## `topk()` and `bottomk()`
-
-`topk(k integer, v instant-vector)` returns the `k` largest elements of `v` by
-sample value.
-
-`bottomk(k integer, v instant-vector` returns the `k` smallest elements of `v`
-by sample value.