Commit 3d37f96d authored by lucperkins's avatar lucperkins

Fix merge conflict

Signed-off-by: 's avatarlucperkins <lucperkins@gmail.com>
parents e638a26a f0e4a70d
root = true
[*]
end_of_line = lf
insert_final_newline = true
charset = utf-8
[*.{css,html,js,md,rb,sh,yaml,yml}]
indent_style = space
indent_size = 2
[Makefile]
indent_style = tab
---
title: Implementing Custom Service Discovery
created_at: 2018-07-05
kind: article
author_name: Callum Styan
---
## Implementing Custom Service Discovery
Prometheus contains built in integrations for many service discovery (SD) systems such as Consul,
Kubernetes, and public cloud providers such as Azure. However, we can’t provide integration
implementations for every service discovery option out there. The Prometheus team is already stretched
thin supporting the current set of SD integrations, so maintaining an integration for every possible SD
option isn’t feasible. In many cases the current SD implementations have been contributed by people
outside the team and then not maintained or tested well. We want to commit to only providing direct
integration with service discovery mechanisms that we know we can maintain, and that work as intended.
For this reason, there is currently a moratorium on new SD integrations.
However, we know there is still a desire to be able to integrate with other SD mechanisms, such as
Docker Swarm. Recently a small code change plus an example was committed to the documentation
[directory](https://github.com/prometheus/prometheus/tree/master/documentation/examples/custom-sd)
within the Prometheus repository for implementing a custom service discovery integration without having
to merge it into the main Prometheus binary. The code change allows us to make use of the internal
Discovery Manager code to write another executable that interacts with a new SD mechanism and outputs
a file that is compatible with Prometheus' file\_sd. By co-locating Prometheus and our new executable
we can configure Prometheus to read the file\_sd-compatible output of our executable, and therefore
scrape targets from that service discovery mechanism. In the future this will enable us to move SD
integrations out of the main Prometheus binary, as well as to move stable SD integrations that make
use of the adapter into the Prometheus
[discovery](https://github.com/prometheus/prometheus/tree/master/discovery) package.
Integrations using file_sd, such as those that are implemented with the adapter code, are listed
[here](https://prometheus.io/docs/operating/integrations/#file-service-discovery).
Let’s take a look at the example code.
## Adapter
First we have the file
[adapter.go](https://github.com/prometheus/prometheus/blob/master/documentation/examples/custom-sd/adapter/adapter.go).
You can just copy this file for your custom SD implementation, but it's useful to understand what's
happening here.
// Adapter runs an unknown service discovery implementation and converts its target groups
// to JSON and writes to a file for file_sd.
type Adapter struct {
ctx context.Context
disc discovery.Discoverer
groups map[string]*customSD
manager *discovery.Manager
output string
name string
logger log.Logger
}
// Run starts a Discovery Manager and the custom service discovery implementation.
func (a *Adapter) Run() {
go a.manager.Run()
a.manager.StartCustomProvider(a.ctx, a.name, a.disc)
go a.runCustomSD(a.ctx)
}
The adapter makes use of `discovery.Manager` to actually start our custom SD provider’s Run function in
a goroutine. Manager has a channel that our custom SD will send updates to. These updates contain the
SD targets. The groups field contains all the targets and labels our custom SD executable knows about
from our SD mechanism.
type customSD struct {
Targets []string `json:"targets"`
Labels map[string]string `json:"labels"`
}
This `customSD` struct exists mostly to help us convert the internal Prometheus `targetgroup.Group`
struct into JSON for the file\_sd format.
When running, the adapter will listen on a channel for updates from our custom SD implementation.
Upon receiving an update, it will parse the targetgroup.Groups into another `map[string]*customSD`,
and compare it with what’s stored in the `groups` field of Adapter. If the two are different, we assign
the new groups to the Adapter struct, and write them as JSON to the output file. Note that this
implementation assumes that each update sent by the SD implementation down the channel contains
the full list of all target groups the SD knows about.
## Custom SD Implementation
Now we want to actually use the Adapter to implement our own custom SD. A full working example is in
the same examples directory
[here](https://github.com/prometheus/prometheus/blob/master/documentation/examples/custom-sd/adapter-usage/main.go).
Here you can see that we’re importing the adapter code
`"github.com/prometheus/prometheus/documentation/examples/custom-sd/adapter"` as well as some other
Prometheus libraries. In order to write a custom SD we need an implementation of the Discoverer interface.
// Discoverer provides information about target groups. It maintains a set
// of sources from which TargetGroups can originate. Whenever a discovery provider
// detects a potential change, it sends the TargetGroup through its channel.
//
// Discoverer does not know if an actual change happened.
// It does guarantee that it sends the new TargetGroup whenever a change happens.
//
// Discoverers should initially send a full set of all discoverable TargetGroups.
type Discoverer interface {
// Run hands a channel to the discovery provider(consul,dns etc) through which it can send
// updated target groups.
// Must returns if the context gets canceled. It should not close the update
// channel on returning.
Run(ctx context.Context, up chan<- []*targetgroup.Group)
}
We really just have to implement one function, `Run(ctx context.Context, up chan<- []*targetgroup.Group)`.
This is the function the manager within the Adapter code will call within a goroutine. The Run function
makes use of a context to know when to exit, and is passed a channel for sending it's updates of target groups.
Looking at the [Run](https://github.com/prometheus/prometheus/blob/master/documentation/examples/custom-sd/adapter-usage/main.go#L153-L211)
function within the provided example, we can see a few key things happening that we would need to do
in an implementation for another SD. We periodically make calls, in this case to Consul (for the sake
of this example, assume there isn’t already a built-in Consul SD implementation), and convert the
response to a set of `targetgroup.Group` structs. Because of the way Consul works, we have to first make
a call to get all known services, and then another call per service to get information about all the
backing instances.
Note the comment above the loop that’s calling out to Consul for each service:
// Note that we treat errors when querying specific consul services as fatal for for this
// iteration of the time.Tick loop. It's better to have some stale targets than an incomplete
// list of targets simply because there may have been a timeout. If the service is actually
// gone as far as consul is concerned, that will be picked up during the next iteration of
// the outer loop.
With this we’re saying that if we can’t get information for all of the targets, it’s better to not
send any update at all than to send an incomplete update. We’d rather have a list of stale targets
for a small period of time and guard against false positives due to things like momentary network
issues, process restarts, or HTTP timeouts. If we do happen to get a response from Consul about every
target, we send all those targets on the channel. There is also a helper function `parseServiceNodes`
that takes the Consul response for an individual service and creates a target group from the backing
nodes with labels.
## Using the current example
Before starting to write your own custom SD implementation it’s probably a good idea to run the current
example after having a look at the code. For the sake of simplicity, I usually run both Consul and
Prometheus as Docker containers via docker-compose when working with the example code.
`docker-compose.yml`
version: '2'
services:
consul:
image: consul:latest
container_name: consul
ports:
- 8300:8300
- 8500:8500
volumes:
- ${PWD}/consul.json:/consul/config/consul.json
prometheus:
image: prom/prometheus:latest
container_name: prometheus
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- 9090:9090
`consul.json`
{
"service": {
"name": "prometheus",
"port": 9090,
"checks": [
{
"id": "metrics",
"name": "Prometheus Server Metrics",
"http": "http://prometheus:9090/metrics",
"interval": "10s"
}
]
}
}
If we start both containers via docker-compose and then run the example main.go, we’ll query the Consul
HTTP API at localhost:8500, and the file_sd compatible file will be written as custom_sd.json. We could
configure Prometheus to pick up this file via the file_sd config:
scrape_configs:
- job_name: "custom-sd"
scrape_interval: "15s"
file_sd_configs:
- files:
- /path/to/custom_sd.json
......@@ -70,7 +70,7 @@ is not used to send alerts.
## High Availability
Alertmanager supports a mesh configuration to create a cluster for high availability.
This can be configured using the [-mesh-*](https://github.com/prometheus/alertmanager#high-availability) flags.
Alertmanager supports configuration to create a cluster for high availability.
This can be configured using the [--cluster-*](https://github.com/prometheus/alertmanager#high-availability) flags.
It's important not to load balance traffic between Prometheus and its Alertmanagers, but instead, point Prometheus to a list of all Alertmanagers.
......@@ -81,8 +81,9 @@ templating.
| Name | Arguments | Returns | Notes |
| ------------- | ------------- | -------- | -------- |
| title | string |[strings.Title](http://golang.org/pkg/strings/#Title), capitalises first character of each word. |
| join | sep string, s []string | [strings.Join](http://golang.org/pkg/strings/#Join), concatenates the elements of s to create a single string. The separator string sep is placed between elements in the resulting string. (note: argument order inverted for easier pipelining in templates.) |
| toUpper | string | [strings.ToUpper](http://golang.org/pkg/strings/#ToUpper), converts all characters to upper case. |
| toLower | string | [strings.ToLower](http://golang.org/pkg/strings/#ToLower), converts all characters to lower case. |
| match | pattern, string | [Regexp.MatchString](https://golang.org/pkg/regexp/#MatchString). Match a string using Regexp. |
| reReplaceAll | pattern, replacement, text | [Regexp.ReplaceAllString](http://golang.org/pkg/regexp/#Regexp.ReplaceAllString) Regexp substitution, unanchored. |
| join | sep string, s []string | [strings.Join](http://golang.org/pkg/strings/#Join), concatenates the elements of s to create a single string. The separator string sep is placed between elements in the resulting string. (note: argument order inverted for easier pipelining in templates.) |
| safeHtml | text string | [html/template.HTML](https://golang.org/pkg/html/template/#HTML), Marks string as HTML not requiring auto-escaping. |
|reReplaceAll | pattern, replacement, text | [Regexp.ReplaceAllString](http://golang.org/pkg/regexp/#Regexp.ReplaceAllString) Regexp substitution, unanchored. |
......@@ -12,6 +12,7 @@ Besides stored time series, Prometheus may generate temporary derived time serie
as the result of queries.
## Metric names and labels
Every time series is uniquely identified by its _metric name_ and a set of
_key-value pairs_, also known as _labels_.
......
---
title: TLS encryption
sort_rank: 1
---
# Securing Prometheus API and UI endpoints using TLS encryption
Prometheus does not directly support [Transport Layer Security](https://en.wikipedia.org/wiki/Transport_Layer_Security) (TLS) encryption for connections to Prometheus instances (i.e. to the expression browser or [HTTP API](../../prometheus/latest/querying/api)). If you would like to enforce TLS for those connections, we recommend using Prometheus in conjunction with a [reverse proxy](https://www.nginx.com/resources/glossary/reverse-proxy-server/) and applying TLS at the proxy layer. You can use any reverse proxy you like with Prometheus, but in this guide we'll provide an [nginx example](#nginx-example).
NOTE: Although TLS connections *to* Prometheus instances are not supported, TLS is supported for connections *from* Prometheus instances to [scrape targets](../prometheus/latest/configuration/configuration/#<tls_config>).
## nginx example
Let's say that you want to run a Prometheus instance behind an [nginx](https://www.nginx.com/) server available at the `example.com` domain (which you own), and for all Prometheus endpoints to be available via the `/prometheus` endpoint. The full URL for Prometheus' `/metrics` endpoint would thus be:
```
https://example.com/prometheus/metrics
```
Let's also say that you've generated the following using [OpenSSL](https://www.digitalocean.com/community/tutorials/openssl-essentials-working-with-ssl-certificates-private-keys-and-csrs) or an analogous tool:
* an SSL certificate at `/root/certs/example.com/example.com.crt`
* an SSL key at `/root/certs/example.com/example.com.key`
You can generate a self-signed certificate and private key using this command:
```bash
mkdir -p /root/certs/example.com && cd /root/certs/example.com
openssl req \
-x509 \
-newkey rsa:4096 \
-nodes \
-keyout example.com.key \
-out example.com.crt
```
Fill out the appropriate information at the prompts, and make sure to enter `example.com` at the `Common Name` prompt.
## nginx configuration
Below is an example [`nginx.conf`](https://www.nginx.com/resources/wiki/start/topics/examples/full/) configuration file. With this configuration, nginx will:
* enforce TLS encryption using your provided certificate and key
* proxy all connections to the `/prometheus` endpoint to a Prometheus server running on the same host (while removing the `/prometheus` from the URL)
```conf
events {}
http {
server {
listen 443 ssl;
server_name example.com;
ssl_certificate /root/certs/example.com/example.com.crt;
ssl_certificate_key /root/certs/example.com/example.com.key;
location /prometheus {
proxy_pass http://localhost:9090/;
}
}
}
```
Start nginx as root (since nginx will need to bind to port 443):
```bash
sudo nginx -c /usr/local/etc/nginx/nginx.conf
```
NOTE: This example uses `/usr/local/etc/nginx` as the location of the nginx configuration file, but this will vary based on the installation. Other [common nginx config directories](http://nginx.org/en/docs/beginners_guide.html) include `/usr/local/nginx/conf` and `/etc/nginx`.
## Prometheus configuration
When running Prometheus behind the nginx proxy, you'll need to set the external URL to `http://example.com/prometheus` and the route prefix to `/`:
```bash
prometheus \
--config.file=/path/to/prometheus.yml \
--web.external-url=http://example.com/prometheus \
--web.route-prefix="/"
```
## Testing
If you'd like to test out the nginx proxy locally using the `example.com` domain, you can add an entry to your `/etc/hosts` file that re-routes `example.com` to `localhost`:
```
127.0.0.1 example.com
```
You can then use cURL to interact with your local nginx/Prometheus setup:
```bash
curl --cacert /root/certs/example.com/example.com.crt \
https://example.com/prometheus/api/v1/label/job/values
```
You can connect to the nginx server without specifying certs using the `--insecure` or `-k` flag:
```bash
curl -k https://example.com/prometheus/api/v1/label/job/values
```
......@@ -28,7 +28,7 @@ The Prometheus monitoring server
. . .
```
Before starting Prometheus, let's configure it.
Before starting Prometheus, let's configure it.
## Configuring Prometheus
......@@ -51,7 +51,7 @@ scrape_configs:
- targets: ['localhost:9090']
```
There are three blocks of configuration in the example configuration file: `global`, `rule_files`, and `scrape_configs`.
There are three blocks of configuration in the example configuration file: `global`, `rule_files`, and `scrape_configs`.
The `global` block controls the Prometheus server's global configuration. We have two options present. The first, `scrape_interval`, controls how often Prometheus will scrape targets. You can override this for individual targets. In this case the global setting is to scrape every 15 seconds. The `evaluation_interval` option controls how often Prometheus will evaluate rules. Prometheus uses rules to create new time series and to generate alerts.
......@@ -86,24 +86,24 @@ tab.
As you can gather from http://localhost:9090/metrics, one metric that
Prometheus exports about itself is called
`http_requests_total` (the total number of HTTP requests the Prometheus server has made). Go ahead and enter this into the expression console:
`promhttp_metric_handler_requests_total` (the total number of `/metrics` requests the Prometheus server has served). Go ahead and enter this into the expression console:
```
http_requests_total
promhttp_metric_handler_requests_total
```
This should return a number of different time series (along with the latest value recorded for each), all with the metric name `http_requests_total`, but with different labels. These labels designate different types of requests.
This should return a number of different time series (along with the latest value recorded for each), all with the metric name `promhttp_metric_handler_requests_total`, but with different labels. These labels designate different requests statuses.
If we were only interested in requests that resulted in HTTP code `200`, we could use this query to retrieve that information:
```
http_requests_total{code="200"}
promhttp_metric_handler_requests_total{code="200"}
```
To count the number of returned time series, you could write:
```
count(http_requests_total)
count(promhttp_metric_handler_requests_total)
```
For more about the expression language, see the
......@@ -113,10 +113,10 @@ For more about the expression language, see the
To graph expressions, navigate to http://localhost:9090/graph and use the "Graph" tab.
For example, enter the following expression to graph the per-second HTTP request rate happening in the self-scraped Prometheus:
For example, enter the following expression to graph the per-second HTTP request rate returning status code 200 happening in the self-scraped Prometheus:
```
rate(http_requests_total[1m])
rate(promhttp_metric_handler_requests_total{code="200"}[1m])
```
You can experiment with the graph range parameters and other settings.
......@@ -127,4 +127,4 @@ Collecting metrics from Prometheus alone isn't a great representation of Prometh
## Summary
Now you've been introduced to Prometheus, installed it, and configured it to monitor your first resources. We've also seen the basics of how to work with time series data scraped using the expression browser. You can find more [documentation](/docs/introduction/overview/) to help you continue to learn more about Prometheus.
In this guide, you installed Prometheus, configured a Prometheus instance to monitor resources, and learned some basics of working with time series data in Prometheus' expression browser. To continue learning about Prometheus, check out the [Overview](/docs/introduction/overview) for some ideas about what to explore next.
......@@ -112,7 +112,7 @@ A remote write endpoint is what Prometheus talks to when doing a remote write.
A sample is a single value at a point in time in a time series.
In Prometheus, each sample consists of a float64 representing some metric and a millisecond-precision timestamp.
In Prometheus, each sample consists of a float64 value and a millisecond-precision timestamp.
### Silence
......
......@@ -17,7 +17,7 @@ bugs. If you find a security bug, please file it in the issue tracker of the
relevant component. If you prefer to report privately, please do so to the
maintainers listed in the MAINTAINERS.md of the relevant repository.
### Prometheus
## Prometheus
It is presumed that untrusted users have access to the Prometheus HTTP endpoint
and logs. They have access to all time series information contained in the
......
This source diff could not be displayed because it is too large. You can view the blob instead.
This diff is collapsed.
%!PS-Adobe-3.0 EPSF-3.0
%%Creator: cairo 1.15.10 (http://cairographics.org)
%%CreationDate: Thu Jul 5 09:38:32 2018
%%Pages: 1
%%DocumentData: Clean7Bit
%%LanguageLevel: 2
%%BoundingBox: 0 0 85 86
%%EndComments
%%BeginProlog
50 dict begin
/q { gsave } bind def
/Q { grestore } bind def
/cm { 6 array astore concat } bind def
/w { setlinewidth } bind def
/J { setlinecap } bind def
/j { setlinejoin } bind def
/M { setmiterlimit } bind def
/d { setdash } bind def
/m { moveto } bind def
/l { lineto } bind def
/c { curveto } bind def
/h { closepath } bind def
/re { exch dup neg 3 1 roll 5 3 roll moveto 0 rlineto
0 exch rlineto 0 rlineto closepath } bind def
/S { stroke } bind def
/f { fill } bind def
/f* { eofill } bind def
/n { newpath } bind def
/W { clip } bind def
/W* { eoclip } bind def
/BT { } bind def
/ET { } bind def
/BDC { mark 3 1 roll /BDC pdfmark } bind def
/EMC { mark /EMC pdfmark } bind def
/cairo_store_point { /cairo_point_y exch def /cairo_point_x exch def } def
/Tj { show currentpoint cairo_store_point } bind def
/TJ {
{
dup
type /stringtype eq
{ show } { -0.001 mul 0 cairo_font_matrix dtransform rmoveto } ifelse
} forall
currentpoint cairo_store_point
} bind def
/cairo_selectfont { cairo_font_matrix aload pop pop pop 0 0 6 array astore
cairo_font exch selectfont cairo_point_x cairo_point_y moveto } bind def
/Tf { pop /cairo_font exch def /cairo_font_matrix where
{ pop cairo_selectfont } if } bind def
/Td { matrix translate cairo_font_matrix matrix concatmatrix dup
/cairo_font_matrix exch def dup 4 get exch 5 get cairo_store_point
/cairo_font where { pop cairo_selectfont } if } bind def
/Tm { 2 copy 8 2 roll 6 array astore /cairo_font_matrix exch def
cairo_store_point /cairo_font where { pop cairo_selectfont } if } bind def
/g { setgray } bind def
/rg { setrgbcolor } bind def
/d1 { setcachedevice } bind def
/cairo_data_source {
CairoDataIndex CairoData length lt
{ CairoData CairoDataIndex get /CairoDataIndex CairoDataIndex 1 add def }
{ () } ifelse
} def
/cairo_flush_ascii85_file { cairo_ascii85_file status { cairo_ascii85_file flushfile } if } def
/cairo_image { image cairo_flush_ascii85_file } def
/cairo_imagemask { imagemask cairo_flush_ascii85_file } def
%%EndProlog
%%BeginSetup
%%EndSetup
%%Page: 1 1
%%BeginPageSetup
%%PageBoundingBox: 0 0 85 86
%%EndPageSetup
q 0 0 85 86 rectclip
1 0 0 -1 0 86 cm q
0.901961 0.321569 0.172549 rg
42.5 0.5 m 19.027 0.5 0 19.527 0 43 c 0 66.469 19.027 85.5 42.5 85.5 c
65.973 85.5 85 66.469 85 43 c 85 19.527 65.973 0.5 42.5 0.5 c h
42.5 80.043 m 35.82 80.043 30.406 75.582 30.406 70.078 c 54.594 70.078
l 54.594 75.578 49.18 80.043 42.5 80.043 c h
62.473 66.781 m 22.527 66.781 l 22.527 59.535 l 62.473 59.535 l h
62.328 55.809 m 22.641 55.809 l 22.508 55.656 22.371 55.508 22.246 55.352
c 18.156 50.387 17.191 47.793 16.258 45.152 c 16.242 45.066 21.215 46.168
24.742 46.961 c 24.742 46.961 26.559 47.383 29.211 47.867 c 26.664 44.879
25.152 41.082 25.152 37.203 c 25.152 28.684 31.684 21.238 29.328 15.223
c 31.621 15.41 34.078 20.062 34.242 27.344 c 36.68 23.973 37.703 17.816
37.703 14.043 c 37.703 10.137 40.277 5.598 42.852 5.441 c 40.555 9.227
43.445 12.469 46.016 20.516 c 46.98 23.539 46.855 28.625 47.602 31.852 c
47.848 25.152 49 15.375 53.25 12 c 51.375 16.25 53.527 21.57 55 24.125
c 57.375 28.25 58.816 31.375 58.816 37.285 c 58.816 41.25 57.352 44.98 54.883
47.898 c 57.691 47.371 59.629 46.895 59.629 46.895 c 68.742 45.117 l 68.742
45.117 67.418 50.562 62.328 55.809 c h
62.328 55.809 m f
Q Q
showpage
%%Trailer
end
%%EOF
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment