README
¶
Expipe
Expipe can record your application's metrics in ElasticSearch
and you can view them with kibana. It can read from any applications
(written in any language) that provides metrics in json format.
Features
- Very lightweight and fast.
- Can read from multiple input.
- Can ship the metrics to multiple databases.
- Shows memory usages and GC pauses of the apps.
- Metrics can be aggregated for different apps (with elasticsearch's type system).
- A kibana dashboard is also provided here.
- Maps values how you define them. For example you can change bytes to megabytes.
- Benchmarks are included.
There are TODO items in the issue section. Feature requests are welcome! Please refer to golang's expvar documentation for more information. Screen shots can be found in this document. Here is an example:
Upcoming Features
- Read from log files.
- Use as a third-party package.
- Record to more repositories:
- InfluxDB
- Log files
Installation
I will provide a docker image soon, but for now it needs to be installed. You need golang >= 1.7 and glide installed. Simply do:
go get github.com/arsham/expipe
cd $GOPATH/src/github.com/arsham/expipe
glide install
go install
You also need elasticsearch and kibana, here is a couple of docker images you can start with:
docker volume create expipe
docker run -d --name expipe --restart always --ulimit nofile=98304:98304 -v expipe:/usr/share/elasticsearch/data -e ES_JAVA_OPTS='-Xms10G -Xmx10G' -e "xpack.security.enabled=false" -e "xpack.monitoring.enabled=true" -e "xpack.graph.enabled=true" -e "xpack.watcher.enabled=false" -p 9200:9200 -e "http.cors.enabled=true" -e 'http.cors.allow-origin=*' docker.elastic.co/elasticsearch/elasticsearch:5.5.3
docker run -d --name kibana --restart always -p 80:5601 --link expipe:elasticsearch docker.elastic.co/kibana/kibana:5.5.3
Kibana
Access the dashboard (or any other ports you have exposed
kibana to, notice the -p:80:5601 above), and enter expipe as Index name or pattern in management section.
Select @timestamp for Time-field name. In case it doesn't show up, click
Index contains time-based events twice, it will provide you with the timestamp.
Then click on create button.
Importing Dashboard
Go to Saved Objects section of management, and click on the import button.
Upload this file and you're done!
One of the provided dashboards shows the expipe's own metrics, and you can use the other one for everything you have defined in the configuration file.
Usage
With Flags
With this method you can only have one reader and ship to one recorder. Consider the next section for more flexible setup. The defaults are sensible to use, you only need to point the app to two endpoints, and it does the rest for you:
expipe -reader="localhost:1234/debug/vars" -recorder="localhost:9200"
For more flags run:
expipe -h
Advanced
Please refer to this document for advanced configuration and mappings.
LICENSE
Use of this source code is governed by the Apache 2.0 license. License that can be found in the LICENSE file.
Enjoy!
Documentation
¶
There is no documentation for this package.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package datatype contains necessary logic to sanitise a JSON object coming from a reader.
|
Package datatype contains necessary logic to sanitise a JSON object coming from a reader. |
|
Package engine can read from any endpoints that provides expvar data and ships them to elasticsearch.
|
Package engine can read from any endpoints that provides expvar data and ships them to elasticsearch. |
|
internal
|
|
|
Package reader contains logic for reading from a provider.
|
Package reader contains logic for reading from a provider. |
|
expvar
Package expvar contains logic to read from an expvar provide.
|
Package expvar contains logic to read from an expvar provide. |
|
self
Package self contains codes for recording expipe's own metrics.
|
Package self contains codes for recording expipe's own metrics. |
|
testing
Package testing is a test suit for readers.
|
Package testing is a test suit for readers. |
|
Package recorder contains logic to record data into a database.
|
Package recorder contains logic to record data into a database. |
|
elasticsearch
Package elasticsearch contains logic to record data to an elasticsearch index.
|
Package elasticsearch contains logic to record data to an elasticsearch index. |
|
testing
Package testing is a test suit for recorders.
|
Package testing is a test suit for recorders. |
|
Package tools contains some internal functionalities needed for expipe.
|
Package tools contains some internal functionalities needed for expipe. |
|
config
Package config contains the logic for reading the configurations from a file in order to bootstrap it.
|
Package config contains the logic for reading the configurations from a file in order to bootstrap it. |
|
token
Package token contains necessary logic for passing messages.
|
Package token contains necessary logic for passing messages. |