A time-series database is a type of database that is optimized for handling time-stamped or time-series data. Time-series data is data that is recorded over time, such as stock prices, sensor readings, or application logs. They are designed to handle large amounts of data that are continuously generated at a high frequency and to enable efficient queries and analysis of this data. They typically store data in a compressed and optimized format that allows for fast and efficient data retrieval and analysis.
Time-series databases offer several features and benefits that include:
- Time-Stamping: Time-series databases are optimized for handling time-stamped data and can efficiently store and retrieve data based on timestamps.
- Scalability: Time-series databases can scale horizontally to handle large volumes of data and can be deployed across multiple nodes or clusters.
- Aggregation and Querying: Time-series databases provide features to efficiently query and aggregate time-series data, such as rolling averages or minimum and maximum values over a given time period.
- High Write Rates: Time-series databases are designed to handle high write rates, with the ability to ingest and store large amounts of data at high frequencies.
The commonly used time-series databases are InfluxDB, Prometheus, OpenTSDB, and TimescaleDB.
What is VictoriaMetrics?
VictoriaMetrics is an open-source, high-performance, and cost-effective time-series database and monitoring solution. It is designed to handle large amounts of time-series data generated by applications, services, and devices, and to enable efficient querying and analysis of this data.
It is fast and scalable, with the ability to handle millions of data points per second and to scale horizontally across multiple nodes or clusters. It uses a column-oriented storage format that optimizes data compression and retrieval for time-series data.
The key features and benefits associated with VictoriaMetrics are:
- High Performance: VictoriaMetrics is optimized for high write and query rates, with the ability to handle millions of data points per second. It ideally works with big amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various Enterprise workloads.
- Low Resource Usage: VictoriaMetrics uses a columnar storage format that optimizes data compression and reduces storage and memory usage.
- Scalability: VictoriaMetrics can be deployed across multiple nodes or clusters and can scale horizontally to handle large amounts of data.
- Querying and Analysis: VictoriaMetrics provides a flexible and efficient query language for analyzing time-series data, as well as integrations with popular visualization tools such as Grafana.
- Monitoring and Alerting: VictoriaMetrics includes built-in monitoring and alerting features for tracking metrics and events in real time.
- PromQL-like query language: MetricsQL provides improved functionality on top of PromQL.
- It supports metrics relabeling.
Step 1: Install VictoriaMetrics time-series database
VictoriaMetrics can be installed in a number of ways. These include:
- Binary file
- Helm charts for single-node and cluster versions of VictoriaMetrics.
- Docker Image
- Kubernetes operator for VictoriaMetrics.
- Ansible role for installing cluster VictoriaMetrics (by VictoriaMetrics).
- Ansible role for installing cluster VictoriaMetrics (by the community).
- Ansible role for installing single-node VictoriaMetrics (by the community).
- Snap package for VictoriaMetrics.
For this guide, to make it easier, we will install VictoriaMetrics on Ubuntu using Snap. First, you need to ensure that snapd is installed and running on your Ubuntu system.
This can be done using the aid provided in the guide below:
With Snap installed and running, you can easily install VictoriaMetrics. Check the available package info:
$ snap info victoriametrics
summary: VictoriaMetrics is fast, cost-effective and scalable time-series
database.
publisher: VictoriaMetrics (f41gh7)
store-url: https://snapcraft.io/victoriametrics
contact: [email protected]
license: Apache-2.0
description: |
* VictoriaMetrics can be used as long-term storage for Prometheus or for
....
channels:
latest/stable: v1.89.1 2023-03-14 (220) 9MB -
latest/candidate: v1.89.1 2023-03-14 (220) 9MB -
latest/beta: v1.89.1 2023-03-14 (220) 9MB -
latest/edge: v1.89.1 2023-03-14 (220) 9MB -
Install VictoriaMetrics with the command:
$ sudo snap install victoriametrics
2023-04-18T18:34:25Z INFO Waiting for automatic snapd restart...
victoriametrics v1.90.0 from VictoriaMetrics (f41gh7) installedOnce complete, verify the installation by checking its health on port 8428 with the command:
$ curl http://localhost:8428/metrics
.....
flag{name="snapshotAuthKey", value="secret", is_set="false"} 1
flag{name="snapshotCreateTimeout", value="0s", is_set="false"} 1
flag{name="snapshotsMaxAge", value="0", is_set="false"} 1
flag{name="sortLabels", value="false", is_set="false"} 1
flag{name="storage.cacheSizeIndexDBDataBlocks", value="0", is_set="false"} 1
flag{name="storage.cacheSizeIndexDBIndexBlocks", value="0", is_set="false"} 1
flag{name="storage.cacheSizeIndexDBTagFilters", value="0", is_set="false"} 1
flag{name="storage.cacheSizeStorageTSID", value="0", is_set="false"} 1
flag{name="storage.maxDailySeries", value="0", is_set="false"} 1
flag{name="storage.maxHourlySeries", value="0", is_set="false"} 1
flag{name="storage.minFreeDiskSpaceBytes", value="10000000", is_set="false"} 1
flag{name="storageDataPath", value="/var/lib/victoriametrics", is_set="true"} 1
flag{name="streamAggr.config", value="", is_set="false"} 1
flag{name="streamAggr.dedupInterval", value="0s", is_set="false"} 1
flag{name="streamAggr.keepInput", value="false", is_set="false"} 1
flag{name="tls", value="false", is_set="false"} 1
flag{name="tlsCertFile", value="", is_set="false"} 1
flag{name="tlsCipherSuites", value="", is_set="false"} 1
flag{name="tlsKeyFile", value="secret", is_set="false"} 1
flag{name="tlsMinVersion", value="", is_set="false"} 1
flag{name="usePromCompatibleNaming", value="false", is_set="false"} 1
flag{name="version", value="false", is_set="false"} 1
flag{name="vmalert.proxyURL", value="", is_set="false"} 1
flag{name="vmui.customDashboardsPath", value="", is_set="false"} 1Now with this output, we have VictoriaMetrics installed and running.
Step 2: Configure VictoriaMetrics time-series database
Once installed with this method, you can do the required configurations to your VictoriaMetrics time-series database. VictoriaMetric allows variables using the %{ENV_VAR} syntax on the command line.
But for the Snap installation, we can set the variables such as the scrape interval with the below command:
echo 'FLAGS="-selfScrapeInterval=10s -search.logSlowQueryDuration=20s"' | sudo tee $SNAP_DATA/var/snap/victoriametrics/current/extra_flagsThis will add the Flags to the /var/snap/victoriametrics/current/extra_flags file. Here, we will not change the -storageDataPath flag, since the snap package has limited access to the host filesystem. The data folder is located at /var/snap/victoriametrics/current/var/lib/victoriametrics
For the changes made to take effect, restart the service:
$ sudo snap restart victoriametrics
Restarted.
You can also change the crape configuration by editing the file below:
sudo vim /var/snap/victoriametrics/current/etc/victoriametrics-scrape-config.yamlOnce the desired changes have been made, you re-read the config with the command:
curl 127.0.0.1:8428/-/reloadStep 3: Install and Configure Prometheus
Prometheus is a free and open-source event monitoring and alerting tool. It was developed in 2012 by SoundCloud and then later in 2016, promoted to the Cloud Native Computing Foundation. It works by collecting and storing metrics as time series data. For use to demonstrate if the VictoriaMetrics database is working perfectly, we will send some metrics to it from Prometheus.
To install Prometheus, use the guide below:
Once Prometheus is installed and running, we will make adjustments to the config file:
sudo vim /etc/prometheus/prometheus.ymlTo allow Prometheus to send data to VictoriaMetrics, add the lines below:
remote_write:
- url: http://127.0.0.1:8428/api/v1/writeYou need to replace 127.0.0.0.1 with the IP address of your VictoriaMetrics server if it is running on a separate host.
To apply the config, use the command:
sudo kill -HUP `pidof prometheus`
sudo systemctl restart prometheus.serviceNow Prometheus will write incoming data to local storage and replicate it to remote storage in parallel. This means that the data will remain available in the local storage for the set –storage.tsdb.retention.time duration even if remote storage is unavailable.
If you are sending data to VictoriaMetrics from multiple Prometheus instances, you need to add the below lines under the Global section.
global:
external_labels:
datacenter: dc-123This will tell Prometheus to add datacenter=dc-123 labelling to each sample before sending it to remote storage. The label datacenter is an example, you need to set this as a unique variable across the Prometheus instances.
Now load the VictoriaMetrics VMUI and see if Prometheus metrics have been received. To achieve this, use the URL http://IP_Address:8428
Once the VMUI page has loaded, execute queries to verify if Prometheus data is being stored. For example:
scrape_samples_scrapedSample Output:
You can also Explore other Prometheus Metrics as shown below:
Step 4: Install and Configure Grafana
Grafan can be used to visualize the dashboards
Once Grafana has been installed, access it using the URL http://IP_Address:3000. Once logged in, add Prometheus as your data source.
If everything is okay, you should see this.
Now run queries for the added Prometheus data source and see the graph output. For testing purposes, we will execute a similar command as we did in the VictoriaMetrics VMUI.
scrape_samples_scrapedOnce the query has been run, you should see the below output:
From the above output, we can all agree that VictoriaMetrics is working as perfectly
Verdict
That marks the end of this detailed guide on how to install and use the VictoriaMetrics time-series database on Ubuntu. I hope this was informative.
Interested in more?
