When a JuiceFS file system is deployed in production, a critical question arises: how to monitor its operational status in real-time? As it may be providing persistent storage support for critical business applications or container workloads, even minor failures or performance degradation can have significant impacts.
JuiceFS publishes its operational status data in real-time after startup. By combining Grafana and Prometheus, two open-source tools, you can establish a comprehensive monitoring system. This system collects key metrics such as read/write throughput, IOPS, latency, storage utilization, and request distribution, presenting them in an intuitive dashboard for complete monitoring and visualization.
This article provides a detailed guide on using Grafana to monitor JuiceFS. Future articles will offer in-depth analysis of JuiceFS status metrics.
Understanding Grafana
Grafana is an open-source dashboard and visualization tool for monitoring and analyzing time-series data. It's primarily used for internet infrastructure and application analysis, sensor data, application metrics, and monitoring. Users can create and customize Grafana dashboards to display their most relevant data, tracking and analyzing key performance indicators (KPIs).
Grafana supports multiple data sources, including Prometheus, InfluxDB, and Elasticsearch, allowing data aggregation from various sources within a unified interface for analysis and visualization. The user-friendly query editor enables deeper data exploration and extraction of valuable insights.
By default, JuiceFS outputs file system status metrics through port 9567. Prometheus can collect these metrics, which can then be visualized graphically through Grafana.
Both Prometheus and Grafana are open-source software that can be downloaded and deployed independently. However, maintaining these systems increases operational overhead. For a simpler approach to building a visual monitoring system for JuiceFS, consider Grafana Cloud, which provides all necessary components in a single solution with a free tier sufficient for initial use.
Grafana Cloud
Grafana Cloud is a SaaS service offering free, pay-as-you-go, and premium subscription plans. The main difference between free and paid plans is the data retention period - free plans retain data for 14 days, while paid plans retain it for 13 months.
The advantage of Grafana Cloud is its provision of Prometheus and Grafana through a SaaS model, addressing data collection and visualization needs in a single solution. Maintenance and scaling are handled by the service provider.
The process for setting up a monitoring system for JuiceFS using Grafana Cloud generally follows these steps:
- Prepare and mount the JuiceFS file system
- Register for a Grafana Cloud account and create a Stack resource
- Install and configure grafana-agent locally to start reporting metrics
- Configure the JuiceFS Dashboard and begin using it
Step 1: Preparing the JuiceFS File System
Both the community edition and cloud service version of JuiceFS publish status metrics in Prometheus format. However, there are important distinctions:
- The community edition publishes Prometheus format metrics through http://127.0.0.1:9567 on the client host
- The cloud service publishes both Prometheus and JSON format metrics through the SaaS platform address (replace VOLUME_NAME with the file system name and API_TOKEN with the corresponding token)
The JuiceFS cloud service console displays status metrics for the past 7 days, with hyperlinks corresponding to the two status publishing formats mentioned above, which can be directly used by Prometheus.
Note that only mounted file systems have status metrics, so ensure the file system is properly prepared before proceeding.
Methods for creating JuiceFS file systems differ slightly between the community edition and cloud service, though the principles are the same. For unfamiliar operations, refer to the official documentation for details.
Step 2: Registering for Grafana Cloud
Grafana Cloud refers to its user resources as Stacks, which include not only Grafana but also Prometheus, Loki, and other monitoring-related services.
After registration, you'll receive a dedicated xxx.grafana.net URL (e.g., https://juicefs.grafana.net) that provides direct access to your Grafana console.
Step 3: Configuring the Prometheus Agent
With a Grafana Cloud account and Stack, you can begin configuring data sources. Grafana Cloud offers various connectable data sources.
Since JuiceFS exposes status information in Prometheus format by default, select "Hosted Prometheus metrics" and configure according to your requirements.
1. Selecting a Metric Collection Method
Two options are available:
- Install the Grafana Agent client locally, which handles uploading JuiceFS status metrics to the cloud Prometheus service
- Copy data from a locally maintained Prometheus instance to the cloud
The first option stores all metrics in the cloud with no local copies, making it suitable for use with Grafana Cloud's alerting services. The second option requires local deployment and maintenance of a Prometheus instance, with data stored locally and copied to the cloud.
This article will focus on the first approach. For Kubernetes environments, consult the relevant documentation.
2. Selecting System and Architecture
Install the Grafana Agent on the host where the JuiceFS client is located, selecting the appropriate system and architecture for your local host.
3. Downloading the Grafana Agent Client
Copy and run the provided code on the JuiceFS host. This will download the grafana-agent client to the current directory and make it executable. The grafana-agent program is essentially a customized Prometheus client.
4. Creating the Configuration File
Generate a configuration file for grafana-agent, which is essentially a Prometheus configuration file with pre-configured Grafana Cloud service addresses and settings.
Copy the generated command to the JuiceFS host to write the configuration information to the current directory's agent-config.yaml file.
To enable collection of JuiceFS file system status metrics, modify this configuration file to add the JuiceFS status publishing address and adjust the data scrape interval.
As mentioned earlier, the community edition of JuiceFS publishes status metrics through port 9567 by default.
However, if multiple file systems are mounted on the same host using different access methods, JuiceFS randomly assigns ports to subsequent access methods. For example, if a host has a file system mounted via both mount and WebDAV methods, a random port will be assigned to the WebDAV protocol for status metric publishing.
Edit the agent-config.yaml based on the actual information found. Here's a modified example:
metrics:
global:
scrape_interval: 15s
configs:
- name: hosted-prometheus
scrape_configs:
- job_name: node
static_configs:
- targets: ['localhost:9100']
- job_name: juicefs
static_configs:
- targets: ["localhost:9567", "localhost:35013"]
remote_write:
- url: https://prometheus-prod-36-prod-us-west-0.grafana.net/api/prom/push
basic_auth:
username: 1480721
password: glc_eyJvIjo`
- In the third line, modifying scrape_interval to 15s (or smaller) ensures capturing real-time status changes in JuiceFS
- In the tenth line, add juicefs-related configuration, including all status publishing addresses in the targets array
5. Starting the Agent
Paste and run the command on the JuiceFS host to start the grafana-agent program.
If needed, configure the grafana-agent client as a systemd service to run automatically on startup. Here's an example grafana-agent.service configuration file:
[Unit]
Description=Grafana Agent
After=network-online.target
[Service]
ExecStart=/opt/grafana-agent/grafana-agent-linux-amd64 --config.file=/opt/grafana-agent/agent-config.yaml
Restart=always
User=root
Group=root
[Install]
WantedBy=multi-user.target
When using this auto-start configuration, copy the grafana-agent-linux-amd64 program and agent-config.yaml file to the /opt/grafana-agent directory.
Step 4: Configuring the Grafana Dashboard
The previous steps connected the local JuiceFS with the Prometheus service provided by Grafana Cloud. The grafana-agent program will scrape JuiceFS status metrics every 15s according to the configuration and push the data to the cloud in real-time.
With real-time status data collected, the next step is to visualize it graphically through a Grafana Dashboard.
There are two common methods for creating a Dashboard:
- Create a completely new Dashboard, defining queries, charts, states, tables, etc., based on actual monitoring needs
- Import a pre-created Dashboard template, either as a JSON file template or from the Grafana official library's public templates
1. Using Dashboard Templates
JuiceFS provides a pre-defined Grafana Dashboard template (grafana_template.json) for displaying mount point, S3 gateway, Kubernetes, and Hadoop Java SDK metrics. This template is included in the official Grafana Dashboard repository and can be loaded directly using the URL or ID 20794.
When importing the template, you can customize the Dashboard name and location, though default settings are usually sufficient.
The official template contains extensive status information, with interface elements called Panels that can be adjusted, repositioned, styled, added, or removed as needed.
2. Manually Creating a Dashboard
You can create a complete new Dashboard from scratch or modify the JuiceFS Dashboard template.
A Dashboard consists of a series of Panels (also called Visualizations in Grafana). Adding a Visualization means adding a Panel.
Each Panel displays a single status metric. In the upper-right dropdown menu, you can select the data presentation format, such as Chart, table, or stat.
In the Query area below, select the Data source (the Prometheus service that collects JuiceFS status metrics). Then, in the Metrics browser, choose the metric to query. The sidebar offers various options for fine-tuning the display, such as color and conditional formatting.
You can freely switch Dashboard Panels, such as changing some Chart visualizations in the JuiceFS official template to Stat type, or creating multiple Panels with different display formats for the same metric.
At this point, the visual monitoring system for JuiceFS is complete. You can refer to documentation to understend the meaning of each status metric. Due to space limitations, a separate article will be dedicated to interpreting JuiceFS status metrics.
Conclusion
This article primarily introduced how to build a visual monitoring system for JuiceFS using Grafana Cloud, selecting the simplest approaches for readers with different experience levels.
In reality, building any IT system involves a series of decisions. As you can see, many aspects of the process covered in this article offer alternative choices. For example, regarding Prometheus data reporting, this article used the Grafana Agent, but readers could also choose a self-hosted Prometheus instance. Therefore, this article serves as an introduction, and the final deployement and usage methods require flexible selection and combination based on individual needs.
Finally, we hope the steps outlined in this article will help you establish a monitoring system for JuiceFS. If you have any questions, feel free to ask and discuss in the JuiceFS community user group.