Apache SkyWalking BookKeeper 集群监控接入实战：基于 OpenTelemetry Collector 与 MAL 的指标采集链路

可观测性APM链路追踪指标监控日志分析微服务【免费下载链接】skywalkingAPM, Application Performance Monitoring System项目地址https://gitcode.com/gh_mirrors/sk/skywalking点击查看免费下载本篇指南讲解如何在 Apache SkyWalking 中监控 Apache BookKeeper 集群通过 OpenTelemetry Collector 抓取 BookKeeper 的 Prometheus 指标并推送到 SkyWalking OAP经由 OpenTelemetry receiver 与 MALMeter Analysis Language完成过滤、聚合与存储。读完你将掌握从 BookKeeper 到 SkyWalking 的完整数据链路、Layer: BOOKKEEPER的实体建模方式、全部预置监控面板与指标名称以及如何基于仓库中的otel-rules规则文件自定义自己的监控指标与表达式。BookKeeper 监控的整体架构SkyWalking 本身并不直接采集 BookKeeper 指标而是采用Prometheus 暴露 OpenTelemetry Collector 中转 OAP 解析的标准接入模式。整体数据流如下BookKeeper 暴露指标BookKeeper 集群通过 Prometheus endpoint 暴露自身指标默认端口8000路径/metrics每个 Bookie 节点即为一个 Prometheus target。OpenTelemetry Collector 抓取与推送Collector 使用 Prometheus Receiver 从 BookKeeper 集群拉取指标再通过 OpenTelemetry gRPC exporterOTLP推送给 SkyWalking OAP Server。OAP 解析与存储SkyWalking OAP Server 使用 MALMeter Analysis Language解析收到的指标执行过滤filter、计算calculate、聚合aggregate后写入存储最终呈现在监控面板上。从实体模型看BookKeeper 集群整体被建模为 OAP 中的一个Service其Layer为BOOKKEEPER集群内的各个节点Bookie则被建模为该 Service 下的Instance。该 Layer 在源码中注册于 Layer.javaregister(BOOKKEEPER, 33, true)因此 BookKeeper 服务与普通业务服务在拓扑、告警、指标面板上完全独立。前置条件与三步接入 Setup接入 BookKeeper 监控共三步搭建 BookKeeper 集群按 BookKeeper 官方部署文档搭建集群并确保其 Prometheus endpoint 可被 Collector 访问。部署 OpenTelemetry CollectorCollector 负责抓取与转发其 Kubernetes 部署方式参考 OpenTelemetry 官方文档。仓库中给出了一个可参考的 Collector 配置示例 otel-collector-config.yaml该示例同时演示了 Pulsar 与 BookKeeper 两类 job 的抓取配置详见下文。配置 SkyWalking OpenTelemetry receiver确保 OAP 的 OpenTelemetry receiver 已激活并加载 BookKeeper 的 MAL 规则。激活 OAP 的 OpenTelemetry receiverOpenTelemetry receiver 是 OAP 接收 OTLP 指标数据的入口。在 application.yml 中receiver-otel模块默认处于激活状态其核心配置如下receiver-otel: selector: ${SW_OTEL_RECEIVER:default} default: enabledHandlers: ${SW_OTEL_RECEIVER_ENABLED_HANDLERS:otlp-traces,otlp-metrics,otlp-logs} enabledOtelMetricsRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES:apisix,nginx/*,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows,aws-s3/*,aws-dynamodb/*,aws-gateway/*,redis/*,elasticsearch/*,rabbitmq/*,mongodb/*,kafka/*,pulsar/*,bookkeeper/*,rocketmq/*,clickhouse/*,activemq/*,kong/*,flink/*,airflow/*,banyandb/*,envoy-ai-gateway/*,ios/*,miniprogram/*,ai-agent/*}关键点selector可通过环境变量SW_OTEL_RECEIVER覆盖只有值为default或等价配置时 receiver 才会生效。enabledHandlers中必须包含otlp-metrics才能接收指标数据示例默认已包含。enabledOtelMetricsRules中已默认包含bookkeeper/*即otel-rules/bookkeeper/目录下的规则会在 OAP 启动时被自动加载。注意如果 OAP 启动时规则文件格式不合法可能导致启动失败修改规则后需谨慎校验。参考的 OpenTelemetry Collector 配置仓库中 otel-collector-config.yaml 给出了同时抓取 Pulsar broker 与 BookKeeper bookie 的完整示例其中与 BookKeeper 相关的片段为receivers: prometheus: config: scrape_configs: - job_name: bookkeeper-monitoring scrape_interval: 10s static_configs: - targets: [bookie:8000] labels: cluster: pulsar-cluster relabel_configs: - source_labels: [ __address__ ] regex: (.) target_label: node replacement: $$1 exporters: otlp: endpoint: oap:11800 tls: insecure: true service: pipelines: metrics: receivers: - prometheus processors: - batch exporters: - otlp这段配置有两个对 SkyWalking MAL 规则至关重要的细节job_name: bookkeeper-monitoringMAL 规则文件中的filter正是通过该 job 名来筛选数据的见下文规则解析job 名必须与规则中的过滤条件一致。cluster与node标签labels.cluster提供集群名如pulsar-clusterrelabel_configs将 target 地址写入node标签这两个标签正是 MAL 规则中服务/实例维度聚合的依据。BookKeeper Cluster 级监控指标BookKeeper 集群级指标定义在 bookkeeper-cluster.yaml 中覆盖 Bookie 的存储与读写能力。全部监控面板如下监控面板指标名称描述数据来源Bookie Ledgers Countmeter_bookkeeper_bookie_ledgers_countBookie 上 ledger 的数量Bookkeeper ClusterBookie Ledger Writable Dirsmeter_bookkeeper_bookie_ledger_writable_dirsBookie 中可写目录的数量Bookkeeper ClusterBookie Ledger Dir Usagemeter_bookkeeper_bookie_ledger_dir_data_bookkeeper_ledgers_usage成功创建的连接数目录数据使用量Bookkeeper ClusterBookie Entries Countmeter_bookkeeper_bookie_entries_countBookie 写入 entry 的数量Bookkeeper ClusterBookie Write Cache Sizemeter_bookkeeper_bookie_write_cache_sizeBookie 写缓存大小MBBookkeeper ClusterBookie Write Cache Entry Countmeter_bookkeeper_bookie_write_cache_countBookie 写缓存中的 entry 数量Bookkeeper ClusterBookie Read Cache Sizemeter_bookkeeper_bookie_read_cache_sizeBookie 读缓存大小MBBookkeeper ClusterBookie Read Cache Entry Countmeter_bookkeeper_bookie_read_cache_countBookie 读缓存中的 entry 数量Bookkeeper ClusterBookie Read Ratemeter_bookkeeper_bookie_read_rateBookie 读速率bytes/sBookkeeper ClusterBookie Write Ratemeter_bookkeeper_bookie_write_rateBookie 写速率bytes/sBookkeeper Cluster集群级 MAL 规则解析bookkeeper-cluster.yaml 的核心头部与规则如下filter: { tags - tags.job_name bookkeeper-monitoring } # OpenTelemetry job 名称 expSuffix: tag({tags - tags.cluster bookkeeper:: tags.cluster}).service([cluster], Layer.BOOKKEEPER) metricPrefix: meter_bookkeeper metricsRules: - name: bookie_ledgers_count exp: bookie_ledgers_count.sum([cluster, node]) - name: bookie_write_rate exp: bookie_WRITE_BYTES.sum([cluster, node]).rate(PT1M) - name: bookie_read_rate exp: bookie_READ_BYTES.sum([cluster, node]).rate(PT1M) # ... 其余规则见仓库文件逐项理解filter只处理job_name bookkeeper-monitoring的数据。OpenTelemetry Collector 的 Prometheus Receiver 会自动把 Prometheus 的job标签转换为service.name而 OAP 的 OTLP 入口又将其回填为job_name标签因此 MAL 中通过tags.job_name即可精确筛选。expSuffix为集群名加上bookkeeper::前缀后以cluster作为维度生成Layer.BOOKKEEPER的 Service 实体.service([cluster], ...)。这保证了 OAP 中的 BookKeeper 服务命名唯一例如bookkeeper::pulsar-cluster。metricPrefix: meter_bookkeeper所有输出指标统一加前缀形成上文表格中的meter_bookkeeper_*名称。sum([cluster, node])按集群和节点维度求和——由于一个集群由多个 Bookie 组成这里实际上把集群内所有节点的同名指标求和为集群级指标。.rate(PT1M)对bookie_WRITE_BYTES/bookie_READ_BYTES这类累加型计数器计算每分钟的变化率输出即为表格中的读写速率bytes/s。PT1M为 ISO-8601 时长格式1 分钟。BookKeeper Node 级监控指标节点级指标定义在 bookkeeper-node.yaml 中覆盖单个 Bookie 的 JVM 运行时状态与线程池状况。监控面板如下监控面板指标名称描述数据来源JVM Memory Pool Usedmeter_bookkeeper_node_jvm_memory_pool_usedbroker JVM 内存池使用情况Bookkeeper BookieJVM Memorymeter_bookkeeper_node_jvm_memory_used、meter_bookkeeper_node_jvm_memory_committed、meter_bookkeeper_node_jvm_memory_initbroker JVM 内存使用情况Bookkeeper BookieJVM Threadsmeter_bookkeeper_node_jvm_threads_current、meter_bookkeeper_node_jvm_threads_daemon、meter_bookkeeper_node_jvm_threads_peak、meter_bookkeeper_node_jvm_threads_deadlockedJVM 线程数Bookkeeper BookieGC Timemeter_bookkeeper_node_jvm_gc_collection_seconds_sum给定 JVM 垃圾回收器花费的时间秒Bookkeeper BookieGC Countmeter_bookkeeper_node_jvm_gc_collection_seconds_count给定 JVM 垃圾回收的次数Bookkeeper BookieThread Executor Completedmeter_bookkeeper_node_thread_executor_completedexecutor 线程完成数Bookkeeper BookieThread Executor Tasksmeter_bookkeeper_node_thread_executor_tasks_completed、meter_bookkeeper_node_thread_executor_tasks_rejected、meter_bookkeeper_node_thread_executor_tasks_failedexecutor 任务计数Bookkeeper BookiePooled Threadsmeter_bookkeeper_node_high_priority_threads、meter_bookkeeper_node_read_thread_pool_threads线程池线程数Bookkeeper BookiePooled Threads Max Queue Sizemeter_bookkeeper_node_high_priority_thread_max_queue_size、meter_bookkeeper_node_read_thread_pool_max_queue_size线程池最大队列大小Bookkeeper Bookie节点级 MAL 规则要点bookkeeper-node.yaml 与集群级规则的最大区别在于expSuffixfilter: { tags - tags.job_name bookkeeper-monitoring } expSuffix: tag({tags - tags.cluster bookkeeper:: tags.cluster}).instance([cluster], [node], Layer.BOOKKEEPER) metricPrefix: meter_bookkeeper_node使用.instance([cluster], [node], ...)而非.service(...)cluster用于定位 Service加上bookkeeper::前缀node用于定位 Instance从而把每个 Bookie 建模为集群 Service 下的实例。metricPrefix变为meter_bookkeeper_node与集群级规则输出名区分。JVM 类指标如jvm_memory_pool_bytes_used、jvm_gc_collection_seconds_count直接来自 BookKeeper 内嵌的 JVM Prometheus exporter线程池类指标则来自bookkeeper_server_*系列如bookkeeper_server_thread_executor_completed、bookkeeper_server_BookieHighPriorityThread_threads等这些是 BookKeeper 自身的ServerStatsPrometheus 指标经.sum([cluster, node])部分含pool、gc维度聚合而来。规则文件的验证方式仓库内的 MAL 测试数据仓库为 BookKeeper 的两份规则文件都提供了配套的测试样例位于 meter-analyzer-scripts-testbookkeeper-cluster.data.yaml构造了cluster: test-cluster、node: test-node的输入样本各指标值为 100.0并断言输出为meter_bookkeeper_bookie_ledgers_count等指标、实体为scope: SERVICE且service: bookkeeper::test-cluster、layer: BOOKKEEPER。其中读写速率期望值为25.0验证了rate(PT1M)的计算正确性100.0 累计值按分钟求导。bookkeeper-node.data.yaml输入覆盖线程池、JVM 内存、JVM 内存池含pool: PS_Eden_Space维度、GC含gc: PS Scavenge维度等全部指标断言输出实体为scope: SERVICE_INSTANCE、service: bookkeeper::test-cluster、instance: test-node精确验证了 Service Instance 两层实体建模与各维度标签的保留。这些测试样例既是规则正确性的回归保障也是理解输入 Prometheus 标签 → 输出 MAL 指标与实体映射关系的最佳参考。自定义指标、表达式与监控面板如果你需要监控 BookKeeper 的其他指标例如 BookKeeper 4.x 新增的指标、或自定义线程池可以遵循以下路径扩展修改/新增 MAL 规则编辑或仿照 bookkeeper-cluster.yaml 与 bookkeeper-node.yaml。在metricsRules下新增name输出指标名与expMAL 表达式即可注意metricPrefix会自动拼接到输出指标名前。MAL 表达式语法参见 MAL 设计文档。校验规则合法性可仿照仓库中的测试数据文件见上文构造input与expected通过 meter-analyzer-scripts-test 模块验证表达式语义避免 OAP 启动时加载非法规则失败。配置面板BookKeeper 的监控面板配置由 SkyWalking Horizon UI bundleapache/skywalking-horizon-ui随发行包提供OAP 后端不再托管 UI dashboard JSON。因此自定义面板应通过 Horizon UI 完成或参照其面板定义新增仪表盘。总结SkyWalking 对 BookKeeper 的监控遵循Prometheus endpoint → OpenTelemetry Collector → OTLP → OAP MAL的标准链路通过 bookkeeper-cluster.yaml 与 bookkeeper-node.yaml 两份规则文件将集群建模为Layer: BOOKKEEPER的 Service、节点建模为 Instance并预置了存储、缓存、读写速率、JVM 与线程池等 19 个监控面板。接入时只需保证 Collector 的job_name与规则filter一致、提供cluster/node标签、并在 application.yml 中启用bookkeeper/*规则默认已启用即可在 SkyWalking 中看到完整的 BookKeeper 运行视图。赞分享可观测性APM链路追踪指标监控日志分析微服务【免费下载链接】skywalkingAPM, Application Performance Monitoring System项目地址https://gitcode.com/gh_mirrors/sk/skywalking点击查看免费下载相关推荐Apache SkyWalking 接入 Flink 集群监控基于 OpenTelemetry Collector 与 MAL 规则的指标采集与多维监控实战Apache SkyWalking 接入 Flink 集群监控基于 OpenTelemetry Collector 与 MAL 规则的指标采集与多维监控实战可观测性APM链路追踪指标监控日志分析微服务SkyWalking Kafka 监控接入指南基于 Prometheus JMX Exporter 与 OpenTelemetry Collector 的指标采集与 MAL 分析SkyWalking Kafka 监控接入指南基于 Prometheus JMX Exporter 与 OpenTelemetry Collector 的指标可观测性APM链路追踪指标监控日志分析微服务Apache SkyWalking RocketMQ 监控接入指南基于 rocketmq-exporter 与 OpenTelemetry 的指标采集与 MAL 聚合Apache SkyWalking RocketMQ 监控接入指南基于 rocketmq exporter 与 OpenTelemetry 的指标采集与 MA可观测性后端微服务云原生上一篇texture-vs-shape项目FAQ全解答从刺激集获取到模型评估的常见问题下一篇BongoCat你的桌面为何需要一只会互动的智能猫咪创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考