Spark Telemetry Listener — Deployment Guide¶

Product Overview¶

Spark Telemetry Listener is a transparent Spark / MapReduce observability solution that exports IO, CPU, GC, and other metrics from big data tasks to an OTel Collector via the OpenTelemetry protocol, persists them to MySQL or ClickHouse through Kafka, and ultimately visualizes them in Grafana.

Core Components¶

Supported Spark Versions¶

System Architecture¶

graph TD
    subgraph Data Collection Layer
        SP["Spark Plugin<br/>(Task/Stage/JVM)"]
        MR["MR Collector<br/>(History Server)"]
        MA["MR Agent<br/>(Bytecode Instrumentation)"]
        HH["Hive Hook<br/>(ExecuteWithHookContext)"]
    end

    SP -- OTLP gRPC --> OTel
    MR -- OTLP gRPC --> OTel
    MA -- OTLP gRPC --> OTel
    HH -- OTLP gRPC --> OTel

    OTel["OTel Collector<br/>(Receive OTLP -> Debug Output + Kafka Export)"]
    Flink["Flink Metrics Consumer<br/>(Kafka -> Batch Write to MySQL / ClickHouse)"]
    DB[("MySQL / ClickHouse + Grafana<br/>(Time-Series Storage + Visualization Dashboards)")]

    OTel -- "Kafka (OTLP Protobuf)" --> Flink
    Flink --> DB

Build¶

Prerequisites¶

• JDK 8 (Spark 2/3) or JDK 17+ (Spark 4)
• Maven 3.6+

Build Commands¶

# Build Spark 3.x version (default)
mvn clean package -DskipTests

# Build Spark 2.x version
mvn clean package -Pspark-2 -DskipTests

# Build Spark 4.x version (requires JDK 17+)
mvn clean package -Pspark-4 -DskipTests

# Build Omnipackage (unified JAR: Spark 2/3/4 + MR Collector + MR Agent + Hive Hook)
chmod +x build-omni.sh && ./build-omni.sh

# Build Flink Consumer
mvn clean package -pl flink/metrics-flink-consumer,flink/metrics-flink-consumer-dist -am -DskipTests

# Build Diagnostic Tool
mvn clean package -pl diagnostic/diagnostic-core -am -DskipTests

Build Artifacts¶

All distribution JARs are built using maven-shade-plugin; OTel, gRPC, Protobuf, and other dependencies are relocated to the x.mg.metrics.shaded.* namespace to avoid conflicts with the host environment.

Omnipackage Build Verification¶

# Check the artifact
ls -lh spark/spark-telemetry-dist-omni/target/spark-telemetry-dist-omni-*.jar

# Verify relocated adapters
jar tf spark/spark-telemetry-dist-omni/target/*.jar | grep "adapter/internal"

# Verify no unshaded OTel classes
jar tf spark/spark-telemetry-dist-omni/target/*.jar | grep "^io/opentelemetry/"
# Should be empty

One-Click Deployment Scripts¶

After building, use the deployment scripts to install the Omnipackage into the Spark / Hive / MR environment and import Grafana dashboards into your Grafana instance.

Omnipackage Installation Script¶

deploy/install-omni.sh copies the Omnipackage JAR into each component's classpath directory and generates the corresponding configuration files. It supports repeated runs and automatically replaces old JAR versions.

Installation Paths:

Usage:

# Basic install (specify component directories and OTel Collector address)
./deploy/install-omni.sh \
  --spark-home=/opt/spark \
  --hive-home=/opt/hive \
  --hadoop-home=/opt/hadoop \
  --otel-endpoint=http://otel-collector:4317 \
  -y

# Install only Spark and Hive, skip MR Collector
./deploy/install-omni.sh \
  --spark-home=/opt/spark \
  --hive-home=/opt/hive \
  --skip-mr -y

# Spark 2.x environment (uses extraListeners instead of SparkPlugin API)
./deploy/install-omni.sh \
  --spark2 \
  --spark-home=/opt/spark-2.4 \
  --hadoop-home=/opt/hadoop \
  --otel-endpoint=http://otel-collector:4317 \
  -y

# Dry-run mode (shows what will be done without executing)
./deploy/install-omni.sh \
  --dry-run \
  --spark-home=/opt/spark \
  --hive-home=/opt/hive \
  --hadoop-home=/opt/hadoop

# Backup old JAR before replacing
./deploy/install-omni.sh \
  --backup \
  --spark-home=/opt/spark \
  --hive-home=/opt/hive \
  -y

Parameters:

Generated Configuration Files:

The script generates the following files in the directory specified by --config-dir:

After installation, follow the prompts to add the configuration snippets to the corresponding configuration files and restart services.

Grafana Dashboard Deployment Script¶

deploy/deploy-grafana.sh bulk-imports all dashboard JSON files from deploy/grafana/ into a Grafana instance using username/password authentication. It supports repeated runs and automatically overwrites existing dashboards.

Usage:

# Deploy all dashboards to Grafana
./deploy/deploy-grafana.sh \
  --grafana-url=http://grafana:3000 \
  --user=admin \
  --password=admin

# Specify target folder name
./deploy/deploy-grafana.sh \
  --grafana-url=http://grafana:3000 \
  --user=admin \
  --password=secret \
  --folder=Production

# Dry-run mode (no actual upload)
./deploy/deploy-grafana.sh \
  --grafana-url=http://grafana:3000 \
  --user=admin \
  --password=admin \
  --dry-run

Parameters:

Prerequisites: curl, python3

The script automatically creates the target folder in Grafana if it does not exist, then uploads each .json file from deploy/grafana/ one by one. Each run overwrites existing dashboards, making it suitable for CI/CD integration.

OTel Collector Configuration¶

Minimal Configuration¶

Create config.yaml:

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  debug:
    verbosity: detailed
  kafka:
    topic: telemetry-metrics
    encoding: otlp_proto
    brokers:
      - kafka:9092
    producer:
      compression: snappy
      max_message_bytes: 1000000

service:
  extensions: [health_check]
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [debug, kafka]

Running¶

docker run -d --name otel-collector \
  -v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml \
  -p 4317:4317 -p 4318:4318 -p 13133:13133 \
  otel/opentelemetry-collector-contrib:0.96.0 \
  --config=/etc/otelcol-contrib/config.yaml

Important: You must use the otel/opentelemetry-collector-contrib image (not the core image), as the core image does not include the Kafka exporter. The configuration must include the health_check extension, otherwise K8s deployments with probes will enter CrashLoopBackOff.

Common Issues and Troubleshooting¶

Q1: Spark plugin not working, no metrics output¶

1. Verify the JAR path is correct and accessible
2. Check the spark.plugins (Spark 3/4) or spark.extraListeners (Spark 2) configuration
3. Check Driver/Executor logs for TelemetryLifecycle initialized
4. Verify the OTel Collector address is reachable
5. Check that config keys include the .otel. segment (common mistake)

Q2: Short-running job metrics missing¶

The plugin triggers flushAsync() non-blocking flush automatically in onJobEnd and calls forceFlush() synchronously on shutdown. If data is still missing, reduce the export interval: spark.telemetry.otel.export.interval.ms=5000.

Q3: MR Collector connection timeout to History Server¶

1. Verify the URL and port (History Server port is 19888 for both Hadoop 2.x and 3.x)
2. Increase timeouts: connect.timeout.secs / read.timeout.secs

Q4: OTel Collector fails to start¶

1. Use otel/opentelemetry-collector-contrib (not the core image)
2. The configuration must include the health_check extension
3. Specify the config file via the --config parameter

Q5: No metrics data visible in Kafka¶

1. Check OTel Collector logs: kubectl logs -l app=otel-collector
2. Verify the Kafka exporter configuration (broker addresses, topic)
3. Use kafka-dump-log.sh --files <log-file> to verify messages exist
4. Note: kafka-console-consumer.sh may time out in single-node KRaft mode

Q6: Omnipackage version detection error¶

1. Check the version detected by OmniContext in Driver/Executor logs
2. Verify there are no conflicting scala-library JARs on the classpath
3. If using a custom classpath, ensure scala-library matches the Spark version

Q7: Omnipackage build fails (adapters-relocated not found)¶

The adapters-relocated module is only activated under the omni profile. Use the ./build-omni.sh script to build; do not build this module individually.

Appendix: Version Compatibility Matrix¶

Appendix: Port Reference¶