OpenTelemetry Observability in Practice: Unified Traces, Metrics, Logs Collection

技术架构

The Three Pillars of Observability: Traces, Metrics, Logs

In microservice and cloud-native architectures, systems consist of dozens or even hundreds of services. Traditional log-based debugging is no longer sufficient. Observability has become essential for understanding system behavior.

Observability ≠ Monitoring. Monitoring detects known problems; observability explores unknown ones.

Pillar Purpose Typical Scenario Example
Traces Full request lifecycle Cross-service call chain User order → inventory → payment → notification
Metrics Quantified system indicators Performance monitoring, capacity planning QPS=1200, P99 latency=230ms
Logs Discrete event records Error investigation, auditing NullPointerException at OrderService:142

OpenTelemetry is a CNCF incubating project providing vendor-neutral APIs, SDKs, and tools to uniformly collect all three signals.

OpenTelemetry Architecture: SDK, Collector, Exporter

OpenTelemetry uses a three-layer architecture, decoupling collection from export:

┌─────────────────────────────────────────────────┐
│                  Application                     │
│  ┌──────┐  ┌──────┐  ┌──────┐                  │
│  │ SDK  │  │ SDK  │  │ SDK  │  ← Auto/Manual   │
│  │(Java)│  │(Go)  │  │(Node)│                  │
│  └──┬───┘  └──┬───┘  └──┬───┘                  │
│     └─────────┼─────────┘                        │
│               │ OTLP(gRPC/HTTP)                  │
│  ┌────────────▼────────────┐                    │
│  │      OTel Collector      │  ← Data Hub       │
│  │  Recv → Proc → Export    │                    │
│  └──────┬───────┬─────────┘                    │
│    ┌────▼──┐ ┌──▼─────┐                        │
│    │Jaeger │ │Prometheus│  ← Backends           │
│    └───────┘ └─────────┘                        │
└─────────────────────────────────────────────────┘
Layer Component Responsibility Deployment
SDK OpenTelemetry SDK Auto/manual instrumentation In-process
Collector OTel Collector Receive, process, route telemetry Standalone
Exporter OTLP Exporter Export data to backends Inside Collector

Java Agent Auto-Instrumentation: Zero Code Intrusion

The OpenTelemetry Java Agent's greatest appeal is zero code intrusion — just add a JVM argument to automatically trace Spring Boot, HTTP clients, JDBC, Redis, and more.

Launch with Java Agent

java -javaagent:opentelemetry-javaagent.jar \
     -Dotel.service.name=order-service \
     -Dotel.traces.exporter=otlp \
     -Dotel.metrics.exporter=otlp \
     -Dotel.logs.exporter=otlp \
     -Dotel.exporter.otlp.endpoint=http://collector:4317 \
     -jar my-app.jar

Key Configuration

Config Default Description
otel.service.name unknown Service name in trace
otel.traces.exporter otlp Trace exporter: otlp/zipkin/none
otel.traces.sampler parentBased_alwaysOn Sampling strategy
otel.exporter.otlp.endpoint http://localhost:4317 Collector address

Auto-Instrumentation Coverage

Component Auto-Trace Auto-Metrics
Spring Web MVC/WebFlux ✅ ✅
RestTemplate/WebClient ✅ ✅
JDBC / R2DBC ✅ ✅
Redis (Lettuce/Jedis) ✅ ✅
Kafka ✅ ✅
gRPC ✅ ✅

Distributed Tracing: Cross-Service Call Chain Visualization

Example Call Chain

[OrderService] POST /api/orders
  ├── [InventoryService] POST /api/inventory/deduct
  │     └── [Redis] GET inventory:product-123
  ├── [PaymentService] POST /api/payments/charge
  │     ├── [DB] INSERT INTO payments
  │     └── [Kafka] PRODUCE payment-success
  └── [NotificationService] CONSUME payment-success

W3C Trace Context Propagation

GET /api/inventory/deduct HTTP/1.1
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067bf0bc902b7-01
Field Format Description
version 00 W3C version
trace-id 32-hex Globally unique trace identifier
parent-id 16-hex Parent span identifier
trace-flags 2-hex 01=sampled, 00=not sampled

Custom Spans and Attributes

Auto-instrumentation covers technical aspects; business semantics require manual custom spans:

@Service
public class OrderService {

    private final Tracer tracer;

    public OrderService(Tracer tracer) {
        this.tracer = tracer;
    }

    public OrderResult createOrder(CreateOrderRequest request) {
        Span span = tracer.spanBuilder("order.create")
            .setAttribute("order.userId", request.getUserId())
            .setAttribute("order.productCount", request.getItems().size())
            .setAttribute("order.totalAmount", request.getTotalAmount())
            .startSpan();

        try (Scope scope = span.makeCurrent()) {
            OrderResult result = doCreateOrder(request);
            span.setAttribute("order.id", result.getOrderId());
            span.setAttribute("order.status", result.getStatus().name());
            return result;
        } catch (Exception e) {
            span.recordException(e);
            span.setStatus(StatusCode.ERROR, e.getMessage());
            throw e;
        } finally {
            span.end();
        }
    }
}

Attribute Naming Conventions

Prefix Semantics Example
order. Order business order.id, order.status
user. User business user.id, user.tier
db. Database db.operation, db.table
http. HTTP http.method, http.status_code

Metrics Collection: Counter, Histogram, Gauge

Type Semantics Monotonic Aggregation Use Case
Counter Ever-increasing count Yes Sum Request count, error count
Histogram Value distribution - Histogram Request latency, response size
Gauge Current instantaneous value No Last Value Active connections, queue depth 
@Service
public class PaymentService {

    private final LongCounter requestCounter;
    private final LongHistogram latencyHistogram;

    public PaymentService(MeterProvider meterProvider) {
        Meter meter = meterProvider.meterBuilder("payment-service").build();

        this.requestCounter = meter.counterBuilder("payment.requests.total")
            .setDescription("Total payment requests")
            .build();

        this.latencyHistogram = meter.histogramBuilder("payment.request.duration")
            .setUnit("ms")
            .ofLongs()
            .build();
    }

    public PaymentResult processPayment(PaymentRequest request) {
        long startTime = System.currentTimeMillis();
        requestCounter.add(1, Attributes.builder()
            .put("payment.method", request.getMethod().name())
            .build());

        try {
            PaymentResult result = doProcessPayment(request);
            latencyHistogram.record(System.currentTimeMillis() - startTime);
            return result;
        } catch (PaymentException e) {
            latencyHistogram.record(System.currentTimeMillis() - startTime);
            throw e;
        }
    }
}

Collector Deployment Modes

Mode Architecture Pros Cons Use Case
Sidecar One per Pod Isolation, flexible config High resource cost Large K8s clusters
DaemonSet One per Node Good resource utilization Single point risk General recommended
Gateway Standalone cluster Centralized, scalable Extra network hops Multi-cluster

Collector Config

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    send_batch_size: 1024
    timeout: 5s
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/jaeger]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]

Sampling Strategies: Head-Based vs Tail-Based

Head-Based Sampling

otel.traces.sampler=parentbased_traceidratio
otel.traces.sampler.arg=0.1

Simple and efficient, but may miss error requests.

Tail-Based Sampling

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    sampling_policies:
      - name: errors-policy
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-policy
        type: latency
        latency:
          threshold_ms: 1000
      - name: fallback-policy
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

Recommendation: Use tail-based sampling in production to ensure 100% visibility of errors and slow requests.

Frontend RUM Integration

import { WebTracerProvider } from '@opentelemetry/sdk-trace-web';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';

const exporter = new OTLPTraceExporter({
  url: 'https://collector.example.com/v1/traces',
});

const provider = new WebTracerProvider();
provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
provider.register();

// Auto-instrument fetch/XHR requests
registerInstrumentations({
  instrumentations: [
    new FetchInstrumentation({
      propagateTraceHeaderCorsUrls: [/api\.example\.com/],
    }),
  ],
});

Result: Frontend HTTP requests carry traceparent headers, enabling end-to-end tracing from browser to backend.

Prometheus + Grafana Classic Stack

OpenTelemetry SDK → OTel Collector → Prometheus → Grafana
                                     → Jaeger/Tempo → Grafana
                                     → Loki → Grafana

Key Grafana Dashboard Metrics

Panel Metric PromQL Alert Threshold
Request Rate QPS rate(http_server_request_total[5m]) >10000
P99 Latency Latency histogram_quantile(0.99, rate(http_server_duration_bucket[5m])) >500ms
Error Rate Errors rate(http_server_request_total{status=~"5.."}[5m]) / rate(http_server_request_total[5m]) >1%

RED Principle: Rate, Errors, Duration — the three golden signals for service monitoring.

Summary

OpenTelemetry's core value in 2026:

  1. Unified Collection: One SDK for Traces, Metrics, Logs
  2. Vendor Neutral: Not locked into any backend
  3. Zero Intrusion: Java Agent auto-instrumentation
  4. End-to-End: From frontend RUM to backend microservices
  5. Mature Ecosystem: CNCF project, SDKs for all major languages

Observability is not a luxury — it's a necessity for microservice architectures. Without it, microservices are black boxes; with OpenTelemetry, system behavior is fully visible.

Try these browser-local tools — no sign-up required →

#OpenTelemetry#可观测性#分布式追踪#Metrics#日志