Skip to main content

Telemetry

Explanation​

Telemetry and distributed tracing is a important part of complete application monitoring.

Tramvai provides a deep integration with OpenTelemetry Node.js SDK, with custom automatic instrumentation for internal Node.js modules and core Tramvai mechanisms.

danger

Browser OpenTelemetry SDK is not supported yet, because of it limitations:

  • experimental status
  • requires zone.js dependency
  • minimum 30+ kb gzip to bundle size and some performance overhead

However, module provides a lightweight BrowserTracer that generates traceId/spanId identifiers and automatically adds traceparent header to HTTP client requests. Span export and context propagation are not supported on the client.

Usage​

Installation​

You need to install @tramvai/module-opentelemetry:

npm i --save @tramvai/module-module-opentelemetry

Then connect to the project:

import { createApp } from '@tramvai/core';
import { OpenTelemetryModule } from '@tramvai/module-opentelemetry';

createApp({
  name: 'tincoin',
  modules: [OpenTelemetryModule],
});

This will enable OpenTelemetry with automatic instrumentation, and provide Tracer into the application.

Create spans​

Simplest way to wrap operation in span is to use Tracer.trace method, e.g.:

import { OPENTELEMETRY_TRACER_TOKEN } from '@tramvai/module-opentelemetry';

const provider = {
  provide: commandLineListTokens.resolvePageDeps,
  useFactory: ({ tracer, apiService }) => {
    return async function getSmth() {
      tracer.trace('get-smth', async (span) => {
        // set attribute to the span
        span.setAttribute('key', 'value');

        // span will be ended automatically after `apiService.get` method resolves or rejects
        return apiService.get('/smth');
      });
    };
  },
  deps: {
    tracer: OPENTELEMETRY_TRACER_TOKEN,
    apiService: API_SERVICE_TOKEN,
  },
};

For more flexibility methods ProxyTracer.startActiveSpan and ProxyTracer.startSpan is available.

Tracer.trace and other span creation methods calls can be nested, with automatic context propagation.

Export traces​

For export traces to OpenTelemetry collector, you need to provide custom span processor and exporter with OPENTELEMETRY_PROVIDER_SPAN_PROCESSOR_TOKEN token.

If you have a gRPC collector (e.g. Jaeger), library @opentelemetry/exporter-trace-otlp-grpc can be used:

import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-node';
import { OPENTELEMETRY_PROVIDER_SPAN_PROCESSOR_TOKEN } from '@tramvai/module-opentelemetry';

const provider = {
  provide: OPENTELEMETRY_PROVIDER_SPAN_PROCESSOR_TOKEN,
  useFactory: ({ envManager }) => {
    const url = envManager.get('OTEL_GRPC_COLLECTOR_ENDPOINT');
    const exporter = new OTLPTraceExporter({ url });

    return new BatchSpanProcessor(exporter);
  },
  deps: {
    envManager: ENV_MANAGER_TOKEN,
  },
};

Resource attributes​

If you need to extend resource attributes, use OPENTELEMETRY_PROVIDER_RESOURCE_ATTRIBUTES_TOKEN token. Also, @opentelemetry/semantic-conventions library contains some of attribute names.

import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions';
import { OPENTELEMETRY_PROVIDER_RESOURCE_ATTRIBUTES_TOKEN } from '@tramvai/module-opentelemetry';

const provider = {
  provide: OPENTELEMETRY_PROVIDER_RESOURCE_ATTRIBUTES_TOKEN,
  useFactory: ({ appInfo }) => {
    return {
      [ATTR_SERVICE_NAME]: appInfo.appName,
    };
  },
  deps: {
    appInfo: APP_INFO_TOKEN,
  },
};

Active span​

If you need to set attributes to the active span, use Tracer.getActiveSpan method:

// tracer from OPENTELEMETRY_TRACER_TOKEN
function doSmt({ tracer }) {
  const span = tracer.getActiveSpan();

  // span can be absent, for example in `init` or `listen` command line stages,
  // and always `undefined` on the client (no context propagation)
  span?.setAttribute('key', 'value');
}

Automatic instrumentation​

OpenTelemetry instrumentation libraries is not supported, because of it limitations:

  • instrumentations are registered before the module to instrument is require'ed
  • modules are not included in a bundle

Instead, module provides custom instrumentation for all significant cases:

Server module​

All incoming requests are automatically wrapped in root span, where current request and response parameters will be set (path, method, status code, error).

Naming and attributes follow semantic conventions.

Context propagation​

OpenTelemetryModule provides context propagation for incoming and outgoing requests.

Pass traceparent context​

Server-side​

OpenTelemetryModule inject traceparent header to all external API requests via Tramvai HTTP Clients.

On the server-side module includes traceparent meta-tag in resulting html according to W3C format. Also, it bypasses traceparent header to external APIs.

Client-side​

On the client-side, if traceparent meta-tag exists need to be used, you can extract in at follows:

import { extractTraceparentHeader } from '@tramvai/module-opentelemetry';

const extractTraceparent = (): string | undefined => {
  const traceparent = extractTraceparentHeader();

  const [version, traceId, spanId, sampled] = tags[0].content.split('-');

  return traceId;
};

OpenTelemetryModule inject traceparent header to all external API requests via Tramvai HTTP Clients in a browser context too. Each request gets a unique traceparent with fresh traceId and spanId. If consumer already provided traceparent header, it will not be overwritten.

Filter out traceparent header​

In a browser context, some cross-origin requests can be blocked by CORS policy (or preflight requests can be not supported) if traceparent header is added.

So, you can filter out traceparent header for some requests with OPENTELEMETRY_HTTP_CLIENT_BROWSER_HEADERS_INCLUDE_TOKEN token, e.g. as a blacklist:

const provider = provide({
  provide: OPENTELEMETRY_HTTP_CLIENT_BROWSER_HEADERS_INCLUDE_TOKEN,
  useFactory:
    ({ envManager }) =>
    (url: string) => {
      const blacklist = ['/excluded-api-path', envManager.get('CDN_WITHOUT_ALLOWED_HEADERS_URL')];

      return !blacklist.some((blockerUrl) => url.includes(blockerUrl));
    },
  deps: {
    envManager: ENV_MANAGER_TOKEN,
  },
});

Logs correlation​

Server-side​

OpenTelemetryModule inject context for logs correlation.

All application logs will be extended with current span and trace ids in spanId and traceId properties.

Client-side​

In a browser context, spanId and traceId properties are not added to all client logs automatically, because it would connect current SSR logs and all corresponding backend's logs for the current browser session, which can lead to a huge amount of logs and inefficient debugging.

However, HTTP client logs are enriched with traceId and spanId on the client side. OpenTelemetryModule registers an HTTP_CLIENT_LOGGER_EXTENSION that extracts the traceparent header from outgoing HTTP requests and adds parsed traceId and spanId to the corresponding log entries. This allows correlating specific HTTP client requests with their server-side traces.

Debug and testing​

In development mode you can enable ConsoleSpanExporter, which prints all spans to the console, with env variable TRAMVAI_OPENTELEMETRY_DEBUG=true.

For testing purposes, you can use InMemorySpanExporter.