Why have this?

We provide distributed tracing ability since the first alpha version. And so many fans are interested in How we trace a distributed services chain, how to save it in memory, and store it.. It is long story, but now, I think we can keep you in the loop, by given your guys a demonstration.

We update our tracer systems, version to version. And many things can be changed or refactor. here you can get the point of How sky-walking v3.0 did?

Now, here we go.

Background story

Here is a typical distributed services chain.

[Browser, e.g. IE, Chrome] -> [Tomcat7 -> Motan-client] -> [Motan-server -> JDBC/MySQL-driver] -> [MySQL Instance]
-----------------------------------------------------------------------------------------------------------------------
[Client, not traced      ]    [        JVM1           ]    [             JVM2                ] -> [Database, not traced]

In OpenTracing Spec, a Trace represents this distributed services chain.

How sky-walking do?

In our tracer, we build two related TraceSegments represent this trace. Each one of them holds part of trace's spans, which generated in each JVM. Well, Span is another concept in OpenTracing Spec, which is from Dapper Paper, see here

TraceSegment

Here is the short version of TraceSegment:

/**
 * {@link TraceSegment} is a segment or fragment of the distributed trace.
 * {@see https://github.com/opentracing/specification/blob/master/specification.md#the-opentracing-data-model}
 * A {@link
 * TraceSegment} means the segment, which exists in current {@link Thread}. And the distributed trace is formed by multi
 * {@link TraceSegment}s, because the distributed trace crosses multi-processes, multi-threads.
 *
 * Created by wusheng on 2017/2/17.
 */
public class TraceSegment{
    private static final String ID_TYPE = "Segment";

    /**
     * The id of this trace segment.
     * Every segment has its unique-global-id.
     */
    @Expose
    @SerializedName(value = "ts")
    private String traceSegmentId;

    /**
     * The start time of this trace segment.
     */
    @Expose
    @SerializedName(value = "st")
    private long startTime;

    /**
     * The end time of this trace segment.
     */
    @Expose
    @SerializedName(value = "et")
    private long endTime;

    /**
     * The refs of parent trace segments, except the primary one.
     * For most RPC call, {@link #refs} contains only one element,
     * but if this segment is a start span of batch process, the segment faces multi parents,
     * at this moment, we use this {@link #refs} to link them.
     */
    @Expose
    @SerializedName(value = "rs")
    private List<TraceSegmentRef> refs;

    /**
     * The spans belong to this trace segment.
     * They all have finished.
     * All active spans are hold and controlled by "skywalking-api" module.
     */
    @Expose
    @SerializedName(value = "ss")
    private List<Span> spans;

    /**
     * The <code>applicationCode</code> represents a name of current application/JVM and indicates which is business
     * role in the cluster.
     *
     * e.g. account_app, billing_app
     */
    @Expose
    @SerializedName(value = "ac")
    private String applicationCode;

    /**
     * The <code>relatedGlobalTraces</code> represent a set of all related trace. Most time it contains only one
     * element, because only one parent {@link TraceSegment} exists, but, in batch scenario, the num becomes greater
     * than 1, also meaning multi-parents {@link TraceSegment}.
     *
     * The difference between <code>relatedGlobalTraces</code> and {@link #refs} is:
     * {@link #refs} targets this {@link TraceSegment}'s direct parent,
     *
     * and
     *
     * <code>relatedGlobalTraces</code> targets this {@link TraceSegment}'s related call chain, a call chain contains
     * multi {@link TraceSegment}s, only using {@link #refs} is not enough for analysis and ui.
     */
    @Expose
    @SerializedName(value = "gt")
    private DistributedTraceIds relatedGlobalTraces;

Span

Here is the short version of Span:

/**
 * Span is a concept from OpenTracing Spec, also from Google Dapper Paper.
 * Traces in OpenTracing are defined implicitly by their Spans.
 *
 * Know more things about span concept:
 * {@see https://github.com/opentracing/specification/blob/master/specification.md#the-opentracing-data-model}
 *
 * Created by wusheng on 2017/2/17.
 */
public class Span{
    @Expose
    @SerializedName(value="si")
    private int spanId;

    @Expose
    @SerializedName(value="ps")
    private int parentSpanId;

    /**
     * The start time of this Span.
     */
    @Expose
    @SerializedName(value="st")
    private long startTime;

    /**
     * The end time of this Span.
     */
    @Expose
    @SerializedName(value="et")
    private long endTime;

    /**
     * The operation name ot this Span.
     * If you want to know, how to set an operation name,
     * {@see https://github.com/opentracing/specification/blob/master/specification.md#start-a-new-span}
     */
    @Expose
    @SerializedName(value="on")
    private String operationName;

    /**
     * Tag is a concept from OpenTracing spec.
     *
     * {@see https://github.com/opentracing/specification/blob/master/specification.md#set-a-span-tag}
     */
    @Expose
    @SerializedName(value="ts")
    private final Map<String, String> tagsWithStr;

    @Expose
    @SerializedName(value="tb")
    private final Map<String, Boolean> tagsWithBool;

    @Expose
    @SerializedName(value="ti")
    private final Map<String, Integer> tagsWithInt;

    /**
     * Log is a concept from OpenTracing spec.
     *
     * {@see https://github.com/opentracing/specification/blob/master/specification.md#log-structured-data}
     */
    @Expose
    @SerializedName(value="lo")
    private final List<LogData> logs;
}

---

As you understood TraceSegment and Span concepts, now, I use the pseudocode to visualize TraceSegment and Span objects for you.

In JVM1

TraceSegment represents this [Tomcat7 -> Motan-client] segment of trace chain.

TraceSegment
  - traceSegmentId : Segment.1487834018996.460155264.76158.1.1, generated by TraceIdGenerator.generate()
  - startTime: 2017-2-23 14:54:30.145
  - endTime: 2017-2-23 14:54:32.382
  - refs: Null. Ref parent TraceSegment, because browser client not be traced.
  - relatedGlobalTraces
    - [0] DistributedTraceId. Actual class is NewDistributedTraceId.class, because this is the first segment of the distributed chain.
      - id: Trace.1487834018996.460155266.76158.1.1
  - applicationCode: WebApplication. Or whatever JVM1 named.
  - spans
    - [0] Span
      - spanId: 0
      - parentSpanId: -1
      - startTime: 2017-2-23 14:54:30.145
      - endTime: 2017-2-23 14:54:32.382
      - operationName: /Web/GetUser
      - tagsWithStr
        - span.layer: http
        - url: http://10.218.9.86:8080/Web/GetUser
        - span.kind: server
        - component: Tomcat
        - peer.host: 202.135.4.12
      - tagsWithInt
        - status_code: 200
    - [1] Span
      - spanId: 1
      - parentSpanId: 0
      - startTime: 2017-2-23 14:54:30.185
      - endTime: 2017-2-23 14:54:32.102
      - operationName: com.a.eye.skywalking.demo.services.GetUserService.findUser(String, String)
      - tagsWithStr
        - span.layer: rpc
        - url: motan://10.20.3.15:3000/com.a.eye.skywalking.demo.services.GetUserService.findUser(String, String)
        - span.kind: client
        - component: Motan
        - peer.host: 10.20.3.15:3000

In JVM2

TraceSegment represents this [Motan-server -> JDBC/MySQL-driver] segment of trace chain.

TraceSegment
  - traceSegmentId : 302017.1487834027215.-2078362600.76161.1.1, generated by TraceIdGenerator.generate()
  - startTime: 2017-2-23 14:54:30.305
  - endTime: 2017-2-23 14:54:32.92
  - refs . This ref to TraceSegment object in JVM1, by its traceSegementId and parent spanId.
    - [0] TraceSegmentRef
      - traceSegmentId: 302017.1487834018996.460155264.76158.1.1
      - spanId: 1. As Motan-client spanId is 2, so primaryRef.spanId = 1.
      - applicationCode: WebApplication
      - peerHost: 10.20.3.15:3000. By the given applicationCode and peerHost, you can say: WebApplication is an alias of host:10.20.3.15:3000. This is very useful for topological graph.
  - applicationCode: MotanServiceApplication. Or whatever JVM2 named.
  - relatedGlobalTraces
    - [0] DistributedTraceId. Actual class is PropagatedTraceId, because it propagates from JVM1.
      - id: Trace.1487834018996.460155266.76158.1.1
  - spans
    - [0] Span
      - spanId: 0
      - parentSpanId: -1
      - startTime: 2017-2-23 14:54:31.5
      - endTime: 2017-2-23 14:54:32.102
      - operationName: com.a.eye.skywalking.demo.services.GetUserService.findUser(String, String)
      - tagsWithStr
        - span.layer: rpc
        - url: motan://10.20.3.15:3000/com.a.eye.skywalking.demo.services.GetUserService.findUser(String, String)
        - span.kind: server
        - component: Motan
        - peer.host: 10.218.9.86
    - [1] Span
      - spanId: 1
      - parentSpanId: 0
      - startTime: 2017-2-23 14:54:31.200
      - endTime: 2017-2-23 14:54:31.950
      - operationName: mysql/jdbi/statement/execute
      - tagsWithStr
        - span.layer: db
        - span.kind: client
        - component: Mysql
        - db.instance: mysql-instance
        - db.statement: select * from users where user_id = 1
        - peer.host: 10.5.34.18