Tracing A Python Grpc Server Deployed On Cloud Run With Opentelemetry

I'm running a Python gRPC server on Cloud Run and attempting to add instrumentation to capture trace information. I have a basic setup currently, however I'm having trouble making

Solution 1:

It turns out that my configuration wasn't correct - or, I should say, it wasn't complete. I'd followed this basic example from the docs for the Google Cloud OpenTelemetry library, but I didn't realize that manually instrumenting wasn't needed.

I removed the call to tracer.start_as_current_span in my gRPC method, installed the gRPC instrumentation package (opentelemetry-instrumentation-grpc), and added it to the tracing configuration step during startup of my gRPC server, which now looks something like this:

from opentelemetry.instrumentation import grpc as grpc_instrumentation
from common import tracing # from my original questiondefmain():
    """Starts up GRPC server."""# Set up tracing
    tracing.configure_tracing()
    grpc_instrumentation.GrpcInstrumentorServer().instrument()

    # Set up the gRPC server
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=100))
    # set up services & start

This approach has solved the issue described in my question - my log messages are now threaded in the expected manner

As someone new to telemetry & instrumentation, I didn't realize that I'd need to take an extra step since I'm tracing gRPC requests, but it makes sense now.

I ended up finding some helpful examples in a different set of docs - I'm not sure why these are separate from the docs linked earlier in this answer.

EDIT: Ah, I believe the gRPC instrumentation, and thus the related docs, are part of a separate but related project wherein contributors can add packages that instrument libraries of interest (i.e. gRPC, redis, etc). It'd be helpful if it was unified, which is the topic of this issue in the main OpenTelemetry Python repo.

Solution 2:

While reviewing Google Documentation of OpenTelemetry using Python, I found some configurations that could help with the issue of tracing the correct ID. Additionally, there is a troubleshooting document to view traces in your Google Cloud Project when you expect trace data to be present.

Python-OpenTelemetry - https://cloud.google.com/trace/docs/setup/python-ot

Google Cloud Trace Troubleshooting - https://cloud.google.com/trace/docs/troubleshooting

For secure channels, you need to pass in chanel_type=’secure’. It is explained in the following link: https://github.com/open-telemetry/opentelemetry-python-contrib/issues/365

You need to use the x-cloud-trace-context header to ensure your traces use the same trace ID as the load balancer and AppServer on Google Cloud Run, and all link up in Google Trace.

The code below works to see you logs alongside traces in Google Trace’s Trace List view:

from opentelemetry import trace 
from opentelemetry.trace.span import get_hexadecimal_trace_id, get_hexadecimal_span_id 
       current_span = trace.get_current_span() 
       if current_span: 
          trace_id = current_span.get_span_context().trace_id 
          span_id = current_span.get_span_context().span_id 
          if trace_id and span_id: 
              logging_fields['logging.googleapis.com/trace'] = f"projects/{self.gce_project}/traces/{get_hexadecimal_trace_id(trace_id)}"
              logging_fields['logging.googleapis.com/spanId'] = f"{get_hexadecimal_span_id(span_id)}" 
              logging_fields['logging.googleapis.com/trace_sampled'] = True

The documentation and code above were tested using Flask Framework.