Logging On Either Gcp Or Local

Suppose there is a system that is run on GCP, but as a backup, can be run locally. When running on the cloud, stackdriver is pretty straightforward. However, I need my system to pu

Solution 1:

Fortunately, this is a story that is handled. Stackdriver Logging is a very versatile framework for logging. However, there are a variety of logging APIs and Google's intent was not that you had to rewrite all your existing applications to leveraging the Stackdriver logging native APIs. Instead, you can use a logging API of your choice (including standard and defacto APIs) and these logging APIs will then map to Stackdriver. If executed outside a GCP environment or you simply wish to switch to an alternate log collector, your applications would not have to be re-coded or recompiled.

A list of the logging APIs available for different languages can be found at Setting Up Language Runtimes and this includes Setting Up Stackdriver Logging for Python.

For Python, at runtime, you have a configuration property (eg an Environment variable) that declares whether or not you wish to use Stackdriver. If set to true, then .. and only then ... would you execute the login that sets up the native Python logging for Stackdriver otherwise that logic would not be called and hence you would have no dependency on Stackdriver.

A possible piece of code might be:

if os.environ.get('USE_STACKDRIVER') == 'true':
  import google.cloud.loggingclient= google.cloud.logging.Client()
  client.setup_logging()

Solution 2:

You do not need to specifically enable or use Stackdriver in your program. You can use the Python logger and write to any file you want. However, Stackdriver only logs specific log files. This means that you would need to manually set up Stackdriver to log "your" log files.

In your example, you are writing to file.log. Modify /etc/google-fluentd/config.d/mylogfile.conf to include the following. You will need to specify the full path for file.log and not just the file name. In this example, I named it /var/log/mylogfile.log. This example also assumes that your logs start each line with a date.

<source>
  @typetail# Parse the timestamp, but still collect the entire line as 'message'
  format /^(?<message>(?<time>[^ ]*\s*[^ ]* [^ ]*) .*)$/

  path /var/log/mylogfile.log
  pos_file /var/lib/google-fluentd/pos/mylogfile.log.pos
  read_from_head true
  tag auth
</source>

For more information read the following document:

Stackdriver - Configuring the Agent

Now your program will run outside GCP and when running on a configured instance, log to Stackdriver.

Note: I would do the opposite of what you have asked. I would always use Stackdriver. When not running in GCP I would manually set up Stackdriver on my desktop, local server, etc and continue to log to Stackdriver.