Logging to Seq from shell scripts

A lot of scripting can be involved in running an IT operation. While your business applications may be logging to Seq, is it any less important that you have visibility of key scripts? What happens when they fail and endanger your critical SLAs?

At a fundamental level, logging is really important. Often, shell scripts already have logging - but this is to a text log file trapped somewhere on disk. You might send it to a SIEM or other logging server for ingestion, but what if you could treat your shell script like any other application?

Enter seqcli, a multi-function and multi-faceted command line tool. I touched before on its functionality that I initially modelled against for Seq Reporter, but one of its most basic functions is logging to Seq.

As a .NET Core app, you can run seqcli on Windows, Linux, and OS X .. and it works great! I've integrated a number of Linux shell scripts to Seq using it. The key thing to remember is that Seq is at its best with well structured logs; exposing variables and properties to Seq for indexing and querying is always ideal. You can do that readily with seqcli ... and suddenly, getting an alert to something such as OpsGenie becomes simple!

In the sample script below, you'll note that I tend to try to pull in relevant environment variables from the Linux scripts for sending to Seq- this in turn allows script troubleshooting and debugging without having to directly log in. 

Logging is only as good as you make it, of course - if you aren't logging each stage of a script (start, processing, end), for example, you won't necessarily find it easy to diagnose where something failed. And more information is better - okay, a file transfer failed, but what file? Where from? Where was it being sent to?

As a principle, try to standardise logging as far as possible. You'll see a familiar AppName property in the below script, which I use in Lurgle.Logging, the Log4j appender, Seq Reporter, Seq Client for Windows Logins, and Seq apps like Event Timeout, Event Threshold, and OpsGenie Heartbeat. This is valuable data that helps build signals, dashboards, and alerts - being able to differentiate between applications that log to Seq is important. Equally, I often include an Environment or MachineName property that allows differentiation between environments for similar reasons.

The logEntry function below is structured to default to Information events, with the ability to pass other log levels such as Warning or Error. It can be pasted into a shell script (eg. Bash, Ksh, etc), but it's always worth taking a look at what environment variables are being set. Including those as Seq properties is simple - just add another -p PropertyName="$VariableName" - and they're invaluable for debugging. 

The command line can wind up quite long with this approach, but of course you can split it into multiple lines (like below) and I can't recommend doing this enough.

#------------------------------------------------------------#
# function <logEntry> :: Log to Seq                          #
#------------------------------------------------------------#
function logEntry {
if [ -z "$3" ]
then
errorlevel="Information"
else
errorlevel=$3
fi

if [ -z "$2" ]
then
messageTemplate="{Summary}"
else
messageTemplate="{Summary} - {Description}"
fi

/etc/seq/seqcli log -l "$errorlevel" -m "{Summary} - {Description}" \
      -p Summary="$1" -p Description="$2" -p AppName="Linux Script Name" \
-p Property="$Property" -p Property2="$Property2" -p Property3="$Property3" \
      -s https://seq.domain.com -a "<apikey>"
}


Property="Important Property"
Property2="Stuff used in script"
Property3="Relevant info"

logEntry "An error occurred" "Stuff happened!" "Error"
And it's as simple as that. A function added to your script that calls the seqcli app on your machine. Obviously you need to download seqcli and place it in a location where the script has execute permissions - but it's reasonably easy to have a common and standard approach to your shell scripts that can help transform your operations with monitoring and alerting that really works.
 

Comments

You may also like:

Lurgle.Logging v1.2.0 - Multi-threaded correlation ids are now a thing

Implementing Multi-Threaded Lurgles If we revisit my example from earlier, we'll see a much simpler implementation for multi-threaded code; Log.Level().Add("Here is my log entry"); Log.Level(LurgLevel.Error).Add("Oh no! An error!"); Log.Level().Add("Phew ... moment passed"); Log.Level(correlationId: Logging.NewCorrelationId()).Add("After all that, I'd really like a different correlation id"); Log.Level(LurgLevel.Debug).Add("CorrelationId is {CorrelationId}"); You can see...

Structured Logging with Seq and Serilog

A few years back, I picked up an old "unloved" business application for document handling, and brought it into the modern era. I completed some work on adding automated OCR, running as a service, and then started to enhance it well beyond its original capabilities, such as moving a manual...

Lurgle.Logging - a standardised Serilog implementation with extra goodies!

Logging is important Logging is a really important, oft-neglected, aspect of business applications. I can't state that enough. If you don't have good logging, you can't troubleshoot and debug problems, and you have little chance of seeing what's actually going on in your enterprise. In Structured Logging with Seq and...