Logging to Seq from shell scripts
A lot of scripting can be involved in running an IT operation. While your business applications may be logging to Seq, is it any less important that you have visibility of key scripts? What happens when they fail and endanger your critical SLAs?
At a fundamental level, logging is really important. Often, shell scripts already have logging - but this is to a text log file trapped somewhere on disk. You might send it to a SIEM or other logging server for ingestion, but what if you could treat your shell script like any other application?
Enter seqcli, a multi-function and multi-faceted command line tool. I touched before on its functionality that I initially modelled against for Seq Reporter, but one of its most basic functions is logging to Seq.
As a .NET Core app, you can run seqcli on Windows, Linux, and OS X .. and it works great! I've integrated a number of Linux shell scripts to Seq using it. The key thing to remember is that Seq is at its best with well structured logs; exposing variables and properties to Seq for indexing and querying is always ideal. You can do that readily with seqcli ... and suddenly, getting an alert to something such as OpsGenie becomes simple!
In the sample script below, you'll note that I tend to try to pull in relevant environment variables from the Linux scripts for sending to Seq- this in turn allows script troubleshooting and debugging without having to directly log in.
Logging is only as good as you make it, of course - if you aren't logging each stage of a script (start, processing, end), for example, you won't necessarily find it easy to diagnose where something failed. And more information is better - okay, a file transfer failed, but what file? Where from? Where was it being sent to?
As a principle, try to standardise logging as far as possible. You'll see a familiar AppName property in the below script, which I use in Lurgle.Logging, the Log4j appender, Seq Reporter, Seq Client for Windows Logins, and Seq apps like Event Timeout, Event Threshold, and OpsGenie Heartbeat. This is valuable data that helps build signals, dashboards, and alerts - being able to differentiate between applications that log to Seq is important. Equally, I often include an Environment or MachineName property that allows differentiation between environments for similar reasons.
The logEntry function below is structured to default to Information events, with the ability to pass other log levels such as Warning or Error. It can be pasted into a shell script (eg. Bash, Ksh, etc), but it's always worth taking a look at what environment variables are being set. Including those as Seq properties is simple - just add another -p PropertyName="$VariableName" - and they're invaluable for debugging.
The command line can wind up quite long with this approach, but of course you can split it into multiple lines (like below) and I can't recommend doing this enough.
#------------------------------------------------------------## function <logEntry> :: Log to Seq ##------------------------------------------------------------#function logEntry {if [ -z "$3" ]thenerrorlevel="Information"elseerrorlevel=$3fiif [ -z "$2" ]thenmessageTemplate="{Summary}"elsemessageTemplate="{Summary} - {Description}"fi/etc/seq/seqcli log -l "$errorlevel" -m "{Summary} - {Description}" \-p Summary="$1" -p Description="$2" -p AppName="Linux Script Name" \-p Property="$Property" -p Property2="$Property2" -p Property3="$Property3" \-s https://seq.domain.com -a "<apikey>"}Property="Important Property"Property2="Stuff used in script"Property3="Relevant info"logEntry "An error occurred" "Stuff happened!" "Error"
Comments