Poor Man's Microservice Configuration Using Environment Variables

tl;dr: This post shows a simple and tech stack neutral way to provide a configuration file for a microservice.

These days a common approach to microservice deployment is to ship them as a standalone binary package. The Uber Jars in the Java world are a prominent example. This simplifies operations - particularly when you are in a pre-Docker environment.
All it needs to run a microservice is a single Linux command line. As an old Linux guy I am delighted with this "back to the basics" approach.

A must-have feature for those kind of processes is the ability to configure the service via an external configuration file. Here, you often find the usual suspects like YAML, JSON or even INI.

This advice of the 12 factor manifesto made along existing config option more prominent again: good old environment variables.

This post shows a demo microservice which consists of three files:

• demoservice-starter.sh - starts demoservice.py and provides it with its configuration
• demoservice.cfg - the configuration file for this service consisting of shell variable definitions
• demoservice.py - the actual microservice (just a modified Flask "Hello World")

The service is started like this:

$ ./demoservice-starter.sh demoservice.cfg &

Outside to inside explaination:

demoservice-starter.sh 

Line 6 reads in the config file provided as command line argument. Technically the content of demoservice.cfg is parsed and executed.

Remarkable here is that the environment variables created in line 6 are just visible for demoservice-starter.sh and its children but not for the rest of the Linux system. In contrast to a user or system "profile" file containing global environment variables this is a decentral, scoped approach to provide environment variables.

Line 7 then starts the microservice. In this case I use Gunicorn as server for my little Flask application. I use "&" to send the gunicorn process to the background and continue the script execution.

Line 8 stores the process id of my just started process. This PID variable is needed two times. On line 9 we enter a "wait" state until the gunicorn process exits. This is basically a more sophisticated version of an endless sleep loop. The latter one works as well but it requires more code ;)

To stop the microservice we just kill demoservice-starter.sh. However, the shell does not kill our gunicorn child process automatically.

To retrofit this behaviour we have to quickly discuss what kill actually does. This is what the man page says:

kill - send a signal to a process. The default signal for kill is TERM.

So when we kill demoservice-starter.sh we actually just send the TERM signal to the script. What we need to do now is to forward this signal to our child gunicorn process.

This is what line 4 does: When the script receives a TERM signal it kills the gunicorn process which then lets our "wait" command continue to the end of the script.

A quick note on line 2: Here we enable two features of the bash shell. "-a" automatically makes variables defined in the script available for child processes. Without "-a" we would need to prepend each variable in demoservice.cfg with a "export" statement.

The other feature is "stop script on error" by using "-e". This is very useful also for build scripts to safe yourself from each time manually checking exit codes.

demoservice.py

There is not much to say here. When the route path "/" of our demo service is accessed via HTTP GET we use Pythons "os.getenv" function to read the content of the environment variables and echo their content. Remarkable here is the usage of Pythons Literal String Interpolation on line 10 which was introduced in version 3.6.

Bash syntax check

When automating your administrative tasks also shell scripts are checked in to a version control system. The built-in command line argument -n allows a syntax check of the shell script without executing it:

$ bash -n myScript.sh
$ echo $?
0

If the syntax check is successful nothing is displayed and the exit code is 0. However, if there is an error (like in the following case a missing parenthesis) an error message is printed and the exit code is 2:

$ bash -n myScript.sh
./myScript.sh: line 23: syntax error: unexpected end of file
$ echo $?
2

When checking your scripts in that way inside your continuous delivery pipeline it is easy to spot simple type errors.

Dependency Breaking in bash

In times of Devops you might want to also (unit) test your shell scripts. A collegue of mine introduced me to a nice trick how to do dependency breaking with bash.

The technique is based on the declare -f command which allows you to print the definition of previously defined function:

$ sayHello() { echo "hello world";}
$ declare -f sayHello
sayHello () 
{ 
    echo "hello world"
}

The following script sketches how to utilize this feature to replace an original function with a stub at run time:

#!/bin/bash

doComplicatedThings() {
    echo "i am doing many complicated things"
    # ...
}

codeOfOriginalFunction=$(declare -f doComplicatedThings)

# stub out "doComplicatedThings"
function doComplicatedThings { :; }

# executes stub (does nothing)
doComplicatedThings

# re-establish orignal function
eval "${codeOfOriginalFunction}"

# executes the original function
doComplicatedThings

The function doComplicatedThings is the production function which might have been loaded from an external script file via the source command. The definition of doComplicatedThings is then saved as a string into codeOfOriginalFunction.

The replacement function is just an empty stub. When executing the stubbed version of doComplicatedThings than as expected nothing happens. Calling eval restores the original function so that the second call of doComplicatedThings prints out the original message.

Stow Manager - A Simple Package Manager

I've just released on github a script for which adds some important features (like repository support) on top of the GNU stow package manager:

This is a working prototype of an application level based package manager based on GNU stow (www.gnu.org/software/stow/).

stow installs a specific version of a program from the so called stow dir to a destination directory.

The intent of this prototype was to find out if adding missing functionality to the core stow application (like repository support) would be a simple replacement for non-root installations of software on a Linux system.

If stow is a very simplified dpkg, then stow-manager is a very simplified (and dumb) version of apt-get.

See the github repository for the code (rather large bash script) and documentation.

Bash - Returning function status and error message at the same time

Fortunately the bash is Turing complete so one can do advanced programing task with it - if there is no other choice to pick a proper dynamically typed scripting language like Python, Ruby or Perl.

Quick detour: Much to my regret, in German Linux magazine long term author of the Perl snapshot series, Michael Schilli, admitted that over the past years Perl got less important. He's part the Perl comunity since a very long time, so this statement might be even more an indicator for looking at the newer kids on the block.

Anyway, I recently found my self again doing rather complex tasks with the bash. I was prototyping some logic which will eventually be implemented in some other (proper ;-) ) language. At some point I started combining two very basic things:

• returning true or false from a function ( return zero or non-zero)
• returning a string from a function ( via echo to the standard output of the subshell the function is running in)

Combining the two gave my some nice syntax which kind of resembles a common C programing pattern:

#!/bin/bash

function isGreaterThan() {
    local firstNo=$1
    local secondNo=$2

    if [ ! ${firstNo} -gt ${secondNo} ]; then
        echo "${firstNo} is not greater than ${secondNo}"
        return 1
    fi

    return 0
}

# main
errorMsg=

if errorMsg=`isGreaterThan 3 1`; then
    echo "here comes more logic"
else
    echo "got error: ${errorMsg}"
fi

The function isGreaterThan returns the result of the evaluation as normal return codes. Additionally, in case of a false an error message is given back. One level above the error message is received by calling the function in a subshell (I prefer the old backticks syntax over the new brackets notation).

This nice construct comes with the usual limitations of a subshell operation - the code inside the subshell can read the outer variables but can't alter them.

Embedding Online Help Inside Shell Script

I recently posted an article on how to document a shell script using the Plain Old Documentation (POD) format and then produce a nice looking HTML document from that.

Based on that idea I want to show you this time how to utilize the in-line POD documentation to generate a professionally looking online help. In detail, that is a man page which is being displayed if the user provides the argument help to the shell script (or any other argument you choose for that purpose).

Since the last post already explained all the required background on embedding POD in shell script, I start straight with the example code:

#!/bin/bash

## =head1 Name
##
## pod_example.sh - A Demo Script
##
## =head1 Synopsis
##
## pod_example.sh [help]
##
## =head1 Description
##
## This script demonstrates embedding online help
## into a shell script.
##
## =head1 Details
##
## The script calls two functions in the following order:
##
## =over
##
## =item *
##
## firstFunction 
##
## =item *
##
## secondFunction
##
## =back
##
  
SCRIPT_VERSION="0.1"

firstFunction() {
    echo "i'm the first function"
}

secondFunction() {
    echo "i'm the second function"
}

showHelp() {
     local podData=$(cat $0 | egrep '^##' | sed 's/^##\s\?//g')

     if [ "${podData}" ]; then
         echo "${podData}" |                             \
         pod2man --center="A Demo Script"                \
                 --date="$(date +%D)"                    \
                 --name="Demo"                           \
                 --release="Version ${SCRIPT_VERSION}" | \
         man -l -
     fi
}

# main part
if [ "$1" = "help" ]; then
    showHelp
    exit
fi

firstFunction
secondFunction

This example script starts with the documentation in POD format. Since the first line of function showHelp extracts the POD documentation from the shell script it is up to you where to place your documentation - as long as your POD data starts with double hash (#). Again, see the previous post for details of the first line of showHelp.

The interesting and new things are happening from line 47 on. Here, we send the plain POD data to the converter pod2man which is part of the standard perl package. The optional arguments supply the text surrounding the online help (see screenshot below).

To no surprise the man data generated is feed into the man program via the standard input (line 52). Option "-l" enables the support of local man pages, "-" indicates that man data is coming from STDIN.

That's the whole magic - made possible by the beauty of Unix pipes. The result of calling "pod_example.sh help" should look very familiar to any Unix nerd:

Documenting Shell Scripts

Intro

I recently was involved in the development of a rather large library of shell script functions. Although I'm aware that one should pick proper scripting languages for more complex programing task ( like Python, Perl, Ruby...) - sometimes you simply don't have a choice.

Abusing Doxygen

On issue was to find a way to generate an API like documentation straight out of the shell script code. Searching the Internet I mostly found obscure ways of using (not saying abusing) Doxygen for that task (here and here). For me that sounded too complex and confusing - I was only trying to add some comments with markup which eventually would turn into some HTML documentation file.

Plain Old Documentation In Shell Script

After a while I found this idea of Milivoj who is using the well proven Plain Old Documentation (POD) markup for documenting shell scripts. POD is the markup used in the Perl world for source code documentation.
Milivoj is utilizing HERE documents to embed POD markup in shell script which is in my opinion as close as you can get to the original Perl way. However, I decided for a simpler, at the same time more shell like approach on embedding POD code.

The POD format

This is an example of a file containing POD markup:

=head1 Important heading

This is inside the important heading section. 

  this is example code inside important heading.

=heading1 Another important heading

This is another important heading content.

=head2 This is a sub heading

This is the content of the subheading.

POD2HTML

Running the following command the POD markup is converted into a HTML page:

cat pod_example.pod | pod2html > pod_example.html

The tool which performs the magic is pod2html which is part of the standard Perl distribution. See the manual page for more details. However, I was more than  happy with the defaults:

Example HTML page, generated from a POD file using pod2html

 

Embedding Into Shell Script

The next step was to include the POD markup somehow in the shell script library. The Perl way of including POD markup into source code is by simply starting with a "=" tag (like =head1 ...) and closing a documentation section with the special tag =cut. After this last tag normal Perl code starts again. Since the Perl interpreter is aware of the "=" tags POD code can be directly inserted into the source.

Unfortunately, the shell interpreter ( in my case bash and ksh are possible) does not know about "=" tags so I needed to look for a different way to embed POD code in shell script. As already mentioned, Milivoj's approach is to use HERE documents to mimic the Perl embedding of POD markup which I felt is smart but hard to read in a large code base.

After thinking for a while I came up with a less smart but more readable solution (as always, in my opinion ;-) ). Here is an example how I use POD markup inside the shell script library:

## =head1 Intro
##
## This is a library with useful shell script functions.
##
## =head1 Functions
##
## This section contains the functions available.
##
## =head2 firstFunction "an argument"
## 
##  # example usage
##  firstFunction "some argument"
##
## This is the first function which accepts one argument.
##
firstFunction() {
    typeset arg1=$1
    ...
}

## =head2 secondFunction "an argument"
##
##  #example again
##  secondFunction "argument"
##
## Also the second function is important.
##
secondFunction() {
    typeset arg1=$1   
    ...
}

# i still can write normal, non-double hash comments
...

The double-hash comments are reserved for POD markup, normal, single hash comments keep their original purpose. I prefer this style of embedding over the HERE document variant since (as with e.g. Javadoc) documentation consists only of specially formated comments, nothing else.

To convert the shell script into an HTML documentation the command line is now slightly longer:

cat pod_example.sh | egrep '^##' | sed 's/^##\s\?//g' | \
pod2html > pod_example.html

egrep is used to only filter out POD markup lines (beginning with double hash), sed is then removing the double hash and if existing the following white space. The result is plain POD which is directly fed into pod2html.

The result is as nice as always:

Example HTML, this time generated out of POD markup inside a shell script