In a world where Dockerfiles, docker-compose.yml files, ansible files, and
others compete for my attention against tech debt in projects that become a bit
complex: I decided to make a Python micro framework to make it as nice to build
an infra as I think it is to build a CRUD with Django.

Shlax is meant to be a multi purpose automation framework, where logic to
acheive a certain state can be easily shared accross container building, host
provisioning, infrastructures with virtual systems and networks. Most of it is
not production ready yet, but the container building part seems to get me the
best I can get:

  • Distribution package host caching
  • Python and NodeJS host caching

No more waiting for all packages to download again, I’m not always near my
local mirrors

I also like that Shlax mounts user home pip and npm cache directories inside
the containers so that the cache is shared with my localhost.

Demo

You can play with :

git clone https://yourlabs.io/oss/shlax.git
cd shlax
pip3 install -e .[test,cli]
./shlaxfile.py build
py.test -sv tests

Caching walkthrough

The container build recipe looks like this:

shlax = Container(
    build=Buildah(
        Packages('python38', 'buildah', 'unzip', 'findutils', upgrade=False),
        Copy('setup.py', 'shlax', '/app'),
        Pip('/app'),
        base='quay.io/podman/stable',
        commit='shlax',
    ),
)

This is what it looks like on my machine with cache hits:

2020-10-27-173154_1342x200_scrot

Maybe should silent errors in the “Invalidating old layers” code.

So if I change the above build script to add the “most” package it will invalidate all layers:

2020-10-27-172742_1434x341_scrot

But reuse the local package cache (what a relief ..):

2020-10-27-173502_1412x802_scrot

Same story of course with a python package, for example adding the “django” package will make the build start from the Pip layer:

2020-10-27-173858_1435x240_scrot

But it won’t hit the network because it shares the same cache as my local pip command which is shared with the Pip action, you can see how it commits the layer at the end:

2020-10-27-173958_1427x623_scrot

And cleaner code which proceeds to umounts of mounts made by different actions, and commit:

2020-10-27-174023_1146x315_scrot

Read More

LEAVE A REPLY

Please enter your comment!
Please enter your name here

*

code