Subtitles for LinuxDays 2016 - Building a free software based development environment - F. Preinstorfer

14   0   0  
  Pause sub
so thank you for coming to my
presentation I'm going to speak about
building a free software development
environment or pipeline for a small
company basically it's some of what we
have and we built during the last three
years at our company
my name is growing plants tougher I'm a
systems engineer at a small Austrian
startup company and we do laboratory
testing equipment for pulp and paper
industry so basically we build devices
that we sell to customers and those
customers use it to test certain aspects
of for example paper so one test case
would be how much water can a piece of
paper absorb the kind of devices nothing
particular fancy from a technology point
of view we're using mostly c++ so most
of the soft to third of it is written in
c++ then there is a Python 3 and
obviously the glucose that you need
everywhere would basically passion and
all the 15
did you know before want to actually
talk about the pipeline i would like to
introduce some of the requirements that
we had and those are just some of them
I think it's important to get your
feeling of of what is required at least
for our use case so one important
requirement was to have offline support
our company headquarters is located in
Austria at the countryside and that is
actually nice if you look at the window
and this you can see green grass in
mountains and basically that's the case
but it's not beneficial in case you want
proper internet connection so we have a
slow internet connection and the rather
flick you want to so we better not rely
too much on the internet that would be
it would be good for us
another thing that we know from the past
it that is quite beneficial to not
depend on any single developer machine
somewhere I'm sure most of you have
the stories or have seen it with your
own eyes that some laptop some machine
in some office from some developer is
actually the most important aspect of
the company because without those
without this laptop certain products can
be built anywhere
I don't know if anyone's so all things
like that but I've seen them and they're
not that great to have so we better
prepare for it
another thing is it should be easy to
use and easy to use means we should use
tools and we should provide developers
tools that they are comfortable with
so if we're using get it would be great
if the developer can interact with a
pipeline and orchestrate the pipeline or
most of the pipeline by just using get
fair enough
and another thing is it should be
understandable i think it's quite
important for a built environment that
that users can understand it they don't
need to get get all the Eternity great
details of it but you should take a look
at it and they should say okay I know I
built this package and it went from this
machine to this machine this machine and
this happened that would be that would
be fair enough
and also having an understandable
pipeline means it is also a hackable one
so it can be can be easily modified and
the adjusted as needed i want to give
you a very minimal overview of what we
it's far away from being complete but it
contains the most important aspects that
are needed for this 20-minute talk so
I'm gonna give a short introduction to
all of the parts here and then I will
provide you with some use cases in order
to get your feeling of of what is
possible so we want to start at the
bottom left here where we have a
developer laptop could be anything that
linux machine windows box mac box
whatever there's no certain requirement
on such machine basically the developers
should feel comfortable
with getting his work done so maybe 90
maybe maybe max and some version control
system would be would be enough to get
started from there we we decided to to
pick get lap as our as a repository
management server or socialism version
control system because we use it
internally a lot of we like it and we
think it's it's here to stay so we read
so we figured it would be great if if we
have something that obviously supports
get and also allows developers to use a
weapon to face if they want to because
some developers don't feel that saving
with with with a command line and maybe
some tasks are quite easy to do on such
a web interface so we figured it would
be a good lap would be would be a great
tool here and we're quite happy with the
choice actually back then when we
started it was it was a bit rough but
right now it feels it feels quite quite
quite good to use from there we we head
over to tank is here in the middle one
here we selected Jenkins because I was
used to it and mike / process whatever
ok and it's free software it's okay
using Jenkins sometimes might you
Jenkins might give you some headaches if
you if you don't use it as a master only
system so you if you want to use Jenkins
you better make sure you use build notes
and do not build anything on Jenkins
itself let Jenkins to all the
orchestration and all the other
reporting and crafting and there would
be there would be enough but do the
builds on other machines and that's
where we have several build nose right
here the bill stuff they build c++
project python projects they run unit
tests they do integration test whatever
is needed
from there we had over to a plea and a
plea is at Debbie n version compatible
um repository server so we decided to
ship our product as a set of Debian
packages and it is most natural and 22
ship Debian packages in a and in a
regular debian repository and so you
need some tool to handle packages and
some of them are available and we stick
with a plea because we think or in our
experience it fits our use case best so
it's I think it's written and go
mature i think so and it's quite handy
to use he has a lot of features and i
think you get most of the book actually
done on the right side of evaporate the
right-hand side of it here you we have a
short the dotted line that means some
network segregation so we have most of
of this poplin in our internal network
or actually all of it most of it and on
the right-hand side on the external
network we have some external mirrors
and that is basically engine X so engine
ax is serving of the repository for our
customers and we have multiple instances
of of that here so yeah I left out some
of the funny stuff actually and some of
the useful stuff for example we also
have a translation server so translators
can actually translate our entire
software using a single single web
interface we use a website for that but
it's it's actually nice it provides your
twitter bootstrap web interface it's
quite easy to use but it's not so
important for for this talk but just to
give you an idea that this is really the
most important components of of the
poplin so I want to get started with a
use case and this use case is to build
and release
a stable piece of software so we assume
that the developer here on his laptop
has a single repository and it contains
a stable release of software and it also
has all the debian package information
in it so this repository is basically
ready to build ok so what a developer
needs to do is basically commit his to
commit the current state of the
repository to a release branch in this
case i call it release now obviously
there can be many more but for this
example releases good enough and from
there all the developer needs to do is
push this repository to get love and get
up is not only the the entrance to this
pipeline but also needs to or is the one
machine that is actually responsible for
archiving all the source code so if you
have commits on a release branch and and
on a branch it could potentially be
shipped to a customer
it would be very advisable if you make
sure that this commit never ever goes
away again
so get that provides you with a feature
that is called protective branches and
you can configure it on a project basis
on a project basis and it allows you to
protect certain branches from being
deleted and from being forced push to so
that developers can can weed out can
weed out branches and from there we had
over 22 Jenkins basically get lap
notifies Jenkins about changes to
certain repositories by using using that
book so this is a stone that the
standard get lab API feature it
basically notifies Jenkins hey Jenkins I
have some changes here maybe you want to
do something with it
so Jenkins on his side has a huge list
of projects that it builds it builds for
example c++ project for release branches
it builds python project as as nightly
versions what-whatever Jenkins basically
knows how to build stuff so it takes the
information from good lap and notices ok
this is a c++ project and it is actually
coming from a release branch so I should
build a release basically so Jenkins a
few steps and they are basically
selecting one of the build not providing
a built a clean build environment we do
not share any built environment across
bills so we make sure that each software
each packet is built-in are in a clean
and minimal environment if you're
interested we are using debian Jenkins
clue for that it's a nice set of
basically shell script that abstract
away a lot of the nitty and gritty
details of debian packaging using when
you use Jenkins Jenkins bills the
package or built the source bills to
build the binary packages runs tests on
the packages and after build node is
done it collects all the results it
collects the builder artifact and it
provides the user with something like
graphs and reports that's what everyone
wants to have so this is the actual
place where developer can look at the
current state of this project so you can
take a look at it and say okay this is
the twentieth build up a package its
builders clean we have 20 unit tests and
and he can he can quickly take a look at
that and at the state of it and this is
also the point where you can customize
quite a lot so if you for example built
oxygen documentation there are some some
warnings and oxygen or in the oxygen
lock that you want that you might want
to take a look at and this reporting and
and graphing feature of Jenkins would be
the right place to to visualize such
changes such details so let's assume
that the building successful and one of
the build artifacts is actually a binary
Debian packages or debian package from
Jenkins takes a package and throws it to
a plea and a plea takes a look at the
incoming package and the building
information that is delivered with it
and notice notices
ok this is a this looks like a release
package because it's coming from a
release branch from the original get
committed to get release branch so it
says okay I know how to deal with
releases I'm going to take it and put it
into my unstable distribution or
repository for that for that matter so
this means that a single or all packages
that are built are basically stuffed
into one single ups repository a plea
distribution we have several of those
and I want to just very quickly get over
or over the different distributions that
we have we have for example unstable we
have a testing and we have a stable and
those serve different use cases so the
unstable one is is the distribution that
packages are thrown into all the time so
if the developer makes a new release of
any package it is stuffed into unstable
now at some point in time you basically
want to ship software to customers
otherwise you would be soon out of
business so at one point in time I
developer needs to decide okay this is
good enough to basically get into
testing and testing means that our
testing Department of QA department can
start to take a look at the current
state of the
auditory so the team can for example
test whether installs from this
repository work if upgrades work if the
packages are complete if the features
are working if all the boxes that are
required are actually in and yeah
from there we require the q18 to
basically start the Jenkins job and Mark
the testing distribution as stable and
stable means that it is mirrored to the
external repositories to the external
mirrors so once a QA member says okay
this this version of the testing
repository is good enough it can be
shipped to customers he can market a
stable and everything is dancing outside
outside of the repository so this is a
complete rundown on the on the entire
pipeline and I want to just quickly get
on with one or two other use cases as
well to give you an impression of of
what is what is possible
so before I want to quickly sum up we
have a stable arm source package with
debian packaging information in it we
sent this package to get get get lap
getting noticed Jenkins Jenkins build
software we have a a binary debian
package that is in present a plea and
once it drips down through the pipeline
through unstable testing and stable it
sooner or later we'll hit the customer
at some point and usually when that
happens the customer will report box
hopefully he does in our case they do so
let's go on with another example and
let's assume ok such customer reports a
buck and let's assume that ok
a man pages missing from a debian
package those bucks usually don't get
reported but let's assume for this
example that it is so the developer is
tasked with ok please at the mom page to
this package and now he does he creates
a deaf branch basically deaf and pick
any name could be deaf packaging for
example and then he starts to work on
the package and he now could obviously
build everything by himself he could
build a been packaged locally here on
all the unit test locally Picard check
the code coverage locally to all the
things but there's no need to because he
can also push this staff this death
packaging branch here just to get up
again and let love do the work for him
or do the latter pipeline do the work so
get up once again notice Jenkins Jenkins
knows how to build packages it comes
from another branch but doesn't really
matter so it builds packages and again
Jenkins crafts and reporting so the user
can quite quite easily spot for example
incheon warnings and lynching is a tool
that you might want to use in case
you're using Debian packages it is a is
a linter for David packages and it can
notify you of things like okay there's a
binary in this package and it is missing
a man page so let's assume this morning
is is not gone and the bill is to build
its ok so besides the graphs and
reporting we also have to have the
debian package and at this point it is
again thrown against a plea and ugly
says okay it's a it's a debian package
it's actually nice but it is coming from
some branch that I don't know it looks
like it's coming from def packaging
I'm not taking it so it basically throw
it away so one might say okay this is
this is a rather pointless example
yes but it is the foundation of building
other things on top that are the turn
out to be quite useful once you have
them for example using this this
pipeline and using the single point a
plea to decide whether or not packages
should hit the repository allows you to
do things like okay we can do nightly
rebuild of all Debian packages at any
time did never will hit any repository
but we can spot things like changing a
pis or gone packages or breakage of
packaged with with such such cases
another interesting factors and I want
to just very quickly jump on that
because I'm already over time but just
to give you an impression
it also allows you to to provide use
cases such as why not create a special
distribution on the applet server that
is called def packaging and if we hit
such a different positive if if a
package hits such a repository it could
go to death packaging is this
in this example so it doesn't go too
unstable it does go to have packaging
and from there we have a regular cab and
repository so it can be used internally
by QA department can be used by by
developers itself and this allows for
example to do big refactorings without
breaking unstable for two months or
three months because the developer can
pick any source package that he wants or
any repository that he wants hack on it
and push to commit not not to release
branch but to a known development branch
and all the packages that fall out of
this known development branch hit this
very repository and never
ever go to customers but that would
allow us some some great internal
testing and once the refactoring is done
the work can just simply be merged to
release practice and you have the first
use case again so obviously my time is
over right now I want to just very
quickly go over to go over several
lessons that we've learned
maybe they're useful to you I would be
great if I had known them before the
first one is you better make sure you
have a really easy to use pipeline and
easy-to-use means for developer point of
view if you using get in your company
make sure that the pipeline is
controllable and usable by get only so
if you have if you require your
developers to setup Jenkins for their
own software building that might be an
issue because some developers even if
it's company policy that Jenkins should
be used for building some developers
will say yep I ignore it
fine others will say yes i'm going to
take Jenkins for building but I've never
heard of unit tests and I don't care
about graphs about unit tests ok fine
other ones will say okay I know how to
how to analyze che unit test results
with Jenkins that's good and he's happy
but he forgot that there is for example
code coverage as well so it would be
great to make sure that that you can
abstract away from the developer and you
do not require a developer to configure
Jenkins he can take a look at it
no problem but basically it's better to
provide for him or her i think it's
important to get services up and running
real quickly so don't don't build all
the features that you think are
important or useful but start with the
most important one and provide that
requirements to change over time mmm
they change very often more often and
you better prepare yourself for that and
I think the best preparation for
changing requirements is to automate as
much as possible and automation means ok
I'm use ansible use puppets chef
whatever you have to do your server
configuration but do not stop there
because for example get loud provides an
API were all the projects con can be
configured programmatically so there's
no need to go through the settings of
several hundred projects and require and
and edit group membership or edit hooks
or something like that that can be
automated away and it should be
automated way
another thing is if you're using Jenkins
there's projects like Jenkins shot
builder I don't know if it's if people
know that but i think it's quite useful
because we have a few i'm not about
thousand checking shops and changing
single parts of the job requires us a
minute do that by hand and you could
have fun for the next two weeks so if
you're using set using Jenkins make sure
you never touch the configure interface
of it except for testing obviously and
the last point is you better choose your
tools wisely
we started with another repository
server and without matchup much much
evaluation because we thought okay we've
known it from the past and it's ok it it
does his job but we've hit limitations
quite quickly and actually when you
build such a pipeline integrations come
over time and you think okay this is
just the one gets hurt at them the one
debian repository server it can't be
that hard to rip that up but actually if
there are five other services that
depend on it and somehow integrate with
it it's get it gets tricky
ok so that's it from my talk thank you
very much for listening
if you have any questions
you might ask them their informant four
minutes or you can find the contact
details about my webpage
ok thanks
yes please
look below i wanted to ask
aside from these developer tools are
there any other pieces of free software
to use from for our other aspects of
your business like accounting core
customer relations maybe or project
management excellent question
we are currently migrating away from a
proprietary enterprise resource planning
system to a free software one it is
called Triton I don't know if you if you
know that right on would be at least the
tool of choice for us right now i think
i'm not sure i think it's called new
health that is built upon tried on so
you might want to look for a new health
to get demos and have a look at what was
yes please i need for presentation and
review some unified to branching model
educate something like it for sort of
that we have a we do have a branching
model yes and it is based on release
names internally so it is the one that i
presented here is actually quite a
minute fide one we have unlimited hard
to explain
so basically we have we have releases
every few years or it may not release is
every few years that out that have names
and below this name we have released
branches and we have development
branches and we have packaging branches
and all the kind of stuff so yes we do
and if you're interested come to me
afterwards and i will explain in detail
to you
yep welcome
yes please yeah how do you deal with
like even have a really old teacher been
like three weeks old
do try to integrate it with the band
released like weekly or just like try to
merge these old branches the new release
which more features added by the unique
in the way and which can cause other
kind of box
ok so the question is when we when we
have such long development branches that
we use and then we will be packaged
separately whether we merge them back
and actually yes we do we basically
rebase so whatever hits the development
branch doesn't matter for us so we r
there are many many committees that are
just working progress stuff like that
and once they get back to master or to
release branch they're usually squashed
into logical commits with proper commit
messages and then the entire pipeline is
rebuilt or the software is rebuilt so
what we don't do is we don't take
development Debian packages that hit a
development repository and move it into
the unstable or stable repository so we
make sure that all builds actually use
the path with releases in there so we
never take binary Debian packages for
deaf for whatever we all we always
rebuild the kind of stuff

Share this subtitle


This talk will cover how to power the development infrastructure of a small (non-cloud) company using free software. I present the goals and our requirements for the infrastructure and dig into some interesting parts, including: * Using a free software pipeline to get from code to the final product. * Discuss the use of free software alternatives to proprietary solutions. * Why the work on infrastructure is never finished. * Lessons learned