Intro

In the current implementation of build scripts everything is tightly integrated into the current official Jenkins CI pipelines.
This makes creating images from source builds quite hard as if a user wants to fetch and build a package from source it needs to fetch build instructions from the Jenkinsfile's before commencing the build.
On a one packet scale this is often a manageable task, but on a "whole VyOS from source" scale this does not scale to a easily manageable solution.
This should be made much more efficient, so a user potentially could create a totally local image without even needing internet on the build device when the build commences (as long as the user have downloaded all sources locally, and yea it's not a realistic scenario as also debian-mirror and other 3rd party repositories needs to be made local).

Current build process

The current build process is like this: First fetch vyos-build repository of the version requested, then after the command to build have started all needed packages are fetched from a Debian repositories, VyOS upstream repository and other third party repositories. After this all local packages in vyos-build/packages/ are installed into the image at build-time offering the possibility to add custom packages to the resulting image generated.

This means that a official VyOS package repository will need to be available at build time to commence the build. this also means that to install any packages other than the latest package is not possible as the package repository only installs the latest version for each package.

The changes described here will not change the need of a official Debian mirror, or third party app mirrors, but only change the need for the internal VyOS packages installed on top of the Debian installed.

New proposal for source builds

These changes will be devided into multiple sub-parts thats tigthly integrated to create the final image:

1) Local package index in vyos-build

To allow a local build like this it is needed to create a index of what "custom" packages are needed to create a image of VyOS.
this could be created using eg. a yaml/toml that lists all packages needed and possibly a reference to what branch/git-tag/commit-reference is needed for a reproduce-able build. this file could also have custom build instructions and info on what files/deb's are needed to be fetched from the resulting build.(discussable)
example:

vyos-1x:
    type: git
    path: https://github.com/vyos/vyos-1x
    branch: current
    script: build.sh

vyatta-cfg:
    type: git
    path: https://github.com/vyos/libvyosconfig
    branch: current
    script: build.sh
......

This file should be extended with all repositoies needed to get a fully working VyOS image and if needed correct references to what version of the packages is being built.

An alternative is to make these build-indexes available for the different versions of VyOS, eg. a new index could be created at every LTS/sub-LTS release with references to each specific git-tag used on each package, but this is not a requirement. these indexes will also be dependent on the version of vyos-build beeing built, so this could be a hit-n-miss to get working if eg. the vyos-build scripts are update and a build is executed with an old index with eg. an old kernel.. maybe do you also need a specific version of the vyos-build repo to do a full build of an old image.

2) Split CI buildscripts into build and "orchestrational" changes (in each repository)

All Jenkins build scripts needs to be splitted, so the part of actually building the package is done in an external script,
for most cases this will be just to move the dpkg-build* commands into an external script that is executable outside jenkins but there are more advanced cases, eg the kernel build pipeline that needs more work done into these scripts.
This could be done in a Makefile, a approriate shell-script, or as a "blueprint" executed by the build-script noted in 3)

example of buildscript "handcrafted" from the vyos-build jenkins pipeline scripts:

#!/bin/sh
set -e
if [ -f debian/control ]; then
    sudo mk-build-deps --install --tool "apt-get --yes --no-install-recommends"
    sudo dpkg -i *build-deps*.deb
fi
dpkg-buildpackage -uc -us -tc -b

3) Source build script

To execute the source build pipeline, an "orchestrator" script is needed. This script will be in charge of:

• fetching the source packages from git using the index in 1) and put them into the correct places.
• execute the build scripts from 2)
• fetching the interesting parts of the build, eg. .deb files used etc. and put them into a directory usable by the image builder at the later stage.
• eventually cleanup after an old build

At this point you should end up with an index of .deb files that could be made into a local-apt-repository usable by the installer.
it will not be wise to install everything built like being done with the /packages/ directory, but instead a local apt repository should be created that is usable by apt directly while the normal VyOS build-scripts are executing.

Wrapup

The hardest parts here i think will be creating the orchestrator script, but as this is just fetching/copying files and executing commands it should not be that hard to execute. Testing this could also be done by removing the original official VyOS apt repository from the build and replacing it with the local repository created in the build.

This was my small plan of how this could be executed, but i would say that it for now have many loose ends and its much here that potentially could need discussions.
I'm all inn for any comments on this.