Give it a Try

Warning: Contractor is made to solve complex problems, for the most part it presents a must less complex interface to you. That being said, there is a bit of a hurdle to get it up and running, depending on what you are trying to automate. So, buckle in, this is going to be a lot of fun.

This demo will result in a VM in a private /24 network that can create and destroy other VMs in that network. You can use either VCenter or VirtualBox. After the Contractor VM is up and running, you can install more blueprints and plugins to do docker or other foundations. The requirement for the private network comes primarally from the setupWizzard’s default configuration, if you feel comfortable with modifying the setupWizzard and the Apache Configurations, you can use blueprints and plugins for hosted providers such as AWS and Azure.

NOTE: setupWizzard is going to re-write some bind config files, so don’t edit them until after the install is complete.

Installing

Create an Ubuntu Xenial VM, name it contractor, set the fqdn to contractor.site1.local Ideally it should be in a /24 network. Offset 1 is assumed to be the gateway. All these values can be adjusted either in the setupWizzard file before it is run, or after it is setup, you can use the API/UI to edit these values. The DNS server will be set for the contractor VM, and bind on the contractor vm will be set to forward to the DNS server that was originally configured on the VM.

From Source

From Source

From Packages

Prebuilt Packages

Setup

Set up Infrastructure

To prevent your new contractor VM from taking over the network you are curently connected to, you will need to configure a issloated network. There are also some other Infrastructure related tasks that need to be done.

Vcenter/ESX Setup

VirtualBox Setup

Install Requred Services

Install Postgres:

sudo apt install -y postgresql-client postgresql-9.5

Create the postgres db:

sudo su postgres -c "echo \"CREATE ROLE contractor WITH PASSWORD 'contractor' NOSUPERUSER NOCREATEDB NOCREATEROLE LOGIN;\" | psql"
sudo su postgres -c "createdb -O contractor contractor"

NOTE, for thoes two command you will see:

could not change directory to "/root": Permission denied

this is from the su command, thoes messages can be ignored

Update Packages

the ubuntu toml package is to old, update with:

sudo apt install -y python3-pip
sudo pip3 install toml --upgrade

if you are using the virtualbox plugin, you will need zeep:

sudo pip3 install zeep

for vcenter/esx install pyvmomi:

sudo pip3 install pyvmomi

Configure Apache

We will need a HTTP site to serve up static resources, as well as a Proxy server to bridge from the issloated network. This proxy server will also cache, that will make things install faster the second time.

First create the directory for the static resources:

mkdir -p /var/www/static

now create the proxy site /etc/apache2/sites-available/proxy.conf with the following:

Listen 3128
<VirtualHost *:3128>
  ServerName proxy
  ServerAlias proxy.site1.local

  DocumentRoot /var/www/static

  ErrorLog ${APACHE_LOG_DIR}/proxy_error.log
  CustomLog ${APACHE_LOG_DIR}/proxy_access.log combined

  ProxyRequests On
  ProxyVia Full

  CacheEnable disk http://
  CacheEnable disk https://

  NoProxy static static.site1.local
  NoProxy contractor contractor.site1.local
</VirtualHost>

now create the static site /etc/apache2/sites-available/static.conf with the following:

<VirtualHost *:80>
  ServerName static
  ServerAlias static.site1.local

  DocumentRoot /var/www/static

  LogFormat "%a %t %D \"%r\" %>s %I %O \"%{Referer}i\" \"%{User-Agent}i\" %X" static_log
  ErrorLog ${APACHE_LOG_DIR}/static_error.log
  CustomLog ${APACHE_LOG_DIR}/static_access.log static_log
</VirtualHost>

Modify /etc/apache2/sites-available/contractor.conf and enable the ServerAlias line, and change the <domain> to site1.local

Now enable the proxy and static site, disable the default site, and reload the apache configuration:

sudo a2ensite proxy
sudo a2ensite static
sudo a2dissite 000-default
sudo a2enmod proxy proxy_connect proxy_ftp proxy_http cache_disk cache
sudo systemctl restart apache2
sudo systemctl start apache-htcacheclean

Setup the database

Now to create the db:

/usr/lib/contractor/util/manage.py migrate

Install the iputils functions, this contains the port check function contractor will use to verify the OS has booted:

sudo respkg -i contractor-plugins-iputils_0.3.1.respkg

Install base os config:

sudo respkg -i contractor-os-base_0.1.respkg

Now to enable plugins. We use manual for misc stuff that is either pre-configured or handled by something else:

sudo respkg -i contractor-plugins-manual_0.3.1.respkg

if you are using esx/vcenter:

sudo respkg -i contractor-plugins-vcenter_0.3.1.respkg

if you are using virtualbox:

sudo respkg -i contractor-plugins-virtualbox_0.3.1.respkg

do manual plugin again so it can cross link to the other plugins:

sudo respkg -i contractor-plugins-manual_0.3.1.respkg

Now to setup some base info, and configure bind:

sudo /usr/lib/contractor/setup/setupWizzard --no-ip-reservation --dns-server=10.0.0.10 --proxy-server=http://10.0.0.10:3128/

It is safe to ignore the message:

rndc: connect failed: 127.0.0.1#953: connection refused
WARNING: "rndc reload" failed

Bind (the DNS server) is not running yet, it will be started later.

And now to create a user for us to login as for the API calls:

/usr/lib/contractor/util/manage.py createsuperuser

that command will ask for a username, email and password. The email address does not need to be a real address.

Environment Setup

We will be using the HTTP API to inject new stuff into contractor. You can run these commands from either the contractor VM, or any place that can make http requests to contractor.

we will be using curl, make sure it is installed:

sudo apt install -y curl

First we will define some Environment values so we don’t have to keep tying redundant info the Contractor server, this is assuming you will be running these commands from the contractor VM, if you are running these steps from someplace else, update the ip address to the ip address of the contractor vm. Replace < network name > with the name of the network created in vcenter (ie: internal) or virtual box (ie: vboxnet0):

export COPS=( --header "CInP-Version: 0.9" --header "Content-Type: application/json" )
export SITE="/api/v1/Site/Site:site1:"
export CHOST="http://127.0.0.1"
export ADRBLK="< network name >"

now we need to login, replace < username > and < password > with the username and password you specified API user (the createsuperuser step):

cat << EOF | curl "${COPS[@]}" --data @- -X CALL $CHOST/api/v1/Auth/User\(login\)
{ "username": "< username >", "password": "< password >" }
EOF

which will output something like:

"k4of9zewijvze0gf72ylb6p6zxv4srol"

which will return a auth token, save that to our headers, replace < username > with the API username, and < auth token > with the result of the last command:

COPS+=( --header "Auth-Id: < username >")
COPS+=( --header "Auth-Token: < auth token >" )

Let’s make sure our login is working:

cat << EOF | curl "${COPS[@]}" --data @- -X CALL $CHOST/api/v1/Auth/User\(whoami\)
{}
EOF

that should output your username, for example:

"root"

Network Configuration

The setupWizzard has pre-loaded the database with a stand in host to represent the contractor VM and has flagged it as pre-built. It has also created a site called site1 and some base DNS configuration. It also took the network of the primary interface and loaded it into the database named ‘main’.

We need to create another address block for the internal network:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Utilities/AddressBlock
{ "site": "$SITE", "name": "$ADRBLK", "subnet": "10.0.0.1", "gateway_offset": null, "prefix": "24" }
EOF

which should output something like:

{"gateway_offset": null, "_max_address": "10.0.0.255", "size": "254", "created": "2019-02-23T14:15:06.830987+00:00", "isIpV4": "True", "netmask": "255.255.255.0", "site": "/api/v1/Site/Site:site1:", "gateway": null, "prefix": 24, "name": "internal", "subnet": "10.0.0.0", "updated": "2019-02-23T14:15:06.830966+00:00"}

Now to add the internal ip of the contractor host, first set the address on eth0 to non-primary, we want the internal ip to be primary:

cat << EOF | curl "${COPS[@]}" --data @- -X UPDATE $CHOST/api/v1/Utilities/Address:1:
{ "is_primary": false }
EOF

result:

{"netmask": "255.255.255.0", "offset": 126, "updated": "2019-03-05T03:16:00.142926+00:00", "prefix": "24", "pointer": null, "networked": "/api/v1/Utilities/Networked:1:", "vlan": 0, "ip_address": "192.168.13.126", "is_primary": false, "interface_name": "eth0", "address_block": "/api/v1/Utilities/AddressBlock:main:", "created": "2019-03-05T02:45:12.304186+00:00", "gateway": "192.168.13.1", "sub_interface": null, "type": "Address", "network": "192.168.13.0"}

create an interface eth1 for the ip to belong to (this represents the new interface we created on the internal network):

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Utilities/RealNetworkInterface
{ "foundation": "/api/v1/Building/Foundation:contractor:", "name": "eth1", "physical_location": "eth1", "is_provisioning": false }
EOF

result:

{"is_provisioning": false, "foundation": "/api/v1/Building/Foundation:contractor:", "name": "eth1", "updated": "2019-03-05T03:18:13.581612+00:00", "mac": null, "created": "2019-03-05T03:18:13.581678+00:00", "pxe": null, "physical_location": "eth1"}

finally the ip it’s self:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Utilities/Address
{ "networked": "/api/v1/Utilities/Networked:1:", "address_block": "/api/v1/Utilities/AddressBlock:$ADRBLK:", "interface_name": "eth1", "offset": 10, "is_primary": true }
EOF

result:

{"netmask": "255.255.255.0", "ip_address": "10.0.0.10", "created": "2019-02-23T16:20:56.567650+00:00", "pointer": null, "vlan": 0, "networked": "/api/v1/Utilities/Networked:1:", "network": "10.0.0.0", "is_primary": false, "type": "Address", "interface_name": "eth1", "offset": 10, "address_block": "/api/v1/Utilities/AddressBlock:internal:", "gateway": "10.0.0.1", "sub_interface": null, "updated": "2019-02-23T16:20:56.567606+00:00", "prefix": "24"}

now to reserve some ip addresses so they do not get auto assigned:

for OFFSET in 2 3 4 5 6 7 8 9 11 12 13 14 15 16 17 18 19 20; do
cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Utilities/ReservedAddress
{ "address_block": "/api/v1/Utilities/AddressBlock:$ADRBLK:", "offset": "$OFFSET", "reason": "Network Reserved" }
EOF
done

result:

{"ip_address": "10.0.0.2", "offset": 2, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.312992+00:00", "address_block": "/api/v1/Utilities/AddressBlock:internal:", "updated": "2019-02-23T16:34:54.312941+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.3", "offset": 3, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.327090+00:00", "address_block": "/api/v1/Utilities/AddressBlock:internal:", "updated": "2019-02-23T16:34:54.327065+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.4", "offset": 4, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.339957+00:00", "address_block": "/api/v1/Utilities/AddressBlock:internal:", "updated": "2019-02-23T16:34:54.339924+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.5", "offset": 5, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.352559+00:00", "address_block": "/api/v1/Utilities/AddressBlock:internal:", "updated": "2019-02-23T16:34:54.352535+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.6", "offset": 6, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.365187+00:00", "address_block": "/api/v1/Utilities/AddressBlock:internal:", "updated": "2019-02-23T16:34:54.365162+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.7", "offset": 7, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.378354+00:00", "address_block": "/api/v1/Utilities/AddressBlock:internal:", "updated": "2019-02-23T16:34:54.378327+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.8", "offset": 8, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.390835+00:00", "address_block": "/api/v1/Utilities/AddressBlock:internal:", "updated": "2019-02-23T16:34:54.390812+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.9", "offset": 9, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.404003+00:00", "address_block": "/api/v1/Utilities/AddressBlock:internal:", "updated": "2019-02-23T16:34:54.403980+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.11", "offset": 11, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.416552+00:00", "address_block": "/api/v1/Utilities/AddressBlock:internal:", "updated": "2019-02-23T16:34:54.416528+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.12", "offset": 12, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.429354+00:00", "address_block": "/api/v1/Utilities/AddressBlock:internal:", "updated": "2019-02-23T16:34:54.429332+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.13", "offset": 13, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.442067+00:00", "address_block": "/api/v1/Utilities/AddressBlock:internal:", "updated": "2019-02-23T16:34:54.442043+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.14", "offset": 14, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.455041+00:00", "address_block": "/api/v1/Utilities/AddressBlock:internal:", "updated": "2019-02-23T16:34:54.455018+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.15", "offset": 15, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.467245+00:00", "address_block": "/api/v1/Utilities/AddressBlock:internal:", "updated": "2019-02-23T16:34:54.467222+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.16", "offset": 16, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.479525+00:00", "address_block": "/api/v1/Utilities/AddressBlock:internal:", "updated": "2019-02-23T16:34:54.479503+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.17", "offset": 17, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.492109+00:00", "address_block": "/api/v1/Utilities/AddressBlock:internal:", "updated": "2019-02-23T16:34:54.492083+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.18", "offset": 18, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.504386+00:00", "address_block": "/api/v1/Utilities/AddressBlock:internal:", "updated": "2019-02-23T16:34:54.504363+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.19", "offset": 19, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.517128+00:00", "address_block": "/api/v1/Utilities/AddressBlock:internal:", "updated": "2019-02-23T16:34:54.517105+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.20", "offset": 20, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.529458+00:00", "address_block": "/api/v1/Utilities/AddressBlock:internal:", "updated": "2019-02-23T16:34:54.529435+00:00", "type": "ReservedAddress"}

Starting DNS

Restart bind with new zones:

sudo systemctl restart bind9

Now to force a re-gen of the DNS files:

/usr/lib/contractor/cron/genDNS

This VM needs to use the contractor generated dns, so edit /etc/network/interfaces to set the dns server to 127.0.0.1 then, reload networking configuration:

sudo systemctl restart networking

now if you ping contractor you should get the internal ip (10.0.0.10):

ping static -c2

result:

PING eth1.contractor.site1.local (10.0.0.10) 56(84) bytes of data.
64 bytes from contractor.site1.local (10.0.0.10): icmp_seq=1 ttl=64 time=0.031 ms
64 bytes from contractor.site1.local (10.0.0.10): icmp_seq=2 ttl=64 time=0.063 ms

now take a look at the contractor ui at http://<contractor ip>, (this ip is the ip you assigned to the first interface)

Subcontractor

install tfptd (used for PXE booting) and the PXE booting agent:

sudo apt install -y tftpd-hpa
sudo respkg -i contractor-ipxe_0.1.respkg

now edit /etc/subcontractor.conf enable the modules you want to use, remove the ‘;’ and set the 0 to a 1. The 1 means one task for that plugin at a time. If you want to be able to process more targets at the same time, you can try 2 or 4 depending on the plugin, the resources of your vm, etc. You may also want to change the poll_delay to 5, this will cause subcontractor to ask for more tasks every 5 seconds instead of the default 20. If we were setting up a system that would be processing a lot of tasks, we would want to slow this down to reduce the overhead on contractor. In the dhcpd section, make sure interface and tftp_server are correct, tftp_server should be the ip of the vm on the new internal interface.

now start up subcontractor:

sudo systemctl start subcontractor
sudo systemctl start dhcpd

make sure it’s running:

sudo systemctl status subcontractor
sudo systemctl status dhcpd

optional, edit /etc/default/tftpd-hpa and add ‘-v ‘ to TFTP_OPTIONS. This will cause tfptd to log transfers to syslog. This can be helpful in troubleshooting boot problems. Make sure to run systemctl restart tftpd-hpa to reload.

Setting up VM Host

First we need to make a pre-built entry on a manual foundation to represent the virtualbox/vcenter/esx host, first creating the foundation:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Manual/ManualFoundation
{ "site": "$SITE", "locator": "host", "blueprint": "/api/v1/BluePrint/FoundationBluePrint:manual-foundation-base:" }
EOF

which should output something like:

{"state": "planned", "id_map": null, "located_at": null, "class_list": "['Metal', 'VM', 'Container', 'Switch', 'Manual']", "blueprint": "/api/v1/BluePrint/FoundationBluePrint:manual-foundation-base:", "created": "2019-02-23T16:48:53.818982+00:00", "built_at": null, "locator": "host", "updated": "2019-02-23T16:48:53.818962+00:00", "site": "/api/v1/Site/Site:site1:", "type": "Manual"}

create the interface:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Utilities/RealNetworkInterface
{ "foundation": "/api/v1/Building/Foundation:host:", "name": "eth0", "physical_location": "eth0", "is_provisioning": true }
EOF

which should output something like:

{"created": "2019-03-06T04:13:47.808962+00:00", "pxe": null, "name": "eth0", "physical_location": "eth0", "is_provisioning": true, "updated": "2019-03-06T04:13:47.808940+00:00", "mac": null, "foundation": "/api/v1/Building/Foundation:host:"}

Now to create the structure:

cat << EOF | curl -i "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Building/Structure
{ "site": "$SITE", "foundation": "/api/v1/Building/Foundation:host:", "hostname": "host", "blueprint": "/api/v1/BluePrint/StructureBluePrint:manual-structure-base:", "auto_build": false }
EOF

which should output something like:

HTTP/1.1 201 CREATED
Date: Sat, 23 Feb 2019 16:49:20 GMT
Server: Apache/2.4.18 (Ubuntu)
Object-Id: /api/v1/Building/Structure:2:
Cinp-Version: 0.9
Cache-Control: no-cache
Access-Control-Expose-Headers: Method, Type, Cinp-Version, Count, Position, Total, Multi-Object, Object-Id, Id-Only
Verb: CREATE
Access-Control-Allow-Origin: *
Content-Length: 412
Content-Type: application/json;charset=utf-8

{"config_uuid": "349c8a47-e123-4234-91de-c387a440ffa5", "auto_build": false, "hostname": "host", "created": "2019-02-23T16:49:20.064258+00:00", "state": "planned", "blueprint": "/api/v1/BluePrint/StructureBluePrint:manual-structure-base:", "built_at": null, "foundation": "/api/v1/Building/Foundation:host:", "config_values": {}, "updated": "2019-02-23T16:49:20.064239+00:00", "site": "/api/v1/Site/Site:site1:"}

look for the header Object-Id: /api/v1/Building/Structure:2:, take note of the sturcture id (the number between the :, in this case 2).

now we need to tell contractor it is allready built so it dosen’t try to build it again. There curently isn’t a API endpoint to manipluate the state of targets, so we will use a command line utility, this command needs to be run on the contractor VM. replace <structure id> with the id from the previous step:

/usr/lib/contractor/util/boss -f host --built
/usr/lib/contractor/util/boss -s <structure id> --built

which will output something like this:

Working with "ManualFoundation host"
No Job to Delete
ManualFoundation host now set to built.
Working with "Structure #2(host) of "manual-structure-base" in "site1""
No Job to Delete
Structure #2(host) of "manual-structure-base" in "site1" now set to built.

Now to define the foundation blueprint and create the complex.

VCenter

Environment setup:

export FBP="/api/v1/BluePrint/FoundationBluePrint:vcenter-vm-base:"
export FMDL="/api/v1/VCenter/VCenterFoundation"
export FDATA=', "vcenter_host": "/api/v1/VCenter/VCenterComplex:demovcenter:"'

First create the VirtualBox Complex, replace < datacenter > with the name of the VCenter datacenter to put the VMs in, if using ESX directly put ‘ha-datacenter’, replace < cluster > with the name of the cluster to put the vms in, if using ESX put the hostname of the ESX server, if it’s still default it will be ‘localhost.’. Replace < structure id > with the strudture id from the host creation above, < username > and < password > replace with the ESX/VCenter username and password:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/VCenter/VCenterComplex
{ "site": "$SITE", "name": "demovcenter", "description": "Demo VCenter/ESX Host/Complex", "vcenter_datacenter": "< datacenter >", "vcenter_cluster": "< cluster >", "vcenter_host": "/api/v1/Building/Structure:< structure id>:", "vcenter_username": "< username >", "vcenter_password": "< password >" }
EOF

should return something like:

{"built_percentage": 90, "state": "planned", "site": "/api/v1/Site/Site:site1:", "created": "2019-02-23T23:51:33.613222+00:00", "vcenter_host": "/api/v1/Building/Structure:2:", "vcenter_password": "vmware", "updated": "2019-02-23T23:51:33.613199+00:00", "vcenter_cluster": null, "name": "demovcenter", "description": "Demo VCenter/ESX Host/Complex", "vcenter_datacenter": "ha-datacenter", "type": "VCenter", "members": [], "vcenter_username": "root"}

Techinically if you are using VCenter, you should create another manual structure so Contractor knows the hosts of the VCenter cluster, however, for the sake of simplicity, we will just add the ESX Host/VCenter cluster we just added as the host of the VCenterCluster as it’s only member, once again the < structure id > is the id of the manual structure we have been using so far:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Building/ComplexStructure
{ "complex": "/api/v1/Building/Complex:demovcenter:", "structure": "/api/v1/Building/Structure:< structure id>:" }
EOF

should return something like:

{"created": "2019-02-24T00:02:06.164123+00:00", "complex": "/api/v1/Building/Complex:demovcenter:", "structure": "/api/v1/Building/Structure:2:", "updated": "2019-02-24T00:02:06.164082+00:00"}

now to set the ip address of the vcenter/esx host. This ip will be used by subcontractor to manipluate vms, and will need to be routeable from the contractor vm, this assumes that address is in the address space of the contractor vm, specifically the network that setupWizzard created, change < offset > to the offset of the host’s ip in that network. If the ip address of the host is 192.168.0.52 the setupWizzard assumed you were in a /24 so the offset is 52, replace structure id with the id from the structure creation step:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Utilities/Address
{ "networked": "/api/v1/Utilities/Networked:< structure id >:", "address_block": "/api/v1/Utilities/AddressBlock:main:", "interface_name": "eth0", "offset": < offset >, "is_primary": true }
EOF

which should output something like:

{"netmask": "255.255.255.0", "updated": "2019-02-23T18:51:53.521628+00:00", "type": "Address", "prefix": "24", "vlan": 0, "ip_address": "192.168.13.22", "interface_name": "eth0", "network": "192.168.13.0", "sub_interface": null, "address_block": "/api/v1/Utilities/AddressBlock:main:", "is_primary": false, "offset": 22, "pointer": null, "gateway": "192.168.13.1", "created": "2019-02-23T18:51:53.521652+00:00", "networked": "/api/v1/Utilities/Networked:2:"}

VirtualBox

Environment setup:

export FBP="/api/v1/BluePrint/FoundationBluePrint:virtualbox-vm-base:"
export FMDL="/api/v1/VirtualBox/VirtualBoxFoundation"
export FDATA=', "virtualbox_host": "/api/v1/VirtualBox/VirtualBoxComplex:demovbox:"'

First create the VirtualBox Complex, replace the < username > and < password > with either your username and password for the machine with vbox running on it, or if you ran the vboxmanage command to disable the auth library, you can leave the username and password a few random alpha letters:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/VirtualBox/VirtualBoxComplex
{ "site": "$SITE", "name": "demovbox", "virtualbox_username": "< username >", "virtualbox_password": "< password >", "description": "Demo VirtualBox Host/Complex" }
EOF

should output something like:

{"description": "Demo VirtualBox Host/Complex", "updated": "2019-03-05T03:29:33.401162+00:00", "site": "/api/v1/Site/Site:site1:", "built_percentage": 90, "virtualbox_password": "asdf", "name": "demovbox", "virtualbox_username": "asdf", "state": "planned", "created": "2019-03-05T03:29:33.401328+00:00", "members": [], "type": "VirtualBox"}

Now we add the structure host we manually created as a member of the complex, replace < structure id > with the id from the manul host structure from above:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Building/ComplexStructure
{ "complex": "/api/v1/Building/Complex:demovbox:", "structure": "/api/v1/Building/Structure:< structure id>:" }
EOF

should output something like:

{"complex": "/api/v1/Building/Complex:demovbox:", "structure": "/api/v1/Building/Structure:2:", "created": "2019-02-20T04:55:31.730431+00:00", "updated": "2019-02-20T04:55:31.730357+00:00"}

now to set the ip address, this is the ip address of virtualbox the host. This is the same ip that we passed to vboxwebsrv, which is offset 1 of the internal network we created:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Utilities/Address
{ "networked": "/api/v1/Utilities/Networked:< structure id >:", "address_block": "/api/v1/Utilities/AddressBlock:$ADRBLK:", "interface_name": "eth0", "offset": 1, "is_primary": true }
EOF

which should output something like:

{"netmask": "255.255.255.0", "updated": "2019-02-23T18:51:53.521628+00:00", "type": "Address", "prefix": "24", "vlan": 0, "ip_address": "192.168.13.22", "interface_name": "eth0", "network": "192.168.13.0", "sub_interface": null, "address_block": "/api/v1/Utilities/AddressBlock:main:", "is_primary": false, "offset": 22, "pointer": null, "gateway": "192.168.13.1", "created": "2019-02-23T18:51:53.521652+00:00", "networked": "/api/v1/Utilities/Networked:2:"}

Contractor is now running, now let’s configure it to make a VM.

Creating a VM (Ubuntu)

First we need to load the ubuntu blueprints:

sudo respkg -i contractor-ubuntu-base_0.1.respkg

Now we create the Foundation of the VM to be created:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/$FMDL
{ "site": "$SITE", "locator": "testvm01", "blueprint": "$FBP" $FDATA }
EOF

output:

{"state": "planned", "site": "/api/v1/Site/Site:site1:", "type": "VirtualBox", "id_map": "", "virtualbox_host": "/api/v1/VirtualBox/VirtualBoxComplex:demovbox:", "blueprint": "/api/v1/BluePrint/FoundationBluePrint:virtualbox-vm-base:", "built_at": null, "locator": "tesvm01", "located_at": null, "updated": "2019-02-20T04:58:52.855473+00:00", "created": "2019-02-20T04:58:52.855507+00:00", "class_list": "['VM', 'VirtualBox']", "virtualbox_uuid": null}

create the interface:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Utilities/RealNetworkInterface
{ "foundation": "/api/v1/Building/Foundation:testvm01:", "name": "eth0", "physical_location": "eth0", "is_provisioning": true }
EOF

output:

{"pxe": null, "name": "eth0", "is_provisioning": true, "physical_location": "eth0", "updated": "2019-02-25T14:28:36.245466+00:00", "mac": null, "foundation": "/api/v1/Building/Foundation:testvm01:", "created": "2019-02-25T14:28:36.245500+00:00"}

Now we will create a VM with the Ubuntu Bionic blueprint:

cat << EOF | curl -i "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Building/Structure
{ "site": "$SITE", "foundation": "/api/v1/Building/Foundation:testvm01:", "hostname": "testvm01", "blueprint": "/api/v1/BluePrint/StructureBluePrint:ubuntu-bionic-base:", "auto_build": true }
EOF

once again take node of the structure id. Now we assign and ip address, we will let contractor pick, we are going to use the helper method nextAddress. Replace < structure id > with the structure id from the previous call:

cat << EOF | curl "${COPS[@]}" --data @- -X CALL "$CHOST/api/v1/Utilities/AddressBlock:$ADRBLK:(nextAddress)"
{ "structure": "/api/v1/Building/Structure:< structure id >:", "interface_name": "eth0", "is_primary": true }
EOF

output:

"/api/v1/Utilities/Address:30:"

Ok, we have a VM in the database, now to see it get built. Pull up the http://<contractor ip> in a web browser if you don’t have it open allready, go to the Job Log should see an entry saying that the foundation build has started. Goto the Jobs should see a Foundation or Structure Job there. The Foundation Job won’t last long. In the top right of the page is a refresh and auto refresh buttons.

After the Foundation job completes, a Structure job will auto start, after it completes your VM should be up and sshable, however the default for ubuntu is to disallow sshing as root, but we can show the ssh service is listening:

nc -vz testvm01 22

should output something like:

Connection to testvm01 22 port [tcp/ssh] succeeded!

If you pull up the console, the default root password is “root”.

After you have verified that it is there, logout of the test vm and kick off a job to delete it and re-build it:

/usr/lib/contractor/util/boss -f testvm01 --do-destroy

We set the structure to auto-build, so after it get’s done with the remove job, it will create it again.

Creating a VM (CentOS)

Ok, let’s create a centos VM now, is’t all the same as the ubuntu VM except the blueprint we choose.

Load the centos Blueprints:

sudo respkg -i contractor-centos-base_0.1.respkg

Foundation:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/$FMDL
{ "site": "$SITE", "locator": "testvm02", "blueprint": "$FBP" $FDATA }
EOF

output:

{"state": "planned", "site": "/api/v1/Site/Site:site1:", "type": "VirtualBox", "id_map": "", "virtualbox_host": "/api/v1/VirtualBox/VirtualBoxComplex:demovbox:", "blueprint": "/api/v1/BluePrint/FoundationBluePrint:virtualbox-vm-base:", "built_at": null, "locator": "tesvm01", "located_at": null, "updated": "2019-02-20T04:58:52.855473+00:00", "created": "2019-02-20T04:58:52.855507+00:00", "class_list": "['VM', 'VirtualBox']", "virtualbox_uuid": null}

create the interface:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Utilities/RealNetworkInterface
{ "foundation": "/api/v1/Building/Foundation:testvm02:", "name": "eth0", "physical_location": "eth0", "is_provisioning": true }
EOF

output:

{"pxe": null, "name": "eth0", "is_provisioning": true, "physical_location": "eth0", "updated": "2019-02-25T14:28:36.245466+00:00", "mac": null, "foundation": "/api/v1/Building/Foundation:testvm02:", "created": "2019-02-25T14:28:36.245500+00:00"}

Now we will create a VM with the CentOS7 blueprint:

cat << EOF | curl -i "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Building/Structure
{ "site": "$SITE", "foundation": "/api/v1/Building/Foundation:testvm02:", "hostname": "testvm02", "blueprint": "/api/v1/BluePrint/StructureBluePrint:centos-7-base:", "auto_build": true }
EOF

output:

HTTP/1.1 201 CREATED
Date: Mon, 11 Mar 2019 13:45:58 GMT
Server: Apache/2.4.18 (Ubuntu)
Cache-Control: no-cache
Verb: CREATE
Cinp-Version: 0.9
Object-Id: /api/v1/Building/Structure:4:
Access-Control-Expose-Headers: Method, Type, Cinp-Version, Count, Position, Total, Multi-Object, Object-Id, Id-Only
Access-Control-Allow-Origin: *
Content-Length: 413
Content-Type: application/json;charset=utf-8

{"hostname": "testvm02", "created": "2019-03-11T13:45:58.963923+00:00", "config_values": null, "auto_build": true, "config_uuid": "d8821d29-f884-4c2d-af63-7d0292b2ce41", "updated": "2019-03-11T13:45:58.963901+00:00", "blueprint": "/api/v1/BluePrint/StructureBluePrint:centos-7-base:", "site": "/api/v1/Site/Site:site1:", "foundation": "/api/v1/Building/Foundation:testvm02:", "built_at": null, "state": "planned"}

and assign the ip address, make sure to use the structure id from the testvm02 structure:

cat << EOF | curl "${COPS[@]}" --data @- -X CALL "$CHOST/api/v1/Utilities/AddressBlock:$ADRBLK:(nextAddress)"
{ "structure": "/api/v1/Building/Structure:< structure id >:", "interface_name": "eth0", "is_primary": true }
EOF

output:

"/api/v1/Utilities/Address:30:"

Again the jobs should be running to create the CentOS VM. When it is done, ssh in:

ssh root@testvm02

go a head and play arround with it for a bit. make sure to try deconfiguring both VMs at the same time so you can see Contractor do more than one thing at a time.

Accessing Configuration Information

Contractor provides three configuration urls for a target. The first two depend on what the target is set to PXE boot to, the thrid is all the configuraiton information for that target in JSON format.

ssh into one of the VMs, this will show what it is like from testvm02:

ssh root@testvm02

First the pxe script, this script is downloaded and run but the iPXE boot loader that the target VM was told to download from the dhcp server.:

curl http://contractor/config/boot_script/

which will output:

#!ipxe

echo Booting form Primary Boot Disk
sanboot --no-describe --drive 0x80 || echo Primary Boot Disk is not Bootable

The VM is curent set to the normal-boot pxe. That script tells it to boot to the first harddrive.

Next the pxe_template:

curl http://contractor/config/pxe_template/

output:

# Normal Boot

once again, with this VM having been set to normal-boot, the pxe template is just a comment at the top. The pxe_template is stored as a Jinja2 template that is combined with the configuration information and served out to the target. This is the URL that is used for the Kickstart and/or Pressed files for the CentOS and Debian installers. The source for the centos and ubuntu boot_scripts and pxe_templates are at https://github.com/T3kton/resources/blob/master/os-bases/centos/usr/lib/contractor/resources/centos.toml and https://github.com/T3kton/resources/blob/master/os-bases/ubuntu/usr/lib/contractor/resources/ubuntu.toml thoes are then packaged during when you built the resources, and installed to /usr/lib/contractor/resource/ when the resource package was installed.

The thrid url is:

curl http://contractor/config/config/

output:

{"installer_pxe": "centos-7", "__pxe_template_location": "http://contractor/config/pxe_template/", "_structure_config_uuid": "118e0e44-457e-47df-b8c0-d157d5dde1b4", "mirror_server": "mirror.centos.org", "_blueprint": "centos-7-base", "__timestamp": "2019-03-11T14:32:27.909856+00:00", "_foundation_state": "built", "domain_name": "site1.local", "dns_search": ["site1.local", "local"], "_structure_state": "built", "__pxe_location": "http://static/pxe/", "distro": "centos", "_hostname": "testvm02", "_foundation_class_list": ["VM", "VCenter"], "dns_servers": ["10.0.0.10"], "memory_size": 2048, "_foundation_type": "VCenter", "_provisioning_interface": "eth0", "_vcenter_host": "demovcenter", "_interface_map": {"eth0": {"physical_location": "eth0", "name": "eth0", "mac": "00:50:56:03:1e:6d", "address_list": [{"vlan": null, "address": "10.0.0.123", "prefix": 24, "netmask": "255.255.255.0", "primary": true, "sub_interface": null, "network": "10.0.0.0", "tagged": false, "gateway": null, "auto": true, "mtu": 1500}]}}, "_foundation_locator": "testvm02", "_vcenter_uuid": "52545577-0025-e8d7-1915-bd64585f47c1", "_vcenter_cluster": "localhost.", "_site": "site1", "ntp_servers": ["ntp.ubuntu.com"], "distro_version": "7", "_fqdn": "testvm02.site1.local", "mirror_proxy": "http://10.0.0.10:3128/", "_foundation_interface_list": [{"physical_location": "eth0", "name": "eth0", "mac": "00:50:56:03:1e:6d", "address_list": [{"vlan": null, "address": "10.0.0.123", "prefix": 24, "netmask": "255.255.255.0", "primary": true, "sub_interface": null, "network": "10.0.0.0", "tagged": false, "gateway": null, "auto": true, "mtu": 1500}]}], "__contractor_host": "http://contractor/", "_foundation_id": "testvm02", "vcenter_guest_id": "rhel7_64Guest", "swap_size": 512, "_structure_id": 4, "__last_modified": "2019-03-11T14:01:18.090983+00:00", "_provisioning_interface_mac": "00:50:56:03:1e:6d", "_vcenter_datacenter": "ha-datacenter", "virtualbox_guest_type": "RedHat_64", "root_pass": "$6$rootroot$oLo.loyMV45VA7/0sKV5JH/xBAXiq/igL4hQrGz3yd9XUavmC82tZm1lxW2N.5eLxQUlqp53wXKRzifZApP0/1"}

This url can be used by what ever scripts/CMS as a source of configuration intormation. See the documentation at Configuration Values for more information on how these values are compiled. One value to point out here is _structure_config_uuid, this value is set when the sturcture record is created or when the structure is destroyed. This way if there is a stale copy of the structure (old VM stapshot, or a VM that didn’t get cleaned up properly, etc) comes online, it (or some other monitoring system) can detect that it is now longer curent and take action.

Contractor uses the source ip address of this URL requests to determine which target’s information to return. You can also use the structure id, foundation locator or config uuid, to tell contractor which configuration to return.

by config uuid:

curl http://contractor/config/config/c/118e0e44-457e-47df-b8c0-d157d5dde1b4

by structure id:

curl http://contractor/config/config/s/4

by foundation locator:

curl http://contractor/config/config/f/testvm02

one way for a target to detect if it is stil good and in cases when the ip address might change, is to request it’s config by the uuid.

Removing the VMs

Deleting the VMs once again uses the boss command, it is possible to do it via the HTTP interface, however it takes a few more steps. Before we used the ``–do-destroy` option, this tells contractor to remove the vms, and with the auth-configure flag, it turns arround and re-builds them. The option –do-pre-delete is simmaler but it disables the auto-configure so contractor dosen’t automatically start building it again. First to delete the structure:

/usr/lib/contractor/util/boss -s <structure id> --do-pre-delete

when the sturcture job completes:

/usr/lib/contractor/util/boss -s <structure id> --do-delete

and the same pattern for the foundation, the locator is the vmname ie testvm01:

/usr/lib/contractor/util/boss -f <locator> --do-pre-delete

and when that job completes run:

/usr/lib/contractor/util/boss -f <locator> --do-delete

Now the VM is no longer in virtualbox/vcenter nor contractor.