Give it a Try

Warning: Contractor is made to solve complex problems, for the most part it presents a much less complex interface to you. That being said, there is a bit of a hurdle to get it up and running, depending on what you are trying to automate. So, buckle in, this is going to be a lot of fun.

This demo will result in a VM in a private /24 network that can create and destroy other VMs in that network. You can use either VCenter or VirtualBox. After the Contractor VM is up and running, you can install more blueprints and plugins to do docker or other foundations. The requirement for the private network comes primarally from the setupWizard’s default configuration, if you feel comfortable with modifying the setupWizard and the Apache Configurations, you can use blueprints and plugins for hosted providers such as AWS and Azure.

NOTE: setupWizard is going to re-write some bind config files, so don’t edit them until after the install is complete.

If you are not failure with how Contractor handles Networking, you will probably want to take a look at the overview on Ip Addresses.

Installing

Create an Ubuntu Xenial VM, name it contractor, set the fqdn to contractor.site1.test Ideally it should be in a /24 network. Offset 1 is assumed to be the gateway. All these values can be adjusted either in the setupWizard file before it is run, or after it is setup, you can use the API/UI to edit these values. The DNS server will be set for the contractor VM, and bind on the contractor vm will be set to forward to the DNS server that was originally configured on the VM.

From Source

From Source

From Packages

Prebuilt Packages

Setup

Set up Infrastructure

To prevent your new contractor VM from taking over the network you are curently connected to, you will need to configure a issloated network. There are also some other Infrastructure related tasks that need to be done.

Vcenter/ESX Setup

VirtualBox Setup

Install Required Services

Install Postgres:

sudo apt install -y postgresql-client postgresql-9.5 mongodb-server

Create the postgres db:

sudo su postgres -c "echo \"CREATE ROLE contractor WITH PASSWORD 'contractor' NOSUPERUSER NOCREATEDB NOCREATEROLE LOGIN;\" | psql"
sudo su postgres -c "createdb -O contractor contractor"

NOTE, for those two commands you may see:

could not change directory to "/root": Permission denied

this is from the su command, thoes messages can be ignored

Update Packages

The Ubuntu toml and jinja2 packages are too old, update with:

sudo apt install -y python3-pip
sudo pip3 install toml jinja2 --upgrade

if you are using the virtualbox plugin, you will need zeep:

sudo pip3 install zeep

for vcenter/esx install pyvmomi:

sudo pip3 install pyvmomi

Configure Apache

We will need a HTTP site to serve up static resources, as well as a Proxy server to bridge from the isolated network. This proxy server will also cache, that will make things install faster the second time.

First create the directory for the static resources:

sudo mkdir -p /var/www/static

now create the proxy site /etc/apache2/sites-available/proxy.conf with the following:

Listen 3128
<VirtualHost *:3128>
  ServerName proxy
  ServerAlias proxy.site1.test

  DocumentRoot /var/www/static

  ErrorLog ${APACHE_LOG_DIR}/proxy_error.log
  CustomLog ${APACHE_LOG_DIR}/proxy_access.log combined

  ProxyRequests On
  ProxyVia Full

  CacheEnable disk http://
  CacheEnable disk https://

  NoProxy static static.site1.test
  NoProxy contractor contractor.site1.test

  # ProxyRemote * http://<up stream proxy>:3128/
</VirtualHost>

NOTE: if you need to relay through an upstream proxy to have access to the ubuntu and centos mirrors, enable the ProxyRemote line and update it with the upstream proxy. Now create the static site /etc/apache2/sites-available/static.conf with the following:

<VirtualHost *:80>
  ServerName static
  ServerAlias static.site1.test

  DocumentRoot /var/www/static

  LogFormat "%a %t %D \"%r\" %>s %I %O \"%{Referer}i\" \"%{User-Agent}i\" %X" static_log
  ErrorLog ${APACHE_LOG_DIR}/static_error.log
  CustomLog ${APACHE_LOG_DIR}/static_access.log static_log
</VirtualHost>

Modify /etc/apache2/sites-available/contractor.conf and enable the ServerAlias line, and change the <domain> to site1.test

Now enable the proxy and static site, disable the default site, and reload the apache configuration:

sudo a2ensite proxy
sudo a2ensite static
sudo a2dissite 000-default
sudo a2enmod proxy proxy_connect proxy_ftp proxy_http cache_disk cache
sudo systemctl restart apache2
sudo systemctl start apache-htcacheclean

Setup the database

Now to create the db:

/usr/lib/contractor/util/manage.py migrate

Install the iputils functions, this contains the port check function contractor will use to verify the OS has booted:

sudo respkg -i contractor-plugins-iputils_0.4.0.respkg

Install base os config:

sudo respkg -i contractor-os-base_0.4.respkg

Now to enable plugins. We use manual for misc stuff that is either pre-configured or handled by something else:

sudo respkg -i contractor-plugins-manual_0.4.0.respkg

if you are using esx/vcenter:

sudo respkg -i contractor-plugins-vcenter_0.4.0.respkg

if you are using virtualbox:

sudo respkg -i contractor-plugins-virtualbox_0.4.0.respkg

do manual plugin again so it can cross link to the other plugins:

sudo respkg -i contractor-plugins-manual_0.4.0.respkg

Now to setup some base info, and configure bind:

sudo /usr/lib/contractor/setup/setupWizard --no-ip-reservation --dns-server=10.0.0.10 --proxy-server=http://10.0.0.10:3128/

It is safe to ignore the message:

rndc: connect failed: 127.0.0.1#953: connection refused
WARNING: "rndc reload" failed

Bind (the DNS server) is not running yet, it will be started later.

And now to create a user for us to login as for the API calls:

/usr/lib/contractor/util/manage.py createsuperuser

that command will ask for a username, email and password. The email address does not need to be a real address.

Environment Setup

We will be using the HTTP API to inject new stuff into contractor. You can run these commands from either the contractor VM, or any place that can make http requests to contractor.

we will be using curl, make sure it is installed:

sudo apt install -y curl

First we will define some Environment values so we don’t have to keep tying redundant info the Contractor server. This is assuming you will be running these commands from the contractor VM, if you are running these steps from someplace else, update the ip address to the ip address of the contractor vm:

export COPS=( --noproxy \* --header "CInP-Version: 0.9" --header "Content-Type: application/json" )
export SITE="/api/v1/Site/Site:site1:"
export CHOST="http://127.0.0.1"

COPS is defining some curl options, in this case some headers that are required by CInP(see https://github.com/cinp/) which is used by Contractor for it’s API. SITE defines the uri of the site we are going to use, and CHOST is the URL to the Contractor server.

now we need to login, replace < username > and < password > with the username and password you specified API user (the createsuperuser step):

cat << EOF | curl "${COPS[@]}" --data @- -X CALL $CHOST/api/v1/Auth/User\(login\)
{ "username": "< username >", "password": "< password >" }
EOF

which will output something like:

"k4of9zewijvze0gf72ylb6p6zxv4srol"

which will return a auth token, save that to our headers, replace < username > with the API username, and < auth token > with the result of the last command:

COPS+=( --header "Auth-Id: < username >")
COPS+=( --header "Auth-Token: < auth token >" )

This is adding more headers to our curl options, from here on our curl operations are authenticated. Let’s make sure our login is working:

cat << EOF | curl "${COPS[@]}" --data @- -X CALL $CHOST/api/v1/Auth/User\(whoami\)
{}
EOF

that should output your username, for example:

"root"

Network Configuration

The setupWizard has pre-loaded the database with a stand in host to represent the contractor VM and has flagged it as pre-built. It has also created a site called site1 and some base DNS configuration. It also took the network of the primary interface and loaded it into the database named ‘main’.

We need to create another address block for the internal network. Replace < network name > with the name of the network created in vcenter (ie: internal) or virtual box (ie: vboxnet0):

cat << EOF | curl -i "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Utilities/AddressBlock
{ "site": "$SITE", "name": "<network name>", "subnet": "10.0.0.1", "gateway_offset": null, "prefix": "24" }
EOF

which should output something like:

HTTP/1.1 201 CREATED
Date: Thu, 23 May 2019 23:42:17 GMT
Server: Apache/2.4.18 (Ubuntu)
Verb: CREATE
Access-Control-Allow-Origin: *
Cinp-Version: 0.9
Access-Control-Expose-Headers: Method, Type, Cinp-Version, Count, Position, Total, Multi-Object, Object-Id, Id-Only
Cache-Control: no-cache
Object-Id: /api/v1/Utilities/AddressBlock:2:
Content-Length: 318
Content-Type: application/json;charset=utf-8

{"name": "vboxnet0", "size": "254", "_max_address": "10.0.0.255", "gateway_offset": null, "updated": "2019-05-23T23:42:17.180084+00:00", "site": "/api/v1/Site/Site:site1:", "netmask": "255.255.255.0", "subnet": "10.0.0.0", "created": "2019-05-23T23:42:17.180121+00:00", "gateway": null, "isIpV4": "True", "prefix": 24}

look for the header Object-Id: /api/v1/Building/AddressBlock:2:, the number between the : may be something else. Set another environment variable to the Id value, replace the < id > to match the number in the Object-Id above:

export ADRBLK="/api/v1/Utilities/AddressBlock:< id >:"

NOTE: the subnet you specify when creating the AddressBlock will be rounded up to the top of the subnet. In this case we could of specified any ip from 10.0.0.0 - 10.0.0.255 would result in the same subnet.

Now to add the internal ip of the contractor host, first set the address on eth0 to non-primary, we want the internal ip to be primary:

cat << EOF | curl "${COPS[@]}" --data @- -X UPDATE $CHOST/api/v1/Utilities/Address:1:
{ "is_primary": false }
EOF

result:

{"netmask": "255.255.255.0", "offset": 126, "updated": "2019-03-05T03:16:00.142926+00:00", "prefix": "24", "pointer": null, "networked": "/api/v1/Utilities/Networked:1:", "vlan": 0, "ip_address": "192.168.13.126", "is_primary": false, "interface_name": "eth0", "address_block": "/api/v1/Utilities/AddressBlock:2:", "created": "2019-03-05T02:45:12.304186+00:00", "gateway": "192.168.13.1", "sub_interface": null, "type": "Address", "network": "192.168.13.0"}

create an interface eth1 for the ip to belong to (this represents the new interface we created on the internal network):

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Utilities/RealNetworkInterface
{ "foundation": "/api/v1/Building/Foundation:contractor:", "name": "eth1", "physical_location": "eth1", "is_provisioning": false }
EOF

result:

{"is_provisioning": false, "foundation": "/api/v1/Building/Foundation:contractor:", "name": "eth1", "updated": "2019-03-05T03:18:13.581612+00:00", "mac": null, "created": "2019-03-05T03:18:13.581678+00:00", "pxe": null, "physical_location": "eth1"}

finally the ip it’s self:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Utilities/Address
{ "networked": "/api/v1/Utilities/Networked:1:", "address_block": "$ADRBLK", "interface_name": "eth1", "offset": 10, "is_primary": true }
EOF

result:

{"netmask": "255.255.255.0", "ip_address": "10.0.0.10", "created": "2019-02-23T16:20:56.567650+00:00", "pointer": null, "vlan": 0, "networked": "/api/v1/Utilities/Networked:1:", "network": "10.0.0.0", "is_primary": false, "type": "Address", "interface_name": "eth1", "offset": 10, "address_block": "/api/v1/Utilities/AddressBlock:2:", "gateway": "10.0.0.1", "sub_interface": null, "updated": "2019-02-23T16:20:56.567606+00:00", "prefix": "24"}

now to reserve some ip addresses so they do not get auto assigned:

for OFFSET in 2 3 4 5 6 7 8 9 11 12 13 14 15 16 17 18 19 20; do
cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Utilities/ReservedAddress
{ "address_block": "$ADRBLK", "offset": "$OFFSET", "reason": "Network Reserved" }
EOF
done

result:

{"ip_address": "10.0.0.2", "offset": 2, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.312992+00:00", "address_block": "/api/v1/Utilities/AddressBlock:2:", "updated": "2019-02-23T16:34:54.312941+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.3", "offset": 3, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.327090+00:00", "address_block": "/api/v1/Utilities/AddressBlock:2:", "updated": "2019-02-23T16:34:54.327065+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.4", "offset": 4, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.339957+00:00", "address_block": "/api/v1/Utilities/AddressBlock:2:", "updated": "2019-02-23T16:34:54.339924+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.5", "offset": 5, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.352559+00:00", "address_block": "/api/v1/Utilities/AddressBlock:2:", "updated": "2019-02-23T16:34:54.352535+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.6", "offset": 6, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.365187+00:00", "address_block": "/api/v1/Utilities/AddressBlock:2:", "updated": "2019-02-23T16:34:54.365162+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.7", "offset": 7, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.378354+00:00", "address_block": "/api/v1/Utilities/AddressBlock:2:", "updated": "2019-02-23T16:34:54.378327+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.8", "offset": 8, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.390835+00:00", "address_block": "/api/v1/Utilities/AddressBlock:2:", "updated": "2019-02-23T16:34:54.390812+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.9", "offset": 9, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.404003+00:00", "address_block": "/api/v1/Utilities/AddressBlock:2:", "updated": "2019-02-23T16:34:54.403980+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.11", "offset": 11, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.416552+00:00", "address_block": "/api/v1/Utilities/AddressBlock:2:", "updated": "2019-02-23T16:34:54.416528+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.12", "offset": 12, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.429354+00:00", "address_block": "/api/v1/Utilities/AddressBlock:2:", "updated": "2019-02-23T16:34:54.429332+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.13", "offset": 13, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.442067+00:00", "address_block": "/api/v1/Utilities/AddressBlock:2:", "updated": "2019-02-23T16:34:54.442043+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.14", "offset": 14, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.455041+00:00", "address_block": "/api/v1/Utilities/AddressBlock:2:", "updated": "2019-02-23T16:34:54.455018+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.15", "offset": 15, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.467245+00:00", "address_block": "/api/v1/Utilities/AddressBlock:2:", "updated": "2019-02-23T16:34:54.467222+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.16", "offset": 16, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.479525+00:00", "address_block": "/api/v1/Utilities/AddressBlock:2:", "updated": "2019-02-23T16:34:54.479503+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.17", "offset": 17, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.492109+00:00", "address_block": "/api/v1/Utilities/AddressBlock:2:", "updated": "2019-02-23T16:34:54.492083+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.18", "offset": 18, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.504386+00:00", "address_block": "/api/v1/Utilities/AddressBlock:2:", "updated": "2019-02-23T16:34:54.504363+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.19", "offset": 19, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.517128+00:00", "address_block": "/api/v1/Utilities/AddressBlock:2:", "updated": "2019-02-23T16:34:54.517105+00:00", "type": "ReservedAddress"}
{"ip_address": "10.0.0.20", "offset": 20, "reason": "Network Reserved", "created": "2019-02-23T16:34:54.529458+00:00", "address_block": "/api/v1/Utilities/AddressBlock:2:", "updated": "2019-02-23T16:34:54.529435+00:00", "type": "ReservedAddress"}

Starting DNS

Restart bind with new zones:

sudo systemctl restart bind9

Now to force a re-gen of the DNS files:

sudo /usr/lib/contractor/cron/genDNS

This VM needs to use the contractor generated dns, so edit /etc/network/interfaces to set the dns server to “127.0.0.1”, and set the dns search to “site1.test site1”. For example:

auto ensXXX
iface ensXXX inet static
  ...
  dns-nameservers 127.0.0.1
  dns-search site1.test site1

then reload networking configuration:

sudo systemctl restart networking

now if you ping contractor you should get the internal ip (10.0.0.10):

ping static -c2

result:

PING eth1.contractor.site1.test (10.0.0.10) 56(84) bytes of data.
64 bytes from contractor.site1.test (10.0.0.10): icmp_seq=1 ttl=64 time=0.031 ms
64 bytes from contractor.site1.test (10.0.0.10): icmp_seq=2 ttl=64 time=0.063 ms

now take a look at the contractor ui at http://<contractor ip>, (this ip is the ip you assigned to the first interface)

Subcontractor

install tfptd (used for PXE booting) and the PXE booting agent:

sudo apt install -y tftpd-hpa
sudo respkg -i contractor-ipxe_0.4.respkg

now edit /etc/subcontractor.conf enable the modules you want to use, remove the ‘;’ and set the 0 to a 1. The 1 means one task for that plugin at a time. If you want to be able to process more targets at the same time, you can try 2 or 4 depending on the plugin, the resources of your vm, etc. You may also want to change the poll_delay to 5, this will cause subcontractor to ask for more tasks every 5 seconds instead of the default 20. If we were setting up a system that would be processing a lot of tasks, we would want to slow this down to reduce the overhead on contractor. In the dhcpd section, make sure listen_interface and tftp_server are correct, tftp_server should be the ip of the vm on the new internal interface.

now start up subcontractor:

sudo systemctl start subcontractor
sudo systemctl start dhcpd

make sure it’s running:

sudo systemctl status subcontractor
sudo systemctl status dhcpd

optional, edit /etc/default/tftpd-hpa and add ‘-v ‘ to TFTP_OPTIONS. This will cause tfptd to log transfers to syslog. This can be helpful in troubleshooting boot problems. Make sure to run systemctl restart tftpd-hpa to reload.

Setting up VM Host

First we need to make a pre-built entry on a manual foundation to represent the virtualbox/vcenter/esx host, first creating the foundation:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Manual/ManualFoundation
{ "site": "$SITE", "locator": "host", "blueprint": "/api/v1/BluePrint/FoundationBluePrint:manual-foundation-base:" }
EOF

which should output something like:

{"state": "planned", "id_map": null, "located_at": null, "class_list": "['Metal', 'VM', 'Container', 'Switch', 'Manual']", "blueprint": "/api/v1/BluePrint/FoundationBluePrint:manual-foundation-base:", "created": "2019-02-23T16:48:53.818982+00:00", "built_at": null, "locator": "host", "updated": "2019-02-23T16:48:53.818962+00:00", "site": "/api/v1/Site/Site:site1:", "type": "Manual"}

create the interface:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Utilities/RealNetworkInterface
{ "foundation": "/api/v1/Building/Foundation:host:", "name": "eth0", "physical_location": "eth0", "is_provisioning": true }
EOF

which should output something like:

{"created": "2019-03-06T04:13:47.808962+00:00", "pxe": null, "name": "eth0", "physical_location": "eth0", "is_provisioning": true, "updated": "2019-03-06T04:13:47.808940+00:00", "mac": null, "foundation": "/api/v1/Building/Foundation:host:"}

Now to create the structure:

cat << EOF | curl -i "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Building/Structure
{ "site": "$SITE", "foundation": "/api/v1/Building/Foundation:host:", "hostname": "host", "blueprint": "/api/v1/BluePrint/StructureBluePrint:manual-structure-base:" }
EOF

which should output something like:

HTTP/1.1 201 CREATED
Date: Sat, 23 Feb 2019 16:49:20 GMT
Server: Apache/2.4.18 (Ubuntu)
Object-Id: /api/v1/Building/Structure:2:
Cinp-Version: 0.9
Cache-Control: no-cache
Access-Control-Expose-Headers: Method, Type, Cinp-Version, Count, Position, Total, Multi-Object, Object-Id, Id-Only
Verb: CREATE
Access-Control-Allow-Origin: *
Content-Length: 412
Content-Type: application/json;charset=utf-8

{"config_uuid": "349c8a47-e123-4234-91de-c387a440ffa5", "hostname": "host", "created": "2019-02-23T16:49:20.064258+00:00", "state": "planned", "blueprint": "/api/v1/BluePrint/StructureBluePrint:manual-structure-base:", "built_at": null, "foundation": "/api/v1/Building/Foundation:host:", "config_values": {}, "updated": "2019-02-23T16:49:20.064239+00:00", "site": "/api/v1/Site/Site:site1:"}

look for the header Object-Id: /api/v1/Building/Structure:2:, take note of the sturcture id (the number between the :, in this case 2).

now we need to tell contractor it is allready built so it dosen’t try to build it again. There curently isn’t a API endpoint to manipluate the state of targets, so we will use a command line utility, this command needs to be run on the contractor VM. replace <structure id> with the id from the previous step:

/usr/lib/contractor/util/boss -f host --built
/usr/lib/contractor/util/boss -s <structure id> --built

which will output something like this:

Working with "ManualFoundation host"
No Job to Delete
ManualFoundation host now set to built.
Working with "Structure #2(host) of "manual-structure-base" in "site1""
No Job to Delete
Structure #2(host) of "manual-structure-base" in "site1" now set to built.

Now to define the foundation blueprint and create the complex.

VCenter

Environment setup:

export FBP="/api/v1/BluePrint/FoundationBluePrint:vcenter-vm-base:"
export FMDL="/api/v1/VCenter/VCenterFoundation"
export FDATA=', "vcenter_complex": "/api/v1/VCenter/VCenterComplex:demovcenter:"'

First create the VCenterBox Complex, replace < datacenter > with the name of the VCenter datacenter to put the VMs in, if using ESX directly put ‘ha-datacenter’, replace < cluster > with the name of the cluster to put the vms in, if using ESX put the hostname of the ESX server, if it’s still default it will be ‘localhost.’. Replace < structure id > with the strudture id from the host creation above, < username > and < password > replace with the ESX/VCenter username and password:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/VCenter/VCenterComplex
{ "site": "$SITE", "name": "demovcenter", "description": "Demo VCenter/ESX Host/Complex", "vcenter_datacenter": "< datacenter >", "vcenter_cluster": "< cluster >", "vcenter_host": "/api/v1/Building/Structure:< structure id>:", "vcenter_username": "< username >", "vcenter_password": "< password >" }
EOF

should return something like:

{"built_percentage": 90, "state": "planned", "site": "/api/v1/Site/Site:site1:", "created": "2019-02-23T23:51:33.613222+00:00", "vcenter_host": "/api/v1/Building/Structure:2:", "vcenter_password": "vmware", "updated": "2019-02-23T23:51:33.613199+00:00", "vcenter_cluster": null, "name": "demovcenter", "description": "Demo VCenter/ESX Host/Complex", "vcenter_datacenter": "ha-datacenter", "type": "VCenter", "members": [], "vcenter_username": "root"}

Technically if you are using VCenter, you should create another structure so Contractor knows the hosts of the VCenter cluster, however, for the sake of simplicity, we will just add the ESX Host/VCenter cluster we just added as the host of the VCenterCluster as it’s only member, once again the < structure id > is the id of the manual structure we have been using so far:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Building/ComplexStructure
{ "complex": "/api/v1/Building/Complex:demovcenter:", "structure": "/api/v1/Building/Structure:< structure id>:" }
EOF

should return something like:

{"created": "2019-02-24T00:02:06.164123+00:00", "complex": "/api/v1/Building/Complex:demovcenter:", "structure": "/api/v1/Building/Structure:2:", "updated": "2019-02-24T00:02:06.164082+00:00"}

now to set the ip address of the vcenter/esx host. This ip will be used by subcontractor to communicate with VCenter to manipulate vms, and will need to be route-able from the contractor vm (where subcontractor is installed), this assumes that address is in the address space of the contractor vm, specifically the network that setupWizard created, Which should be attached to AddressBlock 1. Change < offset > to the offset of the VCenter/ESX host, if the VCenter/ESX host is not in the same network that the contractor was created in, (and thus the same network that was setup bu the setup wizzard), you will need to create another AddressBlock and update the following call to use that AddressBlock in the following. Replace structure id with the id from the structure creation step:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Utilities/Address
{ "networked": "/api/v1/Utilities/Networked:< structure id >:", "address_block": "/api/v1/Utilities/AddressBlock:1:", "interface_name": "eth0", "offset": < offset >, "is_primary": true }
EOF

which should output something like:

{"netmask": "255.255.255.0", "updated": "2019-02-23T18:51:53.521628+00:00", "type": "Address", "prefix": "24", "vlan": 0, "ip_address": "192.168.13.22", "interface_name": "eth0", "network": "192.168.13.0", "sub_interface": null, "address_block": "/api/v1/Utilities/AddressBlock:1:", "is_primary": false, "offset": 22, "pointer": null, "gateway": "192.168.13.1", "created": "2019-02-23T18:51:53.521652+00:00", "networked": "/api/v1/Utilities/Networked:2:"}

VirtualBox

Environment setup:

export FBP="/api/v1/BluePrint/FoundationBluePrint:virtualbox-vm-base:"
export FMDL="/api/v1/VirtualBox/VirtualBoxFoundation"
export FDATA=', "virtualbox_complex": "/api/v1/VirtualBox/VirtualBoxComplex:demovbox:"'

First create the VirtualBox Complex, replace the < username > and < password > with either your username and password for the machine with vbox running on it, or if you ran the vboxmanage command to disable the auth library, you can leave the username and password a few random alpha letters:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/VirtualBox/VirtualBoxComplex
{ "site": "$SITE", "name": "demovbox", "virtualbox_username": "< username >", "virtualbox_password": "< password >", "description": "Demo VirtualBox Host/Complex" }
EOF

should output something like:

{"description": "Demo VirtualBox Host/Complex", "updated": "2019-03-05T03:29:33.401162+00:00", "site": "/api/v1/Site/Site:site1:", "built_percentage": 90, "virtualbox_password": "asdf", "name": "demovbox", "virtualbox_username": "asdf", "state": "planned", "created": "2019-03-05T03:29:33.401328+00:00", "members": [], "type": "VirtualBox"}

Now we add the structure host we manually created as a member of the complex, replace < structure id > with the id from the manul host structure from above:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Building/ComplexStructure
{ "complex": "/api/v1/Building/Complex:demovbox:", "structure": "/api/v1/Building/Structure:< structure id>:" }
EOF

should output something like:

{"complex": "/api/v1/Building/Complex:demovbox:", "structure": "/api/v1/Building/Structure:2:", "created": "2019-02-20T04:55:31.730431+00:00", "updated": "2019-02-20T04:55:31.730357+00:00"}

now to set the ip address, this is the ip address of virtualbox the host. This is the same ip that we passed to vboxwebsrv, which is offset 1 of the internal network we created:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Utilities/Address
{ "networked": "/api/v1/Utilities/Networked:< structure id >:", "address_block": "/api/v1/Utilities/AddressBlock:1:", "interface_name": "eth0", "offset": 1, "is_primary": true }
EOF

which should output something like:

{"netmask": "255.255.255.0", "updated": "2019-02-23T18:51:53.521628+00:00", "type": "Address", "prefix": "24", "vlan": 0, "ip_address": "192.168.13.22", "interface_name": "eth0", "network": "192.168.13.0", "sub_interface": null, "address_block": "/api/v1/Utilities/AddressBlock:1:", "is_primary": false, "offset": 22, "pointer": null, "gateway": "192.168.13.1", "created": "2019-02-23T18:51:53.521652+00:00", "networked": "/api/v1/Utilities/Networked:2:"}

Contractor is now running, now let’s configure it to make a VM.

Creating a VM (Ubuntu)

First we need to load the ubuntu blueprints:

sudo respkg -i contractor-ubuntu-base_0.4.respkg

Now we create the Foundation of the VM to be created:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/$FMDL
{ "site": "$SITE", "locator": "testvm01", "blueprint": "$FBP" $FDATA }
EOF

output:

{"state": "planned", "site": "/api/v1/Site/Site:site1:", "type": "VirtualBox", "id_map": "", "virtualbox_complex": "/api/v1/VirtualBox/VirtualBoxComplex:demovbox:", "blueprint": "/api/v1/BluePrint/FoundationBluePrint:virtualbox-vm-base:", "built_at": null, "locator": "tesvm01", "located_at": null, "updated": "2019-02-20T04:58:52.855473+00:00", "created": "2019-02-20T04:58:52.855507+00:00", "class_list": "['VM', 'VirtualBox']", "virtualbox_uuid": null}

create the interface:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Utilities/RealNetworkInterface
{ "foundation": "/api/v1/Building/Foundation:testvm01:", "name": "eth0", "physical_location": "eth0", "is_provisioning": true }
EOF

output:

{"pxe": null, "name": "eth0", "is_provisioning": true, "physical_location": "eth0", "updated": "2019-02-25T14:28:36.245466+00:00", "mac": null, "foundation": "/api/v1/Building/Foundation:testvm01:", "created": "2019-02-25T14:28:36.245500+00:00"}

Now we will create a VM with the Ubuntu Bionic blueprint:

cat << EOF | curl -i "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Building/Structure
{ "site": "$SITE", "foundation": "/api/v1/Building/Foundation:testvm01:", "hostname": "testvm01", "blueprint": "/api/v1/BluePrint/StructureBluePrint:ubuntu-bionic-base:" }
EOF

once again take node of the structure id. Now we assign and ip address, we will let contractor pick, we are going to use the helper method nextAddress. Replace < structure id > with the structure id from the previous call:

cat << EOF | curl "${COPS[@]}" --data @- -X CALL "${CHOST}${ADRBLK}(nextAddress)"
{ "networked": "/api/v1/Building/Structure:< structure id >:", "interface_name": "eth0", "is_primary": true }
EOF

output:

"/api/v1/Utilities/Address:30:"

Contractor will not auto-start the create (nor destroy) jobs. So we need to add two jobs, one to create the Foundation and one to create the Structure:

curl "${COPS[@]}" -X CALL "${CHOST}/api/v1/Building/Foundation:testvm01:(doCreate)"
curl "${COPS[@]}" -X CALL "${CHOST}/api/v1/Building/Structure:< structure id >:(doCreate)"

Now to see it get built. Pull up the http://<contractor ip> in a web browser if you don’t have it open allready, go to the Job Log should see an entry saying that the foundation build has started. Goto the Jobs should see a Foundation or Structure Job there. The Foundation Job won’t last long. In the top right of the page is a refresh and auto refresh buttons.

After the Foundation job completes, a Structure job will auto start, after it completes your VM should be up and sshable, however the default for ubuntu is to disallow sshing as root, but we can show the ssh service is listening:

nc -vz testvm01 22

should output something like:

Connection to testvm01 22 port [tcp/ssh] succeeded!

If you pull up the console, the default root password is “root”.

After you have verified that it is there, logout of the test vm and kick off a job to delete it and re-build it:

/usr/lib/contractor/util/boss -f testvm01 --do-destroy

We set the structure to auto-build, so after it get’s done with the remove job, it will create it again.

Creating a VM (CentOS)

Ok, let’s create a centos VM now, is’t all the same as the ubuntu VM except the blueprint we choose.

Load the centos Blueprints:

sudo respkg -i contractor-centos-base_0.4.respkg

Foundation:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/$FMDL
{ "site": "$SITE", "locator": "testvm02", "blueprint": "$FBP" $FDATA }
EOF

output:

{"state": "planned", "site": "/api/v1/Site/Site:site1:", "type": "VirtualBox", "id_map": "", "virtualbox_complex": "/api/v1/VirtualBox/VirtualBoxComplex:demovbox:", "blueprint": "/api/v1/BluePrint/FoundationBluePrint:virtualbox-vm-base:", "built_at": null, "locator": "tesvm01", "located_at": null, "updated": "2019-02-20T04:58:52.855473+00:00", "created": "2019-02-20T04:58:52.855507+00:00", "class_list": "['VM', 'VirtualBox']", "virtualbox_uuid": null}

create the interface:

cat << EOF | curl "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Utilities/RealNetworkInterface
{ "foundation": "/api/v1/Building/Foundation:testvm02:", "name": "eth0", "physical_location": "eth0", "is_provisioning": true }
EOF

output:

{"pxe": null, "name": "eth0", "is_provisioning": true, "physical_location": "eth0", "updated": "2019-02-25T14:28:36.245466+00:00", "mac": null, "foundation": "/api/v1/Building/Foundation:testvm02:", "created": "2019-02-25T14:28:36.245500+00:00"}

Now we will create a VM with the CentOS7 blueprint:

cat << EOF | curl -i "${COPS[@]}" --data @- -X CREATE $CHOST/api/v1/Building/Structure
{ "site": "$SITE", "foundation": "/api/v1/Building/Foundation:testvm02:", "hostname": "testvm02", "blueprint": "/api/v1/BluePrint/StructureBluePrint:centos-7-base:" }
EOF

output:

HTTP/1.1 201 CREATED
Date: Mon, 11 Mar 2019 13:45:58 GMT
Server: Apache/2.4.18 (Ubuntu)
Cache-Control: no-cache
Verb: CREATE
Cinp-Version: 0.9
Object-Id: /api/v1/Building/Structure:4:
Access-Control-Expose-Headers: Method, Type, Cinp-Version, Count, Position, Total, Multi-Object, Object-Id, Id-Only
Access-Control-Allow-Origin: *
Content-Length: 413
Content-Type: application/json;charset=utf-8

{"hostname": "testvm02", "created": "2019-03-11T13:45:58.963923+00:00", "config_values": null, "config_uuid": "d8821d29-f884-4c2d-af63-7d0292b2ce41", "updated": "2019-03-11T13:45:58.963901+00:00", "blueprint": "/api/v1/BluePrint/StructureBluePrint:centos-7-base:", "site": "/api/v1/Site/Site:site1:", "foundation": "/api/v1/Building/Foundation:testvm02:", "built_at": null, "state": "planned"}

and assign the ip address, make sure to use the structure id from the testvm02 structure:

cat << EOF | curl "${COPS[@]}" --data @- -X CALL "${CHOST}${ADRBLK}(nextAddress)"
{ "networked": "/api/v1/Building/Structure:< structure id >:", "interface_name": "eth0", "is_primary": true }
EOF

output:

"/api/v1/Utilities/Address:30:"

Once again create the create jobs:

curl "${COPS[@]}" -X CALL "${CHOST}/api/v1/Building/Foundation:testvm02:(doCreate)"
curl "${COPS[@]}" -X CALL "${CHOST}/api/v1/Building/Structure:< structure id >:(doCreate)"

Again the jobs should be running to create the CentOS VM. When it is done, ssh in:

ssh root@testvm02

go a head and play around with it for a bit. Make sure to try deconfiguring both VMs at the same time so you can see Contractor do more than one thing at a time.

Accessing Configuration Information

Contractor provides three configuration urls for a target. The first two depend on what the target is set to PXE boot to, the third is all the configuration information for that target in JSON format.

ssh into one of the VMs, this will show what it is like from testvm02:

ssh root@testvm02

First the pxe script, this script is downloaded and run but the iPXE boot loader that the target VM was told to download from the dhcp server.:

curl http://contractor/config/boot_script/

which will output:

#!ipxe

echo Booting form Primary Boot Disk
sanboot --no-describe --drive 0x80 || echo Primary Boot Disk is not Bootable

The VM is current set to the normal-boot pxe. That script tells it to boot to the first harddrive.

Next the pxe_template:

curl http://contractor/config/pxe_template/

output:

# Normal Boot

once again, with this VM having been set to normal-boot, the pxe template is just a comment at the top. The pxe_template is stored as a Jinja2 template that is combined with the configuration information and served out to the target. This is the URL that is used for the Kickstart and/or Pressed files for the CentOS and Debian installers. The source for the CentOS and Ubuntu boot_scripts and pxe_templates are at https://github.com/T3kton/resources/blob/master/os-bases/centos/usr/lib/contractor/resources/centos.toml and https://github.com/T3kton/resources/blob/master/os-bases/ubuntu/usr/lib/contractor/resources/ubuntu.toml those are then packaged during when you built the resources, and installed to /usr/lib/contractor/resource/ when the resource package was installed.

The third url is:

curl http://contractor/config/config/

output:

{"installer_pxe": "centos-7", "__pxe_template_location": "http://contractor/config/pxe_template/", "_structure_config_uuid": "118e0e44-457e-47df-b8c0-d157d5dde1b4", "mirror_server": "mirror.centos.org", "_blueprint": "centos-7-base", "__timestamp": "2019-03-11T14:32:27.909856+00:00", "_foundation_state": "built", "domain_name": "site1.test", "dns_search": ["site1.test", "test"], "_structure_state": "built", "__pxe_location": "http://static/pxe/", "distro": "centos", "_hostname": "testvm02", "_foundation_class_list": ["VM", "VCenter"], "dns_servers": ["10.0.0.10"], "memory_size": 2048, "_foundation_type": "VCenter", "_provisioning_interface": "eth0", "_vcenter_complex": "demovcenter", "_interface_map": {"eth0": {"physical_location": "eth0", "name": "eth0", "mac": "00:50:56:03:1e:6d", "address_list": [{"vlan": null, "address": "10.0.0.123", "prefix": 24, "netmask": "255.255.255.0", "primary": true, "sub_interface": null, "network": "10.0.0.0", "tagged": false, "gateway": null, "auto": true, "mtu": 1500}]}}, "_foundation_locator": "testvm02", "_vcenter_uuid": "52545577-0025-e8d7-1915-bd64585f47c1", "_vcenter_cluster": "localhost.", "_site": "site1", "ntp_servers": ["ntp.ubuntu.com"], "distro_version": "7", "_fqdn": "testvm02.site1.test", "mirror_proxy": "http://10.0.0.10:3128/", "_foundation_interface_list": [{"physical_location": "eth0", "name": "eth0", "mac": "00:50:56:03:1e:6d", "address_list": [{"vlan": null, "address": "10.0.0.123", "prefix": 24, "netmask": "255.255.255.0", "primary": true, "sub_interface": null, "network": "10.0.0.0", "tagged": false, "gateway": null, "auto": true, "mtu": 1500}]}], "__contractor_host": "http://contractor/", "_foundation_id": "testvm02", "vcenter_guest_id": "rhel7_64Guest", "swap_size": 512, "_structure_id": 4, "__last_modified": "2019-03-11T14:01:18.090983+00:00", "_provisioning_interface_mac": "00:50:56:03:1e:6d", "_vcenter_datacenter": "ha-datacenter", "virtualbox_guest_type": "RedHat_64", "root_pass": "$6$rootroot$oLo.loyMV45VA7/0sKV5JH/xBAXiq/igL4hQrGz3yd9XUavmC82tZm1lxW2N.5eLxQUlqp53wXKRzifZApP0/1"}

This url can be used by what ever scripts/CMS as a source of configuration information. See the documentation at Configuration Values for more information on how these values are compiled. One value to point out here is _structure_config_uuid, this value is set when the structure record is created or when the structure is destroyed. This way if there is a stale copy of the structure (old VM snapshot, or a VM that didn’t get cleaned up properly, etc) comes on-line, it (or some other monitoring system) can detect that it is no longer current and take action.

Contractor uses the source ip address of this URL requests to determine which target’s information to return. You can also use the structure id, foundation locater or config uuid, to tell contractor which configuration to return.

by config uuid:

curl http://contractor/config/config/c/118e0e44-457e-47df-b8c0-d157d5dde1b4

by structure id:

curl http://contractor/config/config/s/4

by foundation locator:

curl http://contractor/config/config/f/testvm02

one way for a target to detect if it is slit good and in cases when the ip address might change, is to request it’s config by the uuid.

Removing the VMs

We can either Delete the VMs with the boss command:

/usr/lib/contractor/util/boss -s <structure id> --do-destroy --wait
/usr/lib/contractor/util/boss -f <locator> --do-destroy --wait
/usr/lib/contractor/util/boss -s <structure id> --delete
/usr/lib/contractor/util/boss -f <locator> --delete

or via the API:

curl "${COPS[@]}" -X CALL "${CHOST}/api/v1/Building/Structure:< structure id >:(doDestroy)"

wait for that job to complete, then:

curl "${COPS[@]}" -X CALL "${CHOST}/api/v1/Building/Foundation:< locator >:(doDestroy)"

wait for that job to complete, and finally:

curl "${COPS[@]}" -X DELETE "${CHOST}/api/v1/Building/Structure:< structure id >:"
curl "${COPS[@]}" -X DELETE "${CHOST}/api/v1/Building/Foundation:< locator >:"

Now the VM is no longer in virtualbox/vcenter nor contractor.

The boss command can also trigger jobs, and set the status of foundations and structures. See:

/usr/lib/contractor/util/boss --help

for more info.