Proxmox Auto Install Notes

https://pve.proxmox.com/wiki/Automated_Installation

Installation on WSL

sudo -i # to become root
echo "deb [arch=amd64] http://download.proxmox.com/debian/pve bookworm pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list
wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg
apt update
apt install -y xorriso proxmox-auto-install-assistant

Answer File

Its a toml file and we use Tomlyn to load and save Models in C# to toml files.

format is like this:

[SECTION]
KEY = "VALUE"
KEY = [
    "VALUE1",
    "VALUE2"
]

it has these sections:

  • global

  • network

  • network.interface-name-pinning

  • network.interface-name-pinning.mapping

  • disk-setup

  • post-installation-webhook

  • first-boot

you can validate with

proxmox-auto-install-assistant validate-answer answer.toml

global

keyboard: de de-ch dk en-gb en-us es fi fr fr-be fr-ca fr-ch hu is it jp lt mk nl no pl pt pt-br se si tr

country: 2 letter code

fqdn option 1: HOSTNAME.DOMAIN

fqdn option 2: fqdn.source = "from-dhcp", fqdn.domain = "FALLBACKDOMAIN" → DHCP Server MUST supply a hostname

mailto: root user email

timezone: standard unix timezone, eg Europe/Zurich, or UTC

root-password: plain text (dont want), NOT use with hashed version

root-password-hashed: pre hashed with mkpasswd (via WSL on windows)

root-ssh-keys: optional ssh keys for root user

reboot-on-error: false (will halt on installation fail)

reboot-mode: reboot, (power-off doesnt make much sense)

network

either sets static from dhcp or all other options must be supplied

dhcp

source = "from-dhcp"

manual

source = "from-answer"

cidr = "x.x.x.x/N"

dns = "DNS IP ADDRESS"

gateway = "ROUTER IP ADDRESS"

filter: required for from-answer, not supported for from-dhcp. Uses dotted-key format with uppercase UDEV property names.

[network]
source = "from-answer"
cidr = "10.10.10.10/24"
dns = "10.10.10.1"
gateway = "10.10.10.1"
filter.ID_NET_NAME = "*"

Or to match by MAC-based name:

filter.ID_NET_NAME_MAC = "*AABBCCDDEEFF"
Note
 MAC filter always has a * prefix because the interface name always has a prefix like "enx".

On Odroid devices, the first MAC is extracted from the serial number (which is a concatenation of all MACs, capped to 32 hex chars).

Liberator install-time override (Phase 1 IP scheme)

ProxmoxAutoInstallExtensions.ToProxmoxAutoInstall Liberator-gates the network section. When the config has an OPNsense VM targeted at this host’s serial (IsLiberatorHost), the answer.toml emits the Mikrotik LAN bootstrap IPs regardless of GatewayIP / customer OU subnet:

Field Value (Liberator only)

cidr

10.0.0.200/24 (LiberatorNetwork.ProxmoxBootstrapIp + MikrotikLanCidr)

gateway

10.0.0.1 (LiberatorNetwork.MikrotikLanIp)

dns

10.0.0.1 (Mikrotik forwards upstream until NixDns is healthy)

Reason: on a fresh Liberator install, OPNsense + NixDns + VLAN 250 trunk do not exist yet, so the only reachable gateway is the factory Mikrotik LAN. Phase 2 re-IPs the host onto mgmt VLAN 250 (vmbr0v250 at 10.0.250.2/24 gw 10.0.250.1 dns NixDns) every netsetup update cycle via ConfigureProxmoxBridges. All constants come from src/NETSetup/Helpers/LiberatorNetwork.cs - no literals in the extension.

Test bench override: LiberatorNetwork.BootstrapOverride is a static Func seam that the Hyper-V e2e test sets so the Proxmox VM boots reachable from the runner (no Mikrotik at 10.0.0.1 on the test bench). Production leaves the override null.

Non-Liberator hosts keep the existing path: GatewayIP (with Get<IRouter>().IP fallback), customer OU subnet CIDR, DNS = config DNS server. See docs/Liberator.adoc "Proxmox Host IP - Two Phases" for the canonical table + Serena memory liberator_proxmox_ip_phases.

network.interface-name-pinning

enabled: true/false → controls if mapping section is used

network.interface-name-pinning.mapping

"MAC ADDRESS" = "INTERFACE NAME"

disk-setup

filesystem: ext4, xfs, zfs, btrfs → we want btrfs because snapshots make backups easy and small

disk-list: list of all disk devnames (eg sda, nvme0n) that should be added, ONLY that or filter

filter-match: "any" or "all" - required when using filter. "any" = match if any criterion matches, "all" = all must match.

filter: uses dotted-key format with uppercase UDEV property names. Supports ID_WWN, ID_SERIAL, ID_MODEL, ID_VENDOR. Wildcards (*) supported. Proxmox installer matches the filter value against udev properties of every disk it sees - the disk(s) that match are the install target.

[disk-setup]
filesystem = "btrfs"
filter-match = "any"
filter.ID_WWN = "*eui.e8238fa6*"
Disk-filter resolution chain (NETSetup)

NETSetup picks ONE udev key per target disk via ProxmoxDiskFilterResolver.Resolve(IDisk, ILogger). Chain runs highest priority to lowest, value lower-cased:

  • ID_WWN - taken from MSFT_PhysicalDisk.UniqueId when UniqueIdFormat is in {1, 2, 3, 8} (SCSI Name String / EUI-64 / NAA / T10). Lower-cased. Proxmox matches udev ID_WWN (which is the same identifier, lower-cased, on Linux). Best identifier for NVMe.

  • ID_MODEL - used when WWN absent AND the model is unique across all MSFT_PhysicalDisk rows on this host. Spaces become _, lower-cased. Also picked when MSFT enumeration is empty (cannot tell collisions → assume unique = fail-safe).

  • ID_SERIAL - last fallback. Lower-cased. SATA / SCSI friendly. Broken for many NVMe (see incident below).

Fail-safe: if WMI throws, returns nothing, or matches but UniqueId is unusable → chain falls through to the next step. Throws NETSetupException only when WWN, Model AND Serial are ALL empty.

The extension layer wraps the chosen value in …​ wildcards before handing it to the builder, so a partial WWN match still works if Proxmox decorates the property.

Why WWN beats serial (Odroid H4+ + WD Red SN700 incident)

Windows WMI Win32_DiskDrive.SerialNumber on NVMe returns a MANGLED EUI - underscore-separated hex chunks plus a trailing dot, e.g. E823_8FA6_…​3C63.. Linux udev ID_SERIAL for the SAME disk is <model><real-namespace-serial> - totally different string. Result: a baked filter.ID_SERIAL = "E823_8FA6_…​_3C63." from Windows NEVER matches the installer’s view, Proxmox auto-install hangs at disk selection.

The real WWN lives in Linux udev ID_WWN (eui.xxxxxxxxxxxxxxxx, lower-case). The same identifier on Windows lives in MSFT_PhysicalDisk.UniqueId (uppercase, no eui. prefix). With UniqueIdFormat = 8 (SCSI Name String) the two strings are equal modulo case. Lower-case it, match on ID_WWN, done.

Concrete repro: Odroid H4+ + WD Red SN700 NVMe used to halt on filter.ID_SERIAL. After the resolver, it emits filter.ID_WWN = "eui.e8238fa6…​" and the installer picks the disk on first try.

How to verify the filter manually

Boot the Proxmox installer. Hit Ctrl+Alt+F2 to get a shell. Then:

# All by-id symlinks Proxmox can match against. WWN, SERIAL, MODEL all appear here.
ls -l /dev/disk/by-id/

# Full udev properties for one disk - look for ID_WWN, ID_SERIAL, ID_MODEL.
udevadm info --query=property --name=/dev/nvme0n1

# Block-device view with model + serial inline.
lsblk -o NAME,MODEL,SERIAL,WWN,SIZE,TYPE

If the value emitted in answer.toml does NOT show up verbatim (modulo * wildcards) in udevadm info, the filter will not match. The 3 most common drifts: WWN appears as eui.e823…​ on Linux but E823…​ on Windows MSFT, NVMe serial is mangled by WMI, model strings differ by trailing whitespace.

Builder methods
Method Use

WithDiskWwn(wwn, filterMatch="any")

NVMe-safe (and the resolver-preferred path). Maps to filter.ID_WWN.

WithDiskSerialNumber(serial, filterMatch="any")

Legacy SATA / SCSI path. Maps to filter.ID_SERIAL.

WithDiskModel(model, filterMatch="any")

When model is unique on the host. Maps to filter.ID_MODEL.

WithDiskFilter(filterMatch, wwn, serial, model, vendor)

Power user - set multiple criteria + choose any/all. WWN is the first optional named arg (after filterMatch) so all existing call-sites that use named args keep working.

WithDiskList(params disks)

Bypass filtering, pin to devnames like /dev/nvme0n1. Used when no targetDisk is known.

ProxmoxAutoInstallExtensions.ToProxmoxAutoInstall calls the resolver for any host that has a target disk and delegates to the matching WithDisk* method based on the resolved key.

zfs…​ options

lvm…​ options

btrfs…​ options ?

all support the optional options:

  • raid: raid0, raid1, raid10

  • hdsize: set max size in GB to be used of disk

  • compress: on, off, zlib, lzo, zstd (especially for btrfs)

post-installation-webhook

sends a post request with the system information to a url

url: post endpoint

cert-fingerprint: optional sha256 of the tls/ssl cert to verify endpoint

first-boot

optional, specify executable to run first time after boot, NETSetup ;)

source = "from-url" OR "from-iso"

ordering = "before-network" or "fully-up" → when to run the executable → then we can already auto create vms :D

url: where to download from, PUBLIC URL DIRECT DOWNLOAD (only for from-url)

cert-fingerprint: optional sha256 of the tls/ssl cert to verify endpoint (only for from-url)

from-iso details

The auto-installer reads the executable from /cdrom/proxmox-first-boot (max 1 MiB). It must be baked into the ISO at build time with prepare-iso --on-first-boot <script>. The installer copies it to /var/lib/proxmox-first-boot/proxmox-first-boot on the target, enables a systemd service (proxmox-first-boot-{ordering}.service), and runs it once on first boot.

Source: pve-installer/proxmox-auto-installer/src/bin/proxmox-auto-installer.rs line 49, pve-installer/Proxmox/Install.pm function setup_proxmox_first_boot_service.

NETSetup first-boot bootstrap (DONE)

A generic bootstrap script (netsetup-first-boot.sh) is baked into every prepared ISO. It runs once on first boot (via PVE’s proxmox-first-boot-fully-up.service) and:

  1. Downloads the NETSetup linux-x64 ELF binary from Dropbox

  2. Copies machine-specific config from the SYSTEM partition (if USB still present)

  3. Creates a systemd service + timer for periodic NETSetup runs (on boot + every 6h)

  4. Runs NETSetup install immediately

The script is generic — machine identity comes from the running system (serial number, MAC, hostname from answer.toml).

Source: src/NETSetup/Resources/netsetup-first-boot.sh

C# side: WithFirstBootFromIso() in ProxmoxAutoInstallBuilder generates the [first-boot] section in answer.toml. Called automatically by ToProxmoxAutoInstall() for all Proxmox configs.

Assistant Usage / ISO Creation

We want 1 iso to rule them all. We achieve this by modifying a regular iso only to look for the partition SYSTEM and in there for the answer file. The file must lie directly on partition root with name "answer.toml".

# 1. download current ISO from https://www.proxmox.com/en/downloads/proxmox-virtual-environment/iso
wget -O ISOPATH https://enterprise.proxmox.com/iso/proxmox-ve_9.1-1.iso

# 2. patch ISO: answer-file from SYSTEM partition + first-boot script baked in
proxmox-auto-install-assistant prepare-iso ISOPATH \
    --fetch-from partition --partition-label SYSTEM \
    --on-first-boot netsetup-first-boot.sh

Now you have the current iso to put into Dropbox Images, so we can later download using NETSetup.

Be sure to update the README.txt so its clear when and what are latest version.

WIP: HTTP Fetch via install.iam.free (per-host answer, zero-rebake)

Status: PLAN. Not implemented. Replaces the SYSTEM-partition flow above once landed.

Goal

One generic Proxmox ISO. No per-host rebake. No per-host USB stick prep. Tech boots target machine off the same ISO every time. Installer pulls host-specific answer.toml from https://install.iam.free/answer keyed by the machine’s host serial. Host serial is resolved upstream in netbase osisa.SystemManagement.Windows.Entities.WindowsComputerHardware.GetSerialNumber() via the chain BIOS → BaseBoard → primary MAC → Mainboard.UUID, so whitebox boards with blank/placeholder DMI (To be filled by O.E.M.) already produce a stable MAC-derived serial without any NETSetup-side fallback helper. Requires netbase pkg osisa.SystemManagement.Windows >= 0.16.0-ModelFeature.9151.

Why

Today: EnsureProxmoxIso + prepare-iso …​ --fetch-from partition writes one custom USB per host. Per-host hostname/IP/pw baked into a per-host answer.toml on the SYSTEM partition. Cost: USB rebake every install, human step, drift between bake-time and install-time config.

HTTP-fetch flow: ISO is generic. Answer is fetched at install time from the always-on osisa endpoint. Single source of truth = the customer config JSON in /NETSetup/Config/<customer>/, served on demand.

Architecture

[target box]                 [install.iam.free]              [osisa infra]
boot generic ISO  --POST-->   answer server   --reads-->    NETSetupConfig
                                                            (customer JSONs)
                <--TOML-----  (matched by serial)

  1. Target boots generic Proxmox ISO baked with HTTP fetch URL + cert fingerprint + auth token.

  2. Installer POSTs system info JSON (board serial, MACs, disks, CPU, RAM) to https://install.iam.free/answer with Authorization: Bearer …​. "Board serial" here is the netbase-resolved host serial (BIOS → BaseBoard → MAC → UUID chain).

  3. Server matches request to a known host via host serial. Looks up customer config. Renders per-host answer.toml. Returns it.

  4. Installer applies answer, installs Proxmox, runs baked netsetup-first-boot.sh (unchanged).

ISO bake (one-time)

proxmox-auto-install-assistant prepare-iso /path/to/source.iso \
    --fetch-from http \
    --url "https://install.iam.free/answer" \
    --cert-fingerprint "<sha256 of install.iam.free TLS cert>" \
    --answer-auth-token "netsetup:<shared-secret>" \
    --on-first-boot netsetup-first-boot.sh

Output ISO goes to Dropbox NETSetup/BOOT/Images/proxmox.iso once per Proxmox-version bump. No per-host variant. Tech burns one USB ever.

Token is plaintext in the ISO → treat ISO as sensitive. Rotate secret on suspected ISO leak.

Answer server contract

Endpoint: POST https://install.iam.free/answer

Request:

Response:

  • 200 application/toml with rendered answer.toml → install proceeds.

  • 404 if board serial unknown to NETSetupConfig → installer halts (we WANT halt, not generic fallback - prevents wrong-machine install).

  • 401 on bad token. 503 on backend lookup failure.

Identity match order (server side):

  1. system.dmi.system.serial (raw BIOS serial) → NETSetupConfig host SerialNumber.

  2. system.dmi.baseboard.serial (raw BaseBoard serial) → NETSetupConfig host SerialNumber.

  3. Any network.nics[].mac → NETSetupConfig host SerialNumber (when host serial was MAC-derived upstream by netbase).

  4. system.dmi.system.uuid (mainboard UUID, last-resort) → NETSetupConfig host SerialNumber.

  5. No match → 404.

Note
 Order matches the netbase WindowsComputerHardware.GetSerialNumber() chain (BIOS → BaseBoard → MAC → UUID). UUID is last because it is a GUID-with-dashes shape that pollutes map-file SerialNumber#TicketNumber.map matching when the upstream BIOS/BaseBoard slots hold placeholder values like To be filled by O.E.M.. Customer JSONs should be populated with whichever value the target box actually emits - the netbase resolver picks it deterministically, so the map-file name will line up.

Render logic on the server: read the customer JSON, find the host by serial, build ProxmoxAutoInstallToml exactly the same way C# does today, serialize via Tomlyn, return.

Hosting / infra

  • DNS: install.iam.free A record + AAAA on osisa cloud.

  • TLS: real cert (Let’s Encrypt) preferred so fingerprint pinning is the belt-and-braces, not the only line. Fingerprint still gets baked because installer’s CA store may not trust LE on day one of a new ISO build (Proxmox installer is minimal).

  • Backend: tiny service. Reads from a synced copy of /NETSetup/Config/<customer>/ (same data NETSetup CLI reads). Could live as a NETSetup subcommand: netsetup answer-server listens on :443, serves answers.

NETSetup code changes

  1. ProxmoxAutoInstallBuilder gains WithFetchFromHttp(url, fingerprint, token). Replaces WithFetchFromPartition() in ToProxmoxAutoInstall() once the new flow ships. Old method kept until cutover.

  2. EnsureProxmoxIso (or whichever writes the ISO today) drops per-host bake. New path = bake-once and publish to Dropbox.

  3. New CLI cmd: netsetup answer-server - hosts the HTTP endpoint, reads customer JSONs, renders TOML per request. Lives on osisa cloud box that resolves to install.iam.free. Same NETSetupConfig + Tomlyn pipeline as today, just behind HTTP.

  4. netsetup-first-boot.sh unchanged - still bootstraps NETSetup binary and registers systemd timer. Machine identity still comes from running system.

  5. Stop writing answer.toml to USB SYSTEM partition. Delete USB-prep paths in RunProxmoxUpdate/EnsureProxmoxIso.

Files to touch:

  • src/NETSetup/Entities/Builders/ProxmoxAutoInstallBuilder.cs - new builder method

  • src/NETSetup/Extensions/ProxmoxAutoInstallExtensions.cs - swap defaults

  • src/NETSetup/Helpers/UpdateMethods.cs - drop per-host USB bake

  • src/NETSetup/Stages/2-WinPE/MainWinPE.cs - drop SYSTEM partition write

  • new src/NETSetup/Commands/AnswerServerCommand.cs + osisa.NETCommand HTTP listener wiring

Failure modes + mitigations

  • install.iam.free unreachable at install time → installer hangs. Mitigation: provisioning tech runs the answer server locally as fallback (netsetup answer-server --bind 192.168.x.x:443), customer LAN routes there via DHCP option 6 + DNS override on tech laptop, or point ISO at tech laptop directly via different bake.

  • Wrong machine boots ISO → server returns 404 by design, installer halts. Safer than today (USB-bound answer can still get plugged into wrong box).

  • Token leak → rotate, rebake ISO, ship new ISO.

  • Cert rotation → fingerprint changes → ISO needs rebake. Solution: use real LE cert (installer trusts CA store), pin fingerprint only as defence-in-depth and accept "fingerprint rotation = rebake".

Open questions

  • Does the Proxmox installer’s CA store trust Let’s Encrypt out of the box on the current ISO version we ship? If yes, fingerprint is optional and rotations become painless.

  • (RESOLVED 2026-05-27) Identity-by-serial vs MAC fallback: handled upstream in netbase osisa.SystemManagement.Windows >= 0.16.0-ModelFeature.9151. WindowsComputerHardware.GetSerialNumber() already walks BIOS → BaseBoard → MAC → UUID and returns a stable MAC-derived serial when BIOS/BaseBoard are blank or placeholders. NETSetup-side Helpers/HostSerialFallback.GetSerialOrMacFallback() was deleted (commit d97e17ab) - MapFileMethods now reads host.SerialNumber.Value directly. Server-side identity match MUST replicate the same order (see above).

  • Where does install.iam.free actually live? osisa cloud VM, Nextcloud-adjacent, or new box?

  • Do we want per-customer subdomains (<customer>.install.iam.free) to scope token blast radius? Or one token for everyone?

  • Cutover: dual-bake (one HTTP ISO + one SYSTEM ISO) during transition, or hard switch?

Implementation order

  1. Stand up answer server skeleton on dev box (echo back a hardcoded answer for one known serial). Validate end-to-end with one target machine.

  2. Wire customer-JSON lookup + Tomlyn render path into server.

  3. Stand up install.iam.free infra + LE cert.

  4. Add WithFetchFromHttp builder + tests. Ship in NETSetup.

  5. Bake first generic ISO, publish to Dropbox.

  6. Pilot on one new customer install. Old USB flow stays available as fallback.

  7. Cutover. Delete SYSTEM-partition bake paths.