Manuales

h1. Proxmox Mastering

Table of Contents

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the color images of this book

Errata

Piracy

Questions

1. Understanding Proxmox VE and Advanced Installation

Understanding Proxmox features

It is free!

Built-in firewall

Open vSwitch

The graphical user interface

KVM virtual machines

Linux containers, or LXC

Storage plugins

Vibrant culture

The basic installation of Proxmox

The advanced installation option

Debugging the Proxmox installation

Proxmox subscription and repositories

Proxmox VE Enterprise repository

Type

Subscription key

Status

Server ID

Sockets

Last checked

Next due date

Proxmox VE No-Subscription repository

Proxmox VE Test repository

Summary

2. Creating a Cluster and Exploring the Proxmox GUI

Creating a Proxmox cluster

Exploring the Proxmox GUI

The GUI menu system

Cluster tree view

Server View

Folder View

Storage View

Pool View

The Datacenter menu

Datacenter | Search

Datacenter | Summary

Datacenter | Options

Datacenter | Storage

Datacenter | Backup

Datacenter | Permissions

Datacenter | Permissions | Users

Datacenter | Permissions | Groups

Datacenter | Permissions | Pools

Datacenter | Permissions | Roles

Datacenter | Permissions | Authentication

Datacenter | HA

Datacenter | Firewall

Datacenter | Support

Node-specific menus

Node | Search

Node | Summary

Node | Shell

Node | System

Node | Network

Node | DNS

Node | Time

Node | Syslog

Node | Updates

Node | Firewall

Node | Disks

Node | Ceph

Node | Task History

Node | Subscription

KVM menu

KVM VM | Summar

KVM | Options

KVM VM | Task History

KVM | Monitor

KVM | Backup

KVM VM | Snapshot

KVM | Firewall

KVM | Permissions

LXC container menu

LXC container | Summary

LXC container | Resources

LXC container | Network

LXC container | DNS

LXC container | Options

LXC container | Task History

LXC container | Backup

LXC container | Snapshots

LXC container | Firewall

LXC container | Permissions

Pool menu

Pool | Summary

Pool | Members

Pool | Permissions

Summary

3. Proxmox under the Hood

The Proxmox cluster file system

Proxmox directory structure

Dissecting the configuration files

The cluster configuration file

logging { }

nodelist { }

quorum { }

totem { }

interface { }

Storage configuration file

User configuration files

The password configuration file

KVM virtual machine configuration file

Arguments in the KVM configuration file

LXC container configuration file

Version configuration file

Member nodes

Virtual machine list file

The cluster log file

Ceph configuration files

Firewall configuration file

Summary

4. Storage Systems

Local storage versus shared storage

Live migration of a virtual machine

Seamless expansion of multinode storage space

Centralized backup

Multilevel data tiering

Central storage management

Local and shared storage comparison

A virtual disk image

Supported image formats

The .qcow2 images

The .raw image type

The .vmdk image type

Virtual device types

Managing disk images

Resizing a virtual disk image

Moving a virtual disk image

Throttling a virtual disk image

Caching a virtual disk image

VirtIO bus type for Windows VMs

Installing VirtIO drivers during Windows installation

Installing VirtIO drivers after Windows installation

Storage types in Proxmox

Directory

iSCSI

Logical Volume Management

NFS

ZFS

Ceph RBD

GlusterFS

Noncommercial/commercial storage options

Summary

5. Installing and Configuring Ceph

Ceph components

A physical node as cluster member

Maps

A cluster map

A CRUSH map

Monitor

OSD

OSD journal

Metadata server

PG

Pools

Ceph components summary

Virtual Ceph for training

Installing a Ceph cluster

Installing Ceph on Proxmox

Preparing a Proxmox node for Ceph

Installing Ceph

Creating mons from the Proxmox GUI

Creating OSDs from Proxmox GUI

Managing a Ceph pool using Proxmox GUI

Creating a Ceph pool using Proxmox GUI

Connecting Ceph to Proxmox

Ceph command list

Summary

6. KVM Virtual Machines

Exploring KVM

Creating a KVM

Creating a KVM using an ISO image

General tab

Node

VM ID

Name

Resource Pool

Help

The OS tab

The CD/DVD tab

The Hard Disk tab

Bus/Device

Storage

Disk size (GB)

Format

Cache

No backup

Discard

IO thread

The CPU tab

Sockets

Cores

Enabling NUMA

Type

The Memory tab

The Network tab

Bridged mode

Firewall

NAT mode

No network device

Model

MAC address

Rate limit (MB/s)

Multiqueues

Disconnect

Creating VM by cloning

Creating VMs from a template

Target node

Mode

Advanced configuration options for VMs

Configuring a sound device

Configuring PCI passthrough

Configuring GPU passthrough

Preparing for hotplug

Configuring VMs with hotplug

Hotplugging vCPUs

Hotplugging memory

Hotplugging disks/vNICs

Migrating KVM virtual machines

Summary

7. LXC Virtual Machines

Exploring LXC virtual machines

Understanding container templates

Creating an LXC container

General tab

Node

CT ID

Hostname

Unprivileged container

Resource Pool

The Template tab

The Root Disk tab

Storage

ACLs

Enable quota

The CPU tab

Cores

The Memory tab

The Network tab

Name

MAC address

Bridge

The VLAN Tag

Rate limit

Firewall

IPv4/IPv6

The DNS tab

The Confirm tab

Managing an LXC container

Adjusting resources using the GUI

Adjusting resources using the CLI

Adjusting resources using direct modification

Migrating an LXC container

Accessing an LXC container

The noVNC console

Direct shell through the CLI

Converting OpenVZ to LXC

Summary

8. Network of Virtual Networks

Exploring virtual networks

Physical networks versus virtual networks

A physical network

A virtual network

Networking components in Proxmox

Virtual Network Interface Cards

Adding/removing vNIC

A virtual bridge

Adding a virtual bridge through the GUI

Name

IP information

Bridge ports

VLAN-aware

Adding a virtual bridge through CLI

Extra bridge options

bridge_stp

bridge_fd

Virtual LAN

Adding a VLAN

Network Address Translation/Translator

Adding NAT/masquerading

Network bonding

Adding a bonding interface

The layer 2 hash policy

The layer 2+3 hash policy

The layer 3+4 hash policy

Multicast

Configuring multicast on Netgear

Open vSwitch

Features of Open vSwitch

Adding an Open vSwitch bridge

Adding the Open vSwitch bond

Adding Open vSwitch IntPort

CLI for Open vSwitch

Practicing Open vSwitch

Configuration requirements

Solutions

Sample virtual networks

Network #1 – Proxmox in its simplest form

Network #2 – the multi-tenant environment

Network #3 – academic institution

A multi-tenant virtual environment

A multi-tenant network diagram

Summary

9. The Proxmox VE Firewall

Exploring the Proxmox VE firewall

Components of the Proxmox firewall

Zones

Security groups

IPSet

Rules

Protocols

Macros

The pve-firewall and pvefw-logger services

Configuration files of a firewall

Configuring the data center-specific firewall

Configuring the Datacenter firewall through the GUI

Creating the Datacenter firewall rules

Creating the Datacenter IPSet

Creating aliases

Configuring the Datacenter firewall through the CLI

[OPTIONS]

[ALIASES]

[IPSET <name>]

[RULES]

[group <name>]

Configuring a host-specific firewall

Creating host firewall rules

Options for the host zone firewall

Enable a firewall

The SMURFS filter

The TCP flags filter

NDP

nf_conntrack_max

nf_conntrack_tcp_timeout_established

log_level_in/out

tcp_flags_log_level

smurf_log_level

Configuring the host firewall through the CLI

Configuring a VM-specific firewall

Creating VM firewall rules

Creating aliases

Creating IPSets

Options for a VM zone firewall

Enable DHCP

The MAC filter

Input/output policy

Configuring a VM-specific firewall through the CLI

Integrating a Suricata IDS/IPS

Installing/configuring Suricata

Limitations of Suricata in Proxmox

Summary

10. Proxmox High Availability

Understanding HA

HA in Proxmox

How Proxmox HA works

Requirements for HA setup

At least three nodes

Shared storage

Fencing

BIOS power-on feature

Configuring Proxmox HA

The HA menu

Status

The Resources menu

The Groups menu

ID

Node

The restricted checkbox

The nofailback checkbox

The Fencing menu

Testing Proxmox HA configuration

The Proxmox HA simulator

Configuring the Proxmox HA simulator

Summary

11. Monitoring the Proxmox Cluster

An introduction to monitoring

Proxmox built-in monitoring

Datacenter Status

Node Status

Zabbix as a monitoring solution

Installing Zabbix

Configuring Zabbix

Configuring a host to monitor

Displaying data using a graph

Configuring the disk health notification

Installing smart monitor tools

Configuring the Zabbix agent

Creating a Zabbix item in the GUI

Creating a trigger in the GUI

Creating graphs in the GUI

Configuring SNMP in Proxmox

Object Identifiers

Management Information Base

Adding an SNMP device in Zabbix

Monitoring the Ceph cluster with the Proxmox GUI

Monitoring a Ceph cluster with third-party options

Summary

12. Proxmox Production-Level Setup

Defining the production level

Key components

Stable and scalable hardware

Redundancy

Node level

Utility level

Network level

HVAC level

Storage level

Current load versus future growth

Budget

Simplicity

Tracking hardware inventory

Hardware selection

Sizing CPU and memory

Single socket versus multi-socket

Hyper-threading – enable versus disable

Start small with VM resources

Balancing node resources

Ceph cluster production

Forget about hardware RAID

Solid State Drive for Ceph Journal

Network bandwidth

Liquid cooling

Total immersion in oil

Total immersion in 3M Novec

Direct contact liquid cooling

Real-world Proxmox scenarios

Scenario 1 – an academic institution

Scenario 2 – multi-tier storage cluster with a Proxmox cluster

Scenario 3 - Virtual infrastructure for a multi-tenant cloud service provider

Scenario 4 – nested virtual environment for a software development company

Scenario 5 – virtual infrastructure for a public library

Scenario 6 – multi-floor office virtual infrastructure with virtual desktops

Scenario 7 – virtual infrastructure for the hotel industry

Scenario 8 – virtual infrastructure for geological survey organization

Summary

13. Back Up and Restore Virtual Machines

Proxmox backup options

A full backup

Full backup modes

Snapshot

Suspend

Stop

Backup compression

None

LZO

GZIP

Snapshots

Configuring backup storage

Show VM configuration from backup

Configuring full backup

Creating a schedule for backup

Node

Storage

Day of week

Start Time

Selection mode

Send email to

Email notification

Compression

Mode

Enable

Creating a manual backup

Creating snapshots

Restoring a virtual machine

Backup/restore through the CLI

Backup using the CLI

Restore using the CLI

Unlocking a VM after a backup error

Virtual machine replication

Creating a replication task through the GUI

Target

Schedule

Rate limit (MB/s)

Enabled

Creating a replication task through the CLI

Replication process

Backup configuration file

The bwlimit option

The lockwait option

The stopwait option

The stdexcludes option

The mailto option

The script option

The exclude-path option

The pigz option

Summary

14. Updating/Upgrading Proxmox

Introducing Proxmox updates

Updating Proxmox through the GUI

Updating Proxmox through the CLI

Difference between upgrade and dist-upgrade

Recovering from the grub2 update issue

Updating after a subscription change

Rebooting dilemma after Proxmox updates

Applying update without reboot

Summary

15. Proxmox Troubleshooting

Proxmox node issues

Issue – fresh Proxmox install stuck with /dev to be a fully populated error during no

de reboot

Issue – rejoining a node to a Proxmox node with the same old IP address

Issue – Proxmox installation completed but grub is in an endless loop after reboot

Issue – LSI MegaRAID 9240-8i/9240-4i causes an error during booting of the Prox

mox node

Downloading and updating the LSI driver

Updating the Supermicro BIOS

Issue – the Upgrade button is disabled on the Proxmox GUI, which prevents the nod

e upgrade

Issue – Proxmox cannot start due to the getpwnam error

Issue – cannot log in to the GUI as root after reinstalling Proxmox on the same node

The main cluster issues

Issue – Proxmox virtual machines are running, but the Proxmox GUI shows that eve

rything is offline

Issue – kernel panic when disconnecting USB devices, such as a keyboard, mouse,

or UPS

Issue – virtual machines on Proxmox will not shut down if shutdown is initiated fro

m the Proxmox GUI

Issue – kernel panic with HP NC360T (Intel 82571EB chipset) only in Proxmox VE

3.2

Issue – the Proxmox cluster is out of quorum and cluster filesystem is in read-only

mode

Issue – VM will not respond to shutdown or restart

Issue – Proxmox GUI not responding after Firefox update

Issue – the Proxmox GUI is not showing RRD graphs

Storage issues

Issue – deleting a damaged LVM from Proxmox with the error read failed from 0 to

4096

Issue – Proxmox cannot mount NFS share due to the timing out error

Issue – how to delete leftover NFS shares in Proxmox or what to do when the NFS s

tale file handle error occurs?

Issue – Proxmox issues —mode session exit code 21 errors while trying to access th

e iSCSI target

Issue – cannot read an iSCSI target even after it has been deleted from Proxmox stor

age

Issue – a Ceph node is removed from the Proxmox cluster, but OSDs still show up i

n PVE

Issue – the no such block device error during creation of an OSD through the Proxm

ox GUI

Issue – the fstrim command does not trim unused blocks for the Ceph storage

Issue – the RBD couldn’t connect to cluster (500) error when connecting Ceph with

Proxmox

Issue – changing the storage type from IDE to VirtIO after the VM has been set up a

nd the OS has been installed

Issue – the pveceph configuration not initialized (500) error when you click on the

Ceph tab in the Proxmox GUI

Issue – the CephFS storage disappears after a Proxmox node reboots

Issue – VM cloning does not parse in the Ceph storage

Issue – VM disk images stored on ZFS is extremely slow

Network connectivity issues

Issue – no connectivity on Realtek RTL8111/8411 rev. 06 network interfaces

Issue – network performance is slower with the E1000 virtual network interfaces

Issue – patch port for Open vSwitch in Proxmox not working

Issue – trying to add a node to a newly created Proxmox cluster when nodes do not f

orm quorum

Issue – implemented IPv6 but firewall rules do not get applied

KVM virtual machine issues

Issue – Windows 7/XP machine converted to Proxmox KVM hangs during boot

Issue – Windows 7 VM does not reboot, instead it shuts down, requiring a manual b

oot from Proxmox

Issue – the qemu-img command does not convert the .vmdk image files created with

the .ova template in Proxmox VE 5.0

Issue – online migration of a virtual machine fails with a failed to sync data error

Issue – no audio in Windows KVM

Issue – the VirtIO virtual disk is not available during the Windows Server installatio

n

LXC container issues

Issue – a Proxmox node hangs when trying to stop or restart an LXC container

Issue – the noVNC console only shows a cursor for LXC containers

Backup/restore issues

Issue – a Proxmox VM is locked after backup crashes unexpectedly

Issue – how can Proxmox back up only the primary OS virtual disk instead of all th

e virtual disks for a VM?

Issue – backup of virtual machines stops prematurely with an operation not permitte

d error

Issue – a backup task takes a very long time to complete, or it crashes when multipl

e nodes are backing up to the same backup storage

Issue – backup of virtual machines aborts a backup task prematurely

Issue – backup storage has a lot of .dat files and .tmp folders using the storage space

VNC/SPICE console issues

Issue – the mouse pointer is not shared with SPICE (virt-viewer) on Windows 8 VM

Issue – remote viewer is unable to connect to a SPICE-enabled virtual machine on t

he Windows OS

Firewall issues

Issue – rules are created and a firewall is enabled for vNIC, but rules do not get appl

ied

Issue – a firewall is enabled for a VM and the necessary rules are created, but nothin

g is being filtered for that VM

Summary

16. Rescuing Proxmox

Recovering from OS drive failure

Physical drive failure

OS data corruption

Migrating VMs from a faulty node

Reinstalling Proxmox

Recovering from a quorum failure

Recovering from a node failure

Recovering from a network failure

Loss of connectivity between Proxmox nodes

Loss of connectivity between Proxmox nodes and users

Loss of connectivity between Proxmox and storage nodes

Recovering from Ceph failure

Best practices for a healthy Ceph cluster

Stuck inconsistent PGs in Ceph

Stuck inactive incomplete PGs in Ceph

Error while moving a Ceph journal to another drive

Ceph node running out of resources during recovery

Summary

Preface

Based on the foundation laid out by the first edition and second edition, this

book, Mastering Proxmox, Third Edition, brings updated information and

details of the new features of Proxmox VE 5.0. Since the first edition of this

book was published, Proxmox has been through many changes. With this

third edition, I am confident that readers will be able to upgrade their skills

while building and managing even better Proxmox clusters.

This book shows the inner workings of Proxmox, including virtual network

components, shared storage systems, the Proxmox fire