Mini Hyper-converged infrastructure (HCI)
Hardware and BIOS Configuration
Configuring the BIOS in the MS-01
Network Design
Data Networks
| Name | VLAN ID | Subnet | Description |
|---|
| 201_INBAND_MGMT | 201 | 172.28.1.0/24 | Inband Management |
| 202_OOB_MGMT | 202 | 172.28.2.0/24 | Out-of-Band Management |
| 203_VMS | 203 | 172.28.3.0/24 | Virtual Machine Network |
Storage Networks
The following VLANs are running on the storage network switches. They are designed to be non-routed, only layer 2 networks. The storage switches will not be uplinked to any external network for security and to prevent any network outages on the VMs side interfering with storage replication.
| Name | VLAN ID | Subnet | Description |
|---|
| 401_PVE_MIGRATION | 401 | 172.20.1.0/24 | VM Migration Network |
| 402_PVE_CEPH | 402 | 172.20.2.0/24 | CEPH Replication Network |
| 403_PVE_BACKUP | 403 | 172.20.3.0/24 | Proxmox Backup (Future) |
| 404_NAS | 404 | 172.20.4.0/24 | Used for connecting external storage use as a NAS (Future) |
IP Address Allocation
Hypervisors
| Interface | PVE01 | PVE02 | PVE03 |
|---|
| INBAND_MGMT (vmbr0) | 172.28.1.11 | .12 | .13 |
| OOB | 172.28.3.11 | .12 | .13 |
| 401_PVE_MIGRATION | 172.20.1.11 | .12 | .13 |
| 402_PVE_CEPH | 172.20.1.11 | .12 | .13 |
| 403_PVE_BACKUP | 172.20.1.11 | .12 | .13 |
| 404_NAS | 172.20.1.11 | .12 | .13 |
Network
| Device | Hostname | Port | IP | Network/VLAN |
|---|
| CRS317-1G-16S | SW_STOR_01 | 172.28.1.21 | 201_INBAND_MGMT | |
| CRS317-1G-16S | SW_STOR_02 | 172.28.1.22 | 201_INBAND_MGMT | |
Storage Switch Configuration
The default login is user: admin and the password is blank. When logging in for the first time you will be asked to change the password. General good practive as well, I would recommend upgrading the RouterOS and the Baseboard firmware to the latest version. This can be done fairly easily using the WinBox GUI. If you have no internet the firmware files can be dragged and dropped on the WinBox UI.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
| # Switch 01
/system identity set name=SW_STOR_01
# Switch 02
/system identity set name=SW_STOR_02
# Switch 01
/ip address add address=172.28.1.21/24 interface=ether1
# Switch 02
/ip address add address=172.28.1.22/24 interface=ether1
# Setup the gateway, default route, DNS servers, and NTP
/ip route add gateway=172.28.1.1
/ip dns set servers=9.9.9.9,1.1.1.1
/system ntp client set enabled=yes
/system ntp client servers add address=jp.pool.ntp.org
# Create Bridge and Enable VLAN Filtering
/interface bridge add name=bridge1 vlan-filtering=yes
# Add Ports to Bridge
/interface bridge port
add bridge=bridge1 interface=sfp-sfpplus1
add bridge=bridge1 interface=sfp-sfpplus2
add bridge=bridge1 interface=sfp-sfpplus3
add bridge=bridge1 interface=sfp-sfpplus13
add bridge=bridge1 interface=sfp-sfpplus14
add bridge=bridge1 interface=sfp-sfpplus15
add bridge=bridge1 interface=sfp-sfpplus16
# Port descriptions
/interface ethernet
set sfp-sfpplus1 comment="Port 1 - PVE01"
set sfp-sfpplus2 comment="Port 2 - PVE02"
set sfp-sfpplus3 comment="Port 3 - PVE03"
set sfp-sfpplus13 comment="Peer Link Port 13"
set sfp-sfpplus14 comment="Peer Link Port 14"
set sfp-sfpplus15 comment="Peer Link Port 15"
set sfp-sfpplus16 comment="Peer Link Port 16"
# Configure the VLANs
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,pve-bond1,pve-bond2,pve-bond3,peer-bond vlan-ids=401-404
add bridge=bridge1 tagged=peer-bond vlan-ids=1
# Configure Inter-Switch Link Aggregation (Peer Bond)
/interface bonding add name=peer-bond slaves=sfp-sfpplus13,sfp-sfpplus14,sfp-sfpplus15,sfp-sfpplus16 mode=802.3ad lacp-rate=fast transmit-hash-policy=layer-2-and-3
# Add the peer-bond to the bridge with pvid=99 to isolate peer communication:
/interface bridge port add bridge=bridge1 interface=peer-bond pvid=99
# Configure MLAG Peer Port
/interface bridge mlag set bridge=bridge1 peer-port=peer-bond
# Configure PVE Ports as MLAG Bonds
/interface bonding
add name=pve-bond1 slaves=sfp-sfpplus1 mode=802.3ad lacp-rate=fast transmit-hash-policy=layer-2-and-3 mlag-id=1
add name=pve-bond2 slaves=sfp-sfpplus2 mode=802.3ad lacp-rate=fast transmit-hash-policy=layer-2-and-3 mlag-id=2
add name=pve-bond3 slaves=sfp-sfpplus3 mode=802.3ad lacp-rate=fast transmit-hash-policy=layer-2-and-3 mlag-id=3
# Add these bonds to the bridge:
/interface bridge port
add bridge=bridge1 interface=pve-bond1
add bridge=bridge1 interface=pve-bond2
add bridge=bridge1 interface=pve-bond3
# Finalize VLAN Configuration
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,pve-bond1,pve-bond2,pve-bond3,peer-bond vlan-ids=401-404
add bridge=bridge1 tagged=peer-bond vlan-ids=1
|