Before you start
This guide is general information about how we approach the design of a small data centre. It is not advice for your specific site, and it is not a substitute for someone who knows your workload, your constraints and your obligations.
Figures and configurations are illustrative. Sizing, redundancy and segmentation all depend on what you are running and what it is worth. MDCS accepts no liability for decisions taken on the basis of this guide.
01 · Separate the networks
The first decision, and the one most often skipped, is how many logically separate networks the estate has and which traffic belongs on each. Collapsing them onto one flat network works at four racks and becomes very difficult to unpick at twelve.
Four are worth separating from the start.
Two firewall tiers. The perimeter pair filters what arrives; the internal pair polices what the zones may say to each other. Storage, backup and management sit inside the dashed boundary with no route out.
| Network | Carries | Why separate |
|---|---|---|
| Production | Guest and application traffic, in and out of the estate. | It is the only one that should ever be reachable from outside. |
| Storage | iSCSI, NFS or equivalent between nodes and storage. | Latency-sensitive and bursty. Sharing it with production makes both unpredictable, and a backup window can stall a database. |
| Backup | Bulk transfer to the backup target. | Sustained high throughput at inconvenient times. It should not compete with production, and a compromised production network should not reach it. |
| Management | Hypervisor management, server BMC or iDRAC, switch and PDU administration. | This is the network that owns everything else. It should be reachable from the fewest places, and never from the internet. |
The mistake that is hardest to undo
Management interfaces placed on the production network. It is convenient during the build and it means anything that compromises a guest is one step from the hypervisor, the storage array and the power distribution. Separate it at the start, when it costs a VLAN and a few minutes.
02 · Switching layout
Two arrangements cover almost everything at this scale.
Top of rack
A switch pair in each rack, servers connect within the rack, switches uplink to a core pair. Cabling is short and contained, a rack can be worked on as a unit, and adding a rack does not disturb the others. This is the default for anything above about four racks.
Collapsed core
One switch pair for the whole room, everything connects to it. Simpler and cheaper below four racks. It becomes a cabling problem and a single maintenance domain as you grow.
What matters more than the topology
Two of everything in the path, with the pair configured so either can fail. A single core switch makes every redundancy decision elsewhere in the design pointless, and it is the most common false economy at this scale.
Uplinks
- Two uplinks from each access switch, to two different core switches. Not two to the same one.
- Aggregate them, so both are carrying traffic rather than one sitting idle waiting for a failure that reveals it was never configured.
- Size for the traffic that actually crosses racks. East-west between nodes is usually larger than people expect once storage or clustering is in play.
03 · Firewalls and segmentation
One firewall at the boundary is the common arrangement in a small estate, and it leaves everything inside implicitly trusted. Two tiers is the better practice, and it is what we run ourselves.
| Tier | Sits between | Decides |
|---|---|---|
| Perimeter | The internet and everything else. | What is allowed to arrive at all. Address translation, published services, intrusion prevention, reputation and geography filtering. |
| Internal | Zones, and anything that reached the DMZ. | What one part of the estate may say to another. Zone to zone, application tier to database tier, user network to server network. |
The difference is not that the second firewall is better. It is that the DMZ is treated as already breached, and something has to be crossed to leave it.
What the second tier actually buys you
A compromised public-facing service in the DMZ has reached the internet-facing tier and nothing more. It still has to pass a separate device, with a separate ruleset and separate credentials, to reach anything internal — and that traffic is inspected and logged rather than assumed.
With one tier, everything past the boundary is on the same footing. The web server that was breached sits on the same network as the database it was never supposed to reach directly.
Mapping the internal tier to the platform
A word on terminology first, because it is a common source of confusion. In Apache CloudStack a zone is the second largest organisational unit and typically corresponds to an entire datacentre. Below it sit pods — usually a rack or a row — then clusters of homogeneous hosts, then the hosts themselves. A room of four to forty racks is normally one zone containing one or more pods, not several zones.
So what the internal firewall polices is not zone to zone. It is traffic between guest networks, or between the tiers of a VPC — web, application, database — which is where segmentation actually matters at this scale.
- One interface or VLAN per segment on the internal pair, so traffic between tiers is routed through policy rather than switched around it.
- Default deny between tiers. Traffic that has to cross is named, justified and recorded. If nobody can say why a rule exists, it is a finding.
- Management is a segment too, and the most restricted one. The internal tier is what stops a compromised guest reaching a hypervisor or the management server.
- The platform’s own controls still apply. CloudStack uses security groups for layer-3 isolation on shared networking, and network ACLs on the VPC virtual router to control traffic between tiers and outward. Those operate inside the platform; the firewall operates between the platform and everything else. Use both, not either.
Know the default before you write rules
On a VPC tier, CloudStack blocks incoming traffic and allows outgoing by default. The moment you add a single egress rule, everything not named in it is blocked. A tier that worked yesterday can stop working because somebody added one outbound rule intending to permit something, not to restrict everything else.
Where each control belongs
Security groups and network ACLs — guest to guest, inside the platform, enforced by the platform. Internal firewall — between guest networks and the rest of the estate, and between anything in the DMZ and anything internal. Perimeter firewall — the internet and everything else.
The cost of the second tier, stated plainly
Two more devices, two more rulesets and another failover path to test. Traffic crossing zones takes an extra hop. It is more to buy and more to operate, and below about four racks with a single workload it is often more than the risk warrants. Above that, and anywhere holding regulated or client data, it is the difference between a contained incident and an estate-wide one.
Common to both tiers
- A pair, in high availability, at each tier. A single unit at either makes that tier a scheduled outage every time it is patched.
- Fail closed, and know what that means. Establish what happens to traffic mid-failover, and test it during commissioning rather than discovering it later.
- Management access to the firewalls comes from the management network, never from the internet and never from a guest.
- Separate credentials per tier. Two firewalls administered with the same account are one firewall with extra cabling.
04 · Node connectivity
How many network interfaces a node needs is a design decision, not a hardware default. Work it backwards from the networks you separated in section one.
| Purpose | Interfaces | Notes |
|---|---|---|
| Production | 2, bonded | To two different switches, so a switch failure or a switch upgrade is not an outage. |
| Storage | 2, bonded or multipath | Multipath rather than bonding where the storage protocol supports it — it fails over faster and uses both paths properly. |
| Backup | 1, or shared with storage | Separate where backup volume is large enough to disturb storage latency. |
| Management | 1 | Hypervisor management. |
| BMC / iDRAC / iLO | 1, dedicated | Out-of-band. Never shared with the hypervisor management interface. |
Six ports is a common answer for a node in a small estate. Four is workable where backup shares the storage path. Two is a decision to accept that a switch maintenance window is a service window.
05 · Laying out a rack
Where equipment sits in a rack is a design decision with consequences for weight, cable length, airflow and how safely someone can work on it at three in the morning. The defaults — fit it where it fits — produce racks that are top-heavy, badly cabled and awkward to service.
Illustrative. The principle that matters is that all racks in the room follow one layout, so anyone can find anything in any of them.
| Position | What goes there | Why |
|---|---|---|
| Top — 40U to 42U | ToR switch pair. | Cable runs to every server below are short and roughly equal, and overhead containment enters at the top. Two switches, not one. |
| Upper — 36U to 39U | Patch panels and cable management. | Directly under the switches, so jumpers never leave the rack. Cable management between panels rather than all of it at the bottom. |
| Middle — 12U to 35U | Servers. | Working height. Rails can be extended and a chassis handled without lifting above shoulder or below knee. |
| Lower — 5U to 11U | Storage. | Heavy, and rarely needs physical access once installed. Low enough to help stability, high enough to service. |
| Base — 1U to 4U | UPS and batteries. | The heaviest items in the rack. At the base they lower the centre of gravity; at the top they make the rack dangerous to move and hard to stabilise. |
The three that get skipped
Blanking plates in every unused U. Without them, hot exhaust air loops around to the equipment intakes and the hot and cold aisle arrangement stops working. They cost almost nothing and are the most commonly omitted item in a small room.
Airflow direction on switches. Network equipment is sold in both front-to-back and back-to-front variants. One switch drawing air the wrong way pushes hot exhaust into the cold aisle for the whole row. Check the part number, not the picture.
Weight, before the rack is populated. A full rack is commonly in the region of half a tonne to a tonne depending on what is in it. That is a question for the structural engineer, and it belongs in the survey rather than in the delivery week.
Lay every rack out the same way
Uniformity is worth more than optimising each rack individually. When every rack in the room has switching in the same U, patching in the same U and servers in the same band, anyone can find anything in any rack — including someone who has never been in the room and is being talked through it over the telephone.
06 · Storage paths
Storage is where most performance problems in a small data centre originate, and the path matters as much as the array.
- Two paths from every node to every storage target, through two switches, with multipathing configured and — this is the part that gets missed — tested by pulling a cable during commissioning.
- Jumbo frames end to end or not at all. Set on the node, the switch and the array. Configured on two of the three, it produces intermittent problems that are very hard to diagnose.
- The storage network does not route. It is a flat, dedicated segment with no gateway. If it needs a route, something is wrong with the design.
- Size the path, not just the array. A fast array behind a single 1G link is a slow array.
Establish the failure behaviour before you rely on it
What happens to a running guest when one storage path drops. How long the pause is, whether the filesystem goes read-only, whether it recovers on its own. That is a ten-minute test at commissioning and an expensive discovery in production.
07 · Backup
Backup is a design element, not an afterthought, and at this scale it is usually the difference between an incident and a closure.
Three copies, two media, one off site
The old rule still holds. What has changed is the addition of immutability — at least one copy that cannot be deleted or altered by any credential in the estate, because ransomware and account compromise both look for backups first.
- The backup target is not reachable from production. Reverse the direction if you can: the backup system pulls, rather than production pushing.
- Separate credentials. A domain administrator should not be able to delete backups. If the same compromise takes both, you have one copy.
- Retention is a decision. Daily for a fortnight, weekly for a couple of months, monthly beyond. Short windows fail when a compromise is discovered weeks later and every retained copy is already affected.
- Restore, and time it. The measured restore time is your real recovery window, and it is almost always longer than the estimate.
08 · Out-of-band access
The question worth answering during design: when the network is down, how do you reach the equipment to fix it?
- BMC on every node, on the management network, with its own credentials.
- Console access to switches and firewalls, either through a serial console server or physically documented so somebody can find the port at 2am.
- Switched PDUs, so a hung device can be power-cycled without a site visit. This is the single most useful thing in a room you do not sit next to.
- A path in that does not depend on the production firewall. Whether that is a VPN on a separate connection or a documented physical procedure, decide it now rather than during an incident.
Naming and documentation
Unglamorous, and it determines whether the estate is operable by anyone other than the person who built it.
- A naming convention decided before the first device is configured. Location, role, number. Applied to switches, servers, storage, VLANs and interfaces alike.
- An address plan written down, with each network’s range, its purpose and its gateway, before addresses are assigned.
- The connection schedule kept current. Device, port, destination, port. It is written during the build and becomes worthless within a year unless changes are recorded.
- Labels at both ends of every cable. The cheapest thing in the room and the one most often skipped.
The test for whether documentation is adequate
Somebody who has never seen the room can identify what a cable does without unplugging it. If that is not true, the documentation is decoration.
Decide before you buy
These are the questions worth answering on paper, because each is expensive to revisit once hardware is on site.
| Question | What it determines |
|---|---|
| Basic or Advanced networking for the zone? | Nearly everything. A CloudStack zone is one or the other for its entire lifetime and cannot be converted — changing your mind means building a new zone and migrating onto it. Advanced gives VLAN-isolated networks and VPCs; Basic gives a flat network with security groups. |
| How many separate networks, and physical or VLAN? | Switch count, port count, node interface count. |
| Top of rack, or collapsed core? | Switch purchase, cabling design, growth path. |
| How many ports per node? | NIC specification, and whether that is fixed at purchase. |
| Which storage protocol? | Whether multipath or bonding, and whether jumbo frames are in play. |
| Where does backup live, and who can delete it? | Target placement, credential separation, immutability. |
| How is the estate reached when the network is down? | Console server, switched PDUs, out-of-band path. |
| What is the growth path to double the size? | Whether that is an addition or a rebuild. |
The last one is the one most often skipped. A design that works at eight racks and cannot become sixteen without recabling is a design with a deadline on it.