Before you start
Before you run anything
This guide is general information about how we approach a CloudStack audit. It is not advice for your specific environment. Every command is intended to be read-only and we have written them that way deliberately, but you are running them on your own systems at your own discretion — read each one and understand it before you do. Thresholds and worked figures are illustrative. MDCS accepts no liability for decisions taken on the basis of this guide.
Two rules before you paste anything into a chat window
Use a read-only account. This is an inspection, not a change. Create a restricted role with list permissions rather than using an admin key — the same principle we would flag in an access review of your estate.
Sanitise the output. API responses carry hostnames, public IP ranges, capacity figures and topology. Strip them, or export redacted, before pasting into any third-party service. Treat that output the way you would treat a network diagram.
You will need the management server URL, an API key and secret for a read-only account, and a way to call the API.
Use cmk, not the old cloudmonkey
The current CloudMonkey is cmk — a Go rewrite, shipped as a single binary, compatible with CloudStack 4.9 and above. The original Python cloudmonkey is no longer maintained. Commands below use cmk. Both the noun form (list hosts) and the direct API form (listHostsMetrics) work. Install and verification steps are in the next section.
Create the read-only account with a restricted role — CloudStack supports custom roles through createRole and role permissions, so a list-only role is a few minutes of work and it is the right way to run an inspection.
On the AI assistant
The prompts below were written against Claude, so that is what the worked examples show. The method is not tied to it — any model capable of reading structured JSON and reasoning about it will do. Substitute freely.
Install and verify cmk
Five minutes, and worth doing properly. You are about to point this tool at production infrastructure, so treat the download the way you would treat any other binary entering your estate.
1 — Import the Apache signing keys
2 — Verify the release you downloaded
sha512sum should print OK. A "Good signature" warning that the key is not certified with a trusted signature is normal — it means you have not personally signed the release manager's key. A BAD signature, or a failed checksum, is not normal. Stop there.
CloudMonkey's current release is 6.5.0. Take KEYS and the signature files from the Apache distribution directory rather than from a mirror.
Know what you are actually downloading
Apache publishes the source release with PGP and SHA512 signatures. The ready-to-run binaries on the GitHub releases page are published by community members as a convenience — they are not part of the signed Apache release.
For most teams the GitHub binary is fine. If your organisation requires verified provenance for anything running against production, build from the signed source instead. Decide which applies to you before you download, not after.
3 — Install the binary
cmk -v should print the CloudMonkey version and build hash. If the shell cannot find cmk, /usr/local/bin is not on your PATH.
4 — Configure a profile
Config lives at ~/.cmk/config in plain text, including the secret key. Keep it 0600, keep it off shared machines, and remove the profile when the audit is finished. Leave verifycert true — turning it off to get past a certificate warning is how you end up trusting the wrong endpoint.
5 — Prove the connection works
6 — Prove the account is actually read-only
This should return nothing. If create, delete or destroy APIs appear, the role is not restricted the way you think it is — fix the role before continuing. Verifying the restriction rather than assuming it is the same discipline as re-checking a finding rather than trusting that the fix worked.
The method
Check
Run the API call. Paste the output with the prompt. The assistant identifies what is wrong.
Improve
Work through each issue with the remediation guidance, and ask follow-up questions where the fix is not obvious.
Re-check
Re-run the same call. The re-check must return empty. If it does not, the issue reopens.
That last phase is the one that proves anything. An issue is not closed because somebody applied a fix — it is closed because the query that found it no longer returns it.
The metrics APIs do half the work for you
Alongside the plain list calls, CloudStack exposes listHostsMetrics, listStoragePoolsMetrics, listVolumesMetrics, listClustersMetrics, listZonesMetrics and listInfrastructure. These return computed utilisation rather than raw records, so the analysis has less to infer and you have less to paste. Start with listInfrastructure for a one-call overview of the estate before working through the areas.
Set the context once
Paste this at the start of the conversation, before any API output. It saves repeating the environment details on every area, and it stops the assistant inventing severity levels that mean nothing to you.
Why point 5 matters
Without it you will get confident answers about things the API response never contained. Asking for the gap to be named is the difference between an analysis you can act on and one you have to verify by hand.
Zones, pods & clusters
What to check
All zones enabled, pods holding valid non-overlapping IP ranges, clusters connected and out of error state.
Run
Prompt
Improvements
- Enable disabled zones and pods, with the reason documented
- Correct overlapping or missing pod IP ranges
- Remove stale cluster entries that no longer hold hosts
Re-check prompt
Re-check re-run list zones and confirm allocationstate reads Enabled on every zone
Note: listZones has no allocationstate request parameter — allocationstate is a response field only. Filter the response, do not try to filter the query.
Hosts & hypervisors
What to check
All hosts up, hypervisor versions consistent within each cluster, and no host carrying more load than it should.
Run
Prompt
Improvements
- Reconnect or rebuild hosts in Down or Alert state
- Bring hypervisor versions into line within each cluster
- Migrate VMs off overloaded hosts; use maintenance mode rather than pulling them
Re-check prompt
Re-check list hosts state=Down → must return empty, and no host left in a Maintenance or Error resourcestate
Use memorywithoverprovisioning rather than memorytotal — memorytotal is deprecated in current releases.
Storage pools
What to check
Primary and secondary pools up, free capacity healthy, over-provisioning within a ratio you can actually honour, and no orphaned volumes.
Run
Prompt
Improvements
- Add capacity to pools approaching the threshold, before they cross it
- Clean up orphaned volumes and stale snapshots
- Review the over-provisioning ratio in global settings against real growth
Re-check prompt
Re-check diskSizeAvailable > 20% of diskSizeTotal on every pool
Virtual machines
What to check
Inventory by state, VMs stuck in Error or Unknown, machines with no network interface, and long-idle VMs nobody owns.
Run
Prompt
Improvements
- Expunge Error and Unknown VMs — after the owner confirms, not before
- Attach missing network interfaces to orphaned VMs
- Put a lifecycle policy in place so this list does not rebuild itself
Re-check prompt
Re-check list virtualmachines state=Error listall=true → must return empty
Without listall=true you see only your own account's instances, which is the most common reason an audit misses machines entirely.
Networks, VLANs & IP pools
What to check
Guest networks implemented, public IP pools not close to exhaustion, VLANs allocated and actually in use.
Run
Prompt
Improvements
- Restart or recreate guest networks stuck outside Implemented
- Expand IP pools before exhaustion, not after the first failed deployment
- Release unused public IPs back to the pool
Re-check prompt
Re-check every active network reports state=Implemented
System VMs — SSVM & CPVM
What to check
Secondary Storage VM and Console Proxy VM running in every zone. These fail quietly and take console access and template operations with them.
Run
Prompt
Improvements
- Restart failed system VMs from the management console
- Verify connectivity between management server and the system VM network
- Check NTP sync — time skew is the most common cause of system VM failure
Re-check prompt
Re-check every system VM reports state=Running
Capacity & global settings
What to check
Headroom across the estate, over-commitment ratios, HA enabled where it matters, and global settings that have drifted from sensible production values.
Run
Prompt
Improvements
- Add compute where CPU or RAM sits consistently above 75%
- Tune timeout values to your actual network latency rather than the defaults
- Enable HA on every production cluster
Re-check prompt
Re-check all capacity types below 75% used
What a healthy environment looks like
Two different kinds of benchmark, and it is worth being clear about which is which. The first set is binary — there is no environment where a different answer is correct. The second set is a convention. Treat those as starting points and tune them to your growth rate and your procurement lead time.
Non-negotiable — these should be true everywhere
| Check | Healthy state | If not |
|---|---|---|
| Zones & pods | All Enabled | Capacity you own but cannot allocate |
| Hosts | All Up | A host in Alert is not carrying load, and HA cannot use it |
| System VMs | SSVM and CPVM Running, every zone | Console access and template operations fail silently |
| Virtual machines | None in Error or Unknown | Resources held by machines nobody can start |
| Networks | All active networks Implemented | Deployments into that network will fail |
| Storage pools | All Up | Volumes on that pool are at risk, not just unavailable |
| Hypervisor versions | Consistent within each cluster | Live migration fails, and behaviour differs host to host |
| NTP | In sync across management and hosts | The most common single cause of system VM failure |
| HA | Enabled on production clusters | A host failure becomes a manual recovery |
Conventions — sensible defaults, not rules
| Measure | Common target | Why that number, and when to move it |
|---|---|---|
| Primary storage free | Above 20% | Leaves room for snapshots and thin-provision growth between procurement cycles. Raise it if your growth is fast or lead times are long; a pool that fills stops writes. |
| Secondary storage free | Above 25% | Templates, ISOs and snapshot copies land here in bursts rather than steadily. |
| CPU allocated | Below 80% per host | Headroom to absorb a host failure without the remaining hosts refusing placement. |
| RAM allocated | Below 85% per host | RAM is the constraint that actually blocks deployment, sooner than CPU in most estates. |
| Cluster capacity | Below 75% used | The point at which adding hardware should already be in a budget, not a discussion. |
| CPU over-provisioning | 2:1 to 4:1 | Depends entirely on workload. Development estates take far more; latency-sensitive production takes less. |
| RAM over-provisioning | 1:1 | Over-committing memory is where over-provisioning stops being free. |
| Public IP pool | Below 90% used | Exhaustion shows up as a failed deployment, usually at the worst moment. |
| Stale VMs | Reviewed at 180 days | Not a fault — a prompt to ask whether anyone still owns it. |
Use these as a starting position
If a number in the second table does not fit your environment, change it — and write down why. A documented deviation is a decision. An undocumented one is something nobody can explain in six months when it causes a problem.
Prompt reference card
Four prompts that sit outside the seven areas. Bookmark these — they are the ones you will reach for repeatedly.
Full environment sweep
Turn findings into commands
Capacity forecast
Confirm resolution
Read the output, do not just run it
Generated commands are a starting point, not an instruction. Check each one against your own environment before running it — particularly anything that expunges, destroys or restarts. Nothing in this guide should be executed on production without you understanding what it does.
Resources worth having open
Everything below is from the Apache project itself unless marked otherwise. If you are standing CloudStack up as your infrastructure layer, these are the pages to bookmark on day one rather than hunt for at 2am.
Releases
| What | Where |
|---|---|
| Downloads — source, signatures, current versions | cloudstack.apache.org/downloads |
| Signing keys | dlcdn.apache.org/cloudstack/KEYS |
| All releases, with notes and CVE advisories per version | GitHub releases |
| Older versions | Archives |
| LTS policy | LTS page on the project wiki |
Current at the time of writing
4.22.1.0 is the current LTS release; 4.22.0.1 is the preceding maintenance release on that branch. LTS branches are supported for 24 months — full updates for the first 18, then blocker defects and security fixes only for the final 6.
We deliberately do not deep-link individual tarballs here. Those URLs change with every release and a stale download link is worse than no link. Take the source, signature and checksum from the downloads page, which is always current.
Packages — community maintained, not part of the signed Apache release
| Platform | Repository |
|---|---|
| Ubuntu / Debian | download.cloudstack.org/ubuntu |
| EL10 · EL9 · EL8 | el/10 · el/9 · el/8 |
| SUSE / openSUSE 15 | download.cloudstack.org/suse/15 |
| ARM64 — experimental | download.cloudstack.org/arm64 |
| EL7 — for 4.19 and older only | download.cloudstack.org/el/7 |
| System VM templates | download.cloudstack.org/systemvm |
| Kubernetes Service ISOs | download.cloudstack.org/cks |
| Third-party packages | ShapeBlue (noredist) |
Documentation
| What | Where |
|---|---|
| Documentation home — pick your version here | docs.cloudstack.apache.org |
| Installation guide | 4.22.1.0 install guide |
| Upgrade guide — read this before every upgrade | 4.22.1.0 upgrading |
| Administration guide | 4.22.1.0 admin guide |
| Release notes | 4.22.1.0 release notes |
| API reference | cloudstack.apache.org/api · 4.22 root admin |
| Project wiki | CloudStack wiki |
Documentation paths follow the pattern docs.cloudstack.apache.org/en/<version>/<section>/ — swap the version to match the release you are actually running.
Read the upgrade section of the release notes for the version you are moving to, and every version you are skipping. Failed upgrades usually trace to skipped steps, not broken software.
Tooling and automation
| Tool | Where |
|---|---|
| CloudMonkey CLI — 6.5.0 | Signed source · Binaries · Wiki |
| Terraform provider — v0.6.0 | Terraform registry · Source |
| Go SDK | cloudstack-go releases |
| Kubernetes provider — v1.2.0 | Source · Docker image |
| Cluster API provider | CAPC documentation |
Security and community
| What | Where |
|---|---|
| Security advisories and how to report | cloudstack.apache.org/security |
| Verifying downloads — the full explanation | apache.org/info/verification |
| Mailing lists — users@ and dev@ | Mailing lists |
| Issue tracker | GitHub issues |
Subscribe to the users list before you need it
The mailing lists are where release announcements and security advisories land first. Subscribing takes a minute and is the cheapest early warning available for a platform you are about to run in production.
Tracking & sign-off
Record what you found, what you did, and whether the re-check passed. A row is not closed until the re-check returns empty — and a failed re-check reopens it rather than quietly staying shut.
| # | Area | Issue found | Action taken | Re-check | Status |
|---|---|---|---|---|---|
| 1 | Hosts | host-04 in Alert state | Reconnected, agent restarted | Passed | Closed |
| 2 | Storage | Pool primary-02 at 12% free | Snapshots pruned | Failed — 17% | Reopened |
| 3 | System VMs | SSVM zone-2 not running | NTP corrected, VM restarted | Passed | Closed |
Audit cadence
| Frequency | Scope |
|---|---|
| Daily | System VMs, host state, active alerts |
| Weekly | VM states, storage capacity, IP pool utilisation |
| Monthly | Full seven-area audit, tracking table updated |
| Quarterly | Capacity planning, global settings, HA policy review |