The Ever-Versatile vCenter Orchestrator

Solving a Simple Backup Problem 

vCenter Orchestrator is a bit of a dark horse in the VMware product portfolio.  Almost every customer has it, because it is licensed along with every vCenter Server, but almost every customer has never touched it.  It is actually a VERY powerful tool to have in your toolbox.
Of course, I’m a strong opponent against using orchestration-centric approaches to building a self-service and/or cloud environment.  The two-part problem is that

1. Orchestration designs, “great” as they might be, need to touch many pieces of technology in the datacenter.  While no single element of integration may be a particular challenge, this up-front implementation of many moving parts is costly and time-consuming – and such solutions will typically take 6 months or more, and cost several hundreds of thousands of dollars in services alone.  
2. Orchestration designs are hypersensitive to any technology changes over time. That is, you are likely to break the intricate machine whenever you perform a software upgrade, firmware upgrade or hardware model change. That is usually guaranteed to happen yearly (or half-yearly) for software, and every few years for hardware. Multiplied by 10-15 moving parts, or more, that means the solution is never actually stable for any length of time – or else it holds the business back from making necessary changes.

Well, having said all that, orchestration has its place. If it was on the food pyramid, it might be “fats and oils”. Rich in energy, necessary as part of a complete diet, but you have to take it easy or else it’ll lead to heart attack, or perhaps an inability to leave your front door!  OK, so it’s not the best analogy… but hopefully the point sticks.

Last year, a customer was seeking a way to achieve a simple backup method, to safeguard their remote offices from VMs breaking through software changes, updates or “tinkering”.  Each site had an ESXi standalone host managed by a central vCenter Server, local storage, and fair to poor WAN links.  We decided to explore using vCenter Orchestrator to create application consistent, on-site, self-managing backups.  What we ended up with looked pretty useful, so I thought I would share it here.   It also only took a few days for us to put together, Peter Marfatia and I, which I thought was pretty reasonable for a team with limited skills in the tool.

I have included a link to the resulting package at the bottom of this article, and also the automatically generated documentation that vCenter Orchestrator provided for me.

To get started with vCenter Orchestrator, there are some great resources out there – some I’ve included at the bottom of this article.  It is a great learning experience to install the Orchestrator Appliance and Client, and just look around at the various actions, workflows and tools that it makes available.

The Overall Backup Process

Perhaps to start with, a view of the overall process we used for this Branch Backup would help. 

1. The workflow is pointed to a folder within vCenter
2. It discovers all virtual machines within that folder, and determines whether they are candidates for backup, or are instances of prior backups.
3. If requiring backup, it performs a snapshot with quiescing, then clones the snapshot to a new VM, which is converted to a template (to prevent accidental power-ons).
4. If looking at prior backups, removes those no longer needed.
5. When all backups have been processed, the workflow emails a report to a nominated address.

Backup Dispatcher

This is the main entry point into the whole workflow. 

You can see that in vCenter Orchestrator, there is a visual layout of the workflow steps, like all other orchestration tools.  Even if this is the first time you’ve seen orchestrator, you can look at the diagrams and have a fair understanding of what is happening when the workflow is run.

When running the job manually, the workflow asks a small number of questions as shown below.  This is the default user interface presented by the Orchestrator Client, and others are available – check out the VMware Labs site for some options.

Invoking the “Backup Dispatcher” workflow, this interface is asking for the following elements.

• Email address to send the job report – listing the VMs backed up and the success/failure results.
• Number of backups to retain – on a per VM basis.  This could be made a global property, but we had fun playing around with different levels here.  I wrote the retention logic to allow for changes to retention, so that during periods of greater change, more backups could be safely kept, and then scaled down later.
• Folder containing the VMs to be backed up – the workflow would collect ALL VMs from the selected folder.  The management of what to backup is then a simple drag’n’drop of any VMs into or out of this folder.

We provide a few things in the static properties, such as mail server and content settings, but most other things are dynamic.  The static properties are present as “Workflow Attributes” – these are essentially working (read/write) variables that don’t act as workflow input (read only) or output (write only).  Before running the package, you will need to have your vCenter Server registered in vCenter Orchestrator, so that it appears within the vCO inventory and enables communication between the two.

Below is a screenshot of the inventory, as viewed from the vCO Client.  This is one of the excellent aspects of vCO – I can pre-configure what things are present in my environment and not have to deal with connection strings, user credentials, and various properties embedded in scripts.  It is done just once, and then the workflows can talk to your datacenter!  If you can’t see your vCenter inventory like this in vCenter Orchestrator, the workflow won’t be able to do much with your environment!

In my case, I have easily connected up:

• 2 x vCenter servers
• UCS Manager
• Active Directory
• vCloud Director
• vCenter Chargeback
• A mail server

When clicking on the hyperlink to point the workflow to a specific folder (initially has the value “Not set”), the workflow interface presents a view of the vCenter inventory (as seen by vCO) for you to choose from, as in the screenshot below.  Again, this is simpler for the workflow user, because I had already made vCenter available to vCO using a service account (although I could have forced a per-user connection if I wanted).

When scheduling the workflow to run on an automatic daily cycle, you can set this parameter during the scheduling process, in the same way as shown here.  In the case of our customer, they wanted to schedule a collection of these backup jobs daily, each pointing to slightly different sets of VMs to backup.  In the vCO Scheduler, each job entry was provided with the distinct folders that it would manage, and the jobs just ran thereafter without any real caretaking.

The first action of the Backup Dispatcher is to “Get All Virtual Machines By Folder Including Sub Folders”.  This is pretty self-explanatory, and was an action already available through the vSphere Plugin shipped with vCO.  Conveniently, this requires only the input folder, and returns an array containing all VMs found.

The next action is to “Sort VMs By Name”.  I implemented this as a subordinate workflow, while I was toying around with various ways to solve a key problem – which was how to determine whether the current backup is to be retained or not.  I wanted to ensure the workflow didn’t use any hard dates – as it doesn’t know if it is being run weekly or daily, or ad hoc, and a whole bunch of other “if’s” and “maybe’s” that came up while I was thinking about it.  Due to the limited amount of time I wanted to spend on it, and my rudimentary skills, I decided to name VM backups according to a certain naming pattern, which contains:

• Original VM name
• A known delimiter, which hopefully won’t pop-up in a normal VM name.  I chose a colon “:” in my example, after checking with the customer that they wouldn’t expect a problem.
• The keyword “BACKUP”
• Another delimiter “:”
• A date/time stamp, in the format yyyyMMddhhmm, such as 201306051609 – which is what my clock says as I write this.

This “Sort VMs By Name” action merely contains a Javascript function, made with a little help from web searching, which sorts VMs according to original name first, and then from most recent to least recent backup.  This helps later on, because the retention policy can skip over the ones to be retained, and delete any subsequent backups older than these.  You’ll see that later in the “Process backup and retention logic” workflow.  Anyway, the point being that this particular point in my effort probably took the longest amount of time as I struggled to remember anything at all about writing some script.  It just goes to show how little I needed to know for the rest of the effort!

The next stage of the main workflow is essentially a “for each” loop.  I implemented it as an explicit loop, just because the easy vCO “ForEach” logic control worked a little differently than I wanted it to here – but I could probably tackle this again in a better way.  For each of the VMs in the (now sorted) array, I submit them each to the “Process backup and retention logic” subordinate workflow.  This subordinate workflow will determine whether or not to backup the VM at hand, and if so, will return an identifier for the backup activity.

Once all VMs have been “processed”, the main workflow then waits for any backup activities that are being performed, using the workflow identifiers kept from each job submission, and then sends a report.  The main framework being used here was derived from some previous examples created by Joerg Lew. (Well, I think it was Joerg!)

OK, so that’s the main flow, but the cool bit is doing the live backup from a quiesced snapshot, so let’s look at that - in a minute.  First, we need to figure out what needs backing up.

Process Backup and Retention Logic

In hindsight, this is a crappy name, but this subordinate workflow is being called for each and every VM that was discovered, and is trying to determine whether this is a ‘real’ virtual machine needing backup, or if it’s a backup that might need to be removed.  This part of the workflow was where I did most of the thinking, trying different approaches that I could make work using my rudimentary skills.

You can also see the passing of inputs and outputs for this workflow, which is an awesome visualisation of where information is flowing.  It's also very easy to just click'n'drag this info around, as you're building the workflow.

Firstly, the workflow tries to separate the discovered VM name into the three separate elements, according to the “OriginalVMname:BACKUP:201306051409” type of format.  If this is not actually a backup, the last two elements will just be empty, of course.  If we find that the current VM is a new name, then any counters – which were being used to count the number of old backups – need to be reset to zero.  Then, if the VM has the special name “BACKUP” in it, then the workflow only needs to determine whether to keep it. 

These decisions were based on a couple of simple bits of Javascript logic – but you may notice that all the decisions are being made with vCO logic elements.  This is also another easy part of vCenter Orchestrator – you drop in an “IF” logic box, give it an input to determine a “true” or “false” choice, and then you simply drag a connection for each choice to the next part of the workflow.  Too easy.

The only smart bits I used in this whole workflow were:

• A few Javascript bits of logic, which I could likely replace with more readable vCO logic elements
• A call to a vCenter action to “Delete Virtual Machine”, if the workflow has found an old backup requiring deletion.
• A function call to submit a new vCO workflow for any VMs found that need backing up.  This is done in the “Backup This VM” script action, and this is what returns the identifier for the running backup job that is tracked later on.  The workflow that we actually call is “Clone VM For Backup”, which is described next.

Clone VM For Backup

This is a pretty straightforward bit of work, which anybody could put together on their Day 2 exploration of vCenter Orchestrator.  It simply takes a VM as an input, and calls the workflows already available in the vSphere Plugin.

There is one element that is a little ‘special’ here, which is the “Clone From Snapshot” workflow.  After creating a snapshot, which is passed the “Quiesce=True” parameter, the next piece is to clone the still-running VM from that snapshot.  This workflow was taken from Joerg Lew from his years-old blog on this topic.  This is a native capability of vSphere, and the vSphere API, but just isn’t readily exposed through other means such as the vSphere Client. 

This workflow is also passed the parameter to make the new clone into a template, which helps avoid accidental power-on operations.  The new template is named according to the “OriginalVMname:BACKUP:yyyMMddhhmm” format mentioned earlier.

The quiescing behaviour called during the snapshot action is the native vSphere capability to invoke VSS for Windows machines, or look for scripting stubs in Linux machines.  It is then up to an application owner to determine if any special actions might be needed to ensure application consistency.  Whatever the owner decides, the workflow doesn’t need to worry about it.

The other “cool” thing I decided would make sense is to “Remove All Snapshots” once the clone has finished.  This was a clear decision that I thought would have the added benefit of ensuring snapshots disappeared on a regular basis.  I have fielded enough urgent calls from customers who have killed their environment because of snapshots filling up datastores. If this was deemed undesirable, however, the workflow could be modified to only remove the snapshot that was created in the prior step, using the available “Remove Snapshot” workflow instead.  The risk here is that something unexpected might prevent this clean up from happening one day, and the snapshot would be effectively “forgotten”.  Hence, my decision to remove all snapshots provides a nice safeguard.

Results

At the end of the process, I can look at the results of running this workflow in the environment.  Below is a view of the job mid-flight in a demo environment.

You can just see that the “Moodle” VM is still being cloned and hasn’t yet been turned into a template.  You can see, however, the other templates from earlier backups having finished – in the current schedule, and also an earlier call.

I put in a certain amount of logging in the vCO workflows, writing to the Server.log(“__“) very handy function call, such as the example below.

            Server.log("Submitting backup for: " + vm.name);

I have included the logging output below, to give you an idea of what this creates.

This solution took only a few days of playing, experimenting and learning.  A lot of the vCenter Orchestrator functionality is self-evident, or if not, it is very comprehensively documented.  The customer was very pleased with this simple approach to solving a simple problem, and we trod a fine line between simplicity and complexity, to ensure the customer could easily understand the results and own it without too much hassle.

I am certainly not suggesting that this is a great backup strategy for your organisation, and that isn’t really the point of sharing it here.  I have used this example of a quick and cheap solution to illuminate one way we have used vCenter Orchestrator.  There are many other use cases that I’m sure you will find, once you discover how excellent this tool is, that you probably already possess.

As I pointed out at the start of this article, technical architects can get carried away with orchestration, and many organisations build very complex systems using this approach.  The temptation is certainly there.  However, it is very sensitive to change.  The simple example here might be robust enough, because it is only talking to one element – vSphere.  But this would quickly become an unmanageable beast if we connected to a server platform, a storage platform, a network manager and a firewall system – just for example.  Each element either becomes frozen in time, or else creates a risk of breaking the orchestration workflows.

The abstraction delivered by virtualization solutions such as vCloud Director, and vCenter itself, introduces standardised, software-based interfaces to the datacenter.  Actions can then be controlled through these software interfaces by the tools’ native functions and policies.  This is the true value of the broader Software Defined Datacenter architecture.  For the large, complex enterprise, orchestration is still useful and necessary “glue” from time to time, and vCenter Orchestrator is a very powerful and friendly tool in this capacity.

Further Areas For Expansion

Thanks to Joerg Lew and Peter Marfatia for their contributions to putting this little solution together.  I also greatly appreciate the community leadership provided by Christophe Decanini and Burke Azbill, who contributed plenty of knowledge and examples on the web for me to follow.

The example given here is a quick run at a solution, and certainly has plenty of opportunities for improvement.  With additional time, I would probably replace some Javascript functions with vCenter Orchestrator logic elements, which would make the workflow easier to understand visually, and make the self-documentation more complete.  I would also re-visit the explicit loop I have used here, and find an elegant way to make use of the “ForEach” construct instead.

Resources

There are a bunch of resources that I have used over time, and that really help with getting an introduction to vCenter Orchestrator.  A couple of them are listed below, to start you on your way.

• http://www.vcoteam.info/ - First and foremost, this site is fantastic, and maintained by very enthusiastic Orchestrator experts.  Start here.
• http://www.vcoportal.de/2011/08/clone-from-old-snapshot/ - This is the snapshot workflow package I leveraged to build this solution.  Thanks Joerg!
• https://blogs.vmware.com/orchestrator/ - The VMware Blog site is full of interesting news and posts, so worth a good trawling every now and then.

I have also uploaded my vCenter Orchestrator package and documentation at the links below.  Please feel free to use and abuse - and if you make it bugger and better, please share!

• vCenter Orchestrator - Simple Snapshot Backup package
• Auto-generated documentation.

Thanks for reading!

Showing Back IT Costs to the Business - A Simple Cost Model for IaaS

Introduction

Let’s get it clear up front – I’ve only ever found two customers who actually have the ability (or interest) to “chargeback” IT costs to their business groups.  But, speaking in my role as a VMware techo, almost all customers are very interested to know how they can communicate IT costs to the rest of the business.  The problem is that this is not really something tackled by most IT groups, and it can be difficult to start, so I thought I’d share some thoughts, which have been helpful in my discussions so far.

The two questions most organisations seem to want answered is: "Where did the money go?"  And: "Who’s using all our stuff?"  This becomes so much more of a problem when establishing a shared virtual platform, and especially when talking about the path to a Private Cloud (or self-service) shared capability.

Perhaps you spent $100k on extra storage, and now it’s almost disappeared – but where?  Who has it?  Or perhaps you want to show real data to highlight the challenge that Project A is using up a ton of your IT capacity, and that is why you need to buy some more servers, or memory upgrades.  The reason for knowing this is that IT is usually begging for money, and the business can begrudge spending extra when it is hard to show why.

I love this quote from “The New CIO Leader”

“Treated as overhead and often lacking even the most rudimentary chargeback systems, IT appears to be free and therefore without much value.  Lacking a price point to encourage responsible decision making, business leaders demand more service than they fund, creating a perpetual backlog of demand and placing IS in the position of continually saying no.”

The New CIO Leader

Dr Marianne Broadbent & Dr Ellen Kitzis

Harvard Business Press, 2004

In other words, IT keeps costing more money.  It looks like a burden, even if it is critical to the business.  And on the other hand, IT kind of appears to be “Free” (like beer), so why can’t I ask for 16 CPUs and 64GB memory in my new SQL database? After all, we now have a shared, pooled platform (thanks VMware!), so there is always a little spare capacity, and no reason I can’t have some, right?  Sound familiar?

What Does IT Cost?

It can be hard to draw the line on what you include under your cost models.  I think it is useful to start with technical resources – such as servers, licenses, storage and that sort of thing.  So other than staff, make a list of EVERYTHING that is involved in delivering a business application, and then we’ll find a way to allocate these costs to a metric, such as a VM hourly charge.  Having said that, don’t go crazy and turn this into an exercise for the Accounting Department just yet – keep it simple so you can get started.  Then you can refine.

CAVEAT:  I have used Hourly costs throughout my example here, because I am building a model that can be easily plugged into vCenter Chargeback Manager’s hourly consumption metering.  If you would rather use a monthly/daily basis, then adjust accordingly.

A Simple/Starter Example

Step 1 – Find your costs

A simple list to start with might look like the following.  

• Physical servers
• Network/storage switches
• Firewalls and other networking/security devices
• Storage controllers and disks
• Software licenses
• Rack space, power and cooling 

Step 2 – When does it cost?

This is important – you need to know what makes your incur these costs.  Let’s take physical servers as a good example to start with.  The core question to ask is “What event or resource causes me to buy another server?”  For most customers, the answer is memory consumption.  When I look across recent deployments, CPU capacity seems to be in plentiful supply, and memory is still the most constrained resource.  So in most cases, a project asking for 16GB memory is going to tip me over into a new server purchase, rather than the 4, 8, or 16 vCPUs that came with the request.

Your environment might be different, but the process will be similar.

Perhaps a physical server costs $20,000, including hardware maintenance over its planned life.  And perhaps this server has 96GB memory installed in it.  So on a resource basis:    

• Cost of server: $20,000
• Memory capacity: 96GB  (the most constrained resource)
• Upfront cost per GB: $20,000/96GB = $208.33 per GB.  

That’s an upfront purchase cost – but some VMs live for 4 months, others for 8 years, so how does this figure?  The answer is to have a time-based rate, such as an Hourly cost, applied against the life of the equipment.

Step 3 – How long does it last?

This is often missed – when will you buy a replacement for this component?  I spoke to a customer last year who charged projects per GB of storage, but only once, ever.  That meant that IT was responsible for the entire cost of a SAN refresh – not a pretty overhead, and again IT just looked like a dead weight every few years.

Going back to our physical server example, perhaps you plan to refresh your servers every 3 years.  That means the costs should be spread over that 3 year life, as below.  Don’t forget leap-years!

• Cost of server: $20,000
• Memory capacity: 96GB  (the most constrained resource)
• Life of server: 3 years (1,095.75 days, or 26,298 hours!)
• Hourly cost per GB: $20,000 / 96GB / 3 years = $0.007922/GB/hour.  

Fantastic! Now there’s an hourly rate for the cost of a physical server to support a VM using a memory consumption model!  At least, it’s something to start with.  How accurate is it?  Well, let’s work on that.

Step 4 – Refining the model

The reality is that cost models can get as complicated as you can tolerate.  Again, don’t go nuts, as usually the main goal here is to find out a relative measure of who is using what amount of resources, and a rough cost impact.  But let’s consider a few more things in our simple model.

In a VMware vSphere environment, there will be clusters of hosts, and very often in an “N+1” availability configuration.  That is, the resource consumption should go only so high as still being able to tolerate one host being down without impacting performance.  Also, we tend not to absolutely fill a server before considering it “full”.  Lastly, VMware resource management is pretty awesome, allowing safe over-commitment, and we can eke out much more resources than just the raw consumption or allocation we assume up-front.

These considerations so far boil down to a net impact on actual memory capacity of a server.  Instead of 96GB being “available”, we modify it as below.

• Raw physical server capacity: 96GB
• Clustering: 10 hosts (9 + 1, for High Availability)
• Level of “fullness”: 85%
• Degree of over-commitment: 20%
• Adjusted memory capacity: 96GB * (9/10) * 85% * 120% = 88GB

• Adjusted hourly cost per GB: $20,000/88GB/3 years = $0.008642/GB/hour. 

I raised an important consideration above, which is "consumption or allocation".  To some degree this depends a little on whether you are trying to account for used resources (consumption) or requested resources (allocation).  You will need to make some assumptions in either case.  But for internal cost reporting (as opposed to chargeback) I find consumption to be a more accurate representation of "where stuff goes".

Step 5 – Expanding to other costs

So, now do you just rinse and repeat for all the other costs?  Well, actually not for that many.  A good number of the “other” costs will also roll into the cost of a physical server.  How so?  I’m glad you asked – let’s look at some examples below.

• VMware licensing for vSphere or vCloud Suite
• Per physical CPU, usually 2 or 4 CPUs/licenses per server.
• The “life span” of a perpetual software license is tricky to account for.  You may elect to recover the cost of the license over 3 years, even though you won’t be re-buying the license when you refresh the server.  Annual Support and Subscription (or Software Assurance for Windows) has no such complications.
• Microsoft Windows Server licensing
• If you’re smart, you’ll be using Datacenter Edition licensing, which is also per physical CPU
• Blade chassis
• Will be an added cost to spread over the blade servers that it supports, which may be 8 servers perhaps.
• However, a chassis may survive 6 years, or 2 server refreshes, so we need to take that into account.
• Network switches
• Purchases usually tied to ports being populated, rather than bandwidth.  Again, we can equate this to a per server charge to some degree.
• Let’s assume 48 ports.  Perhaps 6 ports per server (a false example if using blade servers, of course).  That breaks down to the cost of one switch per 8 servers, for example.
• The same theory could apply to Fibre Channel switches for storage, if FC based.
• Switches will probably be refreshed on a different cycle, as per the blade chassis above, so again needs to be considered.
• Rack space, power and cooling
• Again, this can be tied to physical equipment, although it will be a rough estimate.

If I add these all together using some rough numbers out of thin air, I get to $0.023167 per GB of virtual machine memory per hour.  It is interesting that this is nearly 3 times the raw server cost, but this single rate is now able to account for servers, platform licensing, switches and datacentre costs.

I can likewise apply this type of approach to the remaining components from our short list above.  Storage per GB is easy enough, but perhaps the reason for buying another shelf of disk is actually I/O performance – some thought is needed to account for IOPS versus space.  You can probably generalize across the environment as a whole to determine the most obvious metric to use, and treat exceptional workloads as just that – exceptions. 

Keeping It Simple

Firewalls and security services are interesting examples of complexity.  The tipping point for expanding a firewall/IDS to the next bigger model, or another node, is probably network throughput or perhaps the number of rules processed.  You can either meter the network I/O of VMs to account for these, or perhaps assume an average of 6 rules per VM and meter on that basis. 

By the time we’re digging around at this level, however, we’re really tweaking a much more detailed model.  The overall accuracy of the model is only going to change in small increments.  There needs to be a decision about how far down this path you progress.  The overall goal is to get a rough idea of cost allocation.  Accurately accounting for absolutely all costs is an exercise for another article, as this would incorporate non-technical costs such as staff labour, facilities, and maintenance activities, for example.

The model above has boiled down to a few simple rates, which you can plug in to vCenter Chargeback Manager, as per the below screenshot.  You now have a method of finding out where your costs have gone, after buying all your new shiny servers and disks, and who in your organization is consuming how much infrastructure.

Driving Behaviour

One of the strong motivators to introduce cost reporting to a business is to help the business understand the consequences of their IT requests.  The model above merely accounts for resources are they are likely to be consumed - again going back to the question of consumption versus allocation.  A savvy business user might notice that CPU allocation and usage are not charged at all.  So we come back to the earlier question posed – why shouldn’t Project A ask for 16 CPUs for all their virtual machines?

On a technical level, we know that uncontrolled over-allocation of vCPUs will start to cause poor performance, from CPU Ready Time particularly.  While the example model I’ve given here is probably enough for cost recovery, there is reason to include a metric to encourage good behavior – such as splitting costs between memory consumption and vCPU allocation.  This will help the IT Department have a fact-based conversation with Project A about reducing its current allocation of CPUs, or asking for less in the first place.  Exactly how you split the cost, or if perhaps it is an overlay/additional cost, is up to each organization to decide.  This ties into a future discussion of reporting on Cost Reporting, as we’ve done here, and Charging Back costs to the business groups.  That is a different topic, as we must consider “cost perception” as well as cost allocation.

Further Areas For Expansion

This article is long enough already.  So some areas that are left to future write-ups could include:

• Cost Reporting reporting versus Charging Back to business groups
• Accounting for the entire IT budget, beyond the technical resources

Have any other thoughts?  Or perhaps you disagree with my approach here?  Please let me know and write a comment below!