How WAKE TSI saved Christmas for Montgomery Hospital Medical Center

It was 4 or 5 years ago on Christmas day when Ed Hemschoot, Director of Information Services for Montgomery Hospital Medical Center, called WAKE TSI regarding a network issue plaguing Montgomery Hospital Medical Center. Chris Witt, CEO of WAKE TSI, was sitting down with his family for holiday dinner when he received the call. He finished his apple pie and immediately came in on Christmas day to help Montgomery Hospital Medical Center. Here is the testimonial in Ed’s own words:

What Can the IT Community Learn from the Disaster in Japan?

We all hope that we never have to experience the devastation that is being felt in Japan. They are trying to cope with numerous challenges relating to basic human needs like food, water, and shelter let alone basic infrastructure like power and communications. Would you be prepared if your data center was in the path of
such destruction?

As a data center manager, you are responsible to account for all scenarios that could impact data center availability. Let’s look at the challenges you would face if your data center was located in or near the areas of devastation. The areas of concern are power, people, structure, and communications.

Power
Odds are that you will not have power. This is a basic concern that every data center manager accounts for. The problem with this situation is the power will probably be out for an extended period of time. Normally this would not be too much of an issue because you have negotiated those diesel fuel delivery contracts. The only
problem is that there are no usable roads between you and the fuel source. Even if you are using natural gas, there is no guarantee the pipeline would survive the earthquake.

People
What is the status of your staff and can they get to the facility? Even if they can, would they? In a disaster, your staff is going to be more concerned with their own safety and the well being of their family. Your data center will be pretty far down the list.

Structure
An obvious question is: What is the status of the structure? The building may be damaged to the point of being unusable or dangerous.

Communications
If your data center has survived, you have personnel and power, you may still have issues if you don’t have your communication lines. Most organization will employ diverse, redundant connectivity. However, during a regional disaster, there is a very good probably that all of your connections will be impacted.

Key to Survival: Reundancy
So what does this mean? Because of the multiple catastrophic events, there is not a single data center that would be able to maintain operations. Your only solution is to fail over to a redundant data center. Many (but not all) organizations are using redundant data centers to account for catastrophic failures of their
primary facilities. However, many are keeping the data centers within close proximity to limit the amount of communication latency so they can maintain synchronous storage replication. This is around 30 miles or so. Again, using Japan as an example, the destruction radii would be large enough that there would be a good
chance of losing both data centers.

The bottom line is understand your risks and the probabilities. Make sure management is in agreement. Don’t get caught surprised; be prepared to handle the worse case scenarios. Remember, if you think “it can’t happen here”, so did every Japanese data center manager right before the earthquake hit.

Do I Need a Data Center?

If you still own your own data center and this question has not already crossed your mind, then you might be spending more money than you need, subjecting your organization to unnecessary risk, or both. Your data center is the foundation for all your core computing resources. Any flaws in the data center puts everything installed in the data center, no matter how well implemented, at risk as well.

So what do you do? You need to securely house your servers and storage somewhere, and not go broke doing it correctly. Today, you have far more choices than you did just ten years ago. The added choices include numerous viable offsite solutions. While there are many permutations, the two basic choices boil down to keeping your data center in house or have someone else host it for you.

Regardless of the number of choices you want to consider, we always recommend to our clients that they evaluate fixing what they already have. It’s the devil you know.

Evaluate Your Options

Before you evaluate your options, you need to understand your risks and how much risk you are willing to accept. Are you a 9 – 5 business that only requires computing functions during the typical workday or are you a 7/24/365 operation that tolerates no downtime? If you have an outage, can it affect the delivery of patient care? Are you putting someone’s life at risk? Do you know how much it costs your organization for every hour of unplanned downtime? Did you know if costs about $100,000 per hour of downtime for a small to medium sized hospital?

Know Your Risks

Risks come in all shapes and sizes. Knowing your particular risk areas empowers you to either accept that risk and the consequences, or fix it. When looking at the data center, your risk areas are in facilities, operations, and monitoring. Is the physical plant appropriate to house a data center? Do you have sufficient power and cooling capacity? Is the data center structured correctly – hot/cold isles, airflow, rack layout? Is it secure? Do you have a dedicated data center operations group who oversees all activities? Are they always on hand when the data center is operating? Do they have well defined policies, procedures, and processes? And are they following them? Are you able to monitor, alert, and report on all activities taking place in and around your data center? There are a lot of questions and until you objectively assess your situation, you will never know where the bodies are buried.

Making A Decision

Ultimately, you need to make a decision. This is not a decision to be made emotionally. It is a strategic business decision. Is owning and operating a data center a core strategic part of your business? If you are an IBM, Comcast, or Verizon, I would say Yes – Owning and operating your own data center is very strategic to your core business. If you are a hospital, manufacturer of goods, or a services oriented company, a data center plays a supporting role rather than being a core strategic function. You also have to evaluate financial models for keeping it in house and moving to an outside managed services solution. This is not always a pleasant exercise because it involves making hard decisions about giving up control of your assets and potentially reducing the size of your staff.

Every option has its pros and cons. There is definitely no “one size, fits all” type of solution out there. But knowing your current risks, understanding the risk tolerance of your organization, and having a clear organizational strategy will help guide you through the process. As a local clothing retailer used to say, “An educated consumer is our best customer”. This is also true in the data center world. We are “Architecting IT for the Future”.

So why are you in the data center business?

Making the Most of Data Center Upgrades

As usual in IT, some the greatest obstacles are not technical but social or political. The drive to service-oriented architectures and cloud computing involves a lot of disruptive technologies, and CIOs cannot afford to lose sight of the fact that what’s being disrupted are human attitudes and relationships.

One of the most critical components in data center upgrades and new builds is staff training. Yet, despite the educational environment, training may be scanted in higher education. Gene Kern is Executive Vice President and CTO at WAKE Technology Services, Inc., an infrastructure consulting firm with a large higher-education practice. “It’s really strange, but especially in smaller universities, the hardest thing to get may be training dollars for IT staff.”
He notes that training involves more than “course work.” “Something that’s rarely well-enough funded is attendance at more of the annual events where IT professionals can network and get more training in new technologies.”

Read More

Avoidance, Recovery, Continuity

Businesses can be interrupted by events as simple as a network outage, or as devastating as a tornado, fire, or even an act of terrorism. The most appropriate step you can take to reduce the impact of a disaster is to first realize that it could happen to your business.

Let’s make sure we’re all on the same page, definition-wise.

Disaster Recovery (DR) describes the strategy an organization employs to deal with potential technology disasters so that the effects will be minimized and the organization will be able to either maintain or quickly resume its mission-critical functions. It follows then that Disaster Recovery Planning (DRP) needs to focus on the data, hardware and software critical for a business to restart operations that have been shut down by a disaster.

Disaster Avoidance, as the name implies, is the process of preventing or significantly reducing the probability that a disaster caused by humans, machines, or forces of nature will occur; or if such an event does occur that the effects upon the organization’s technology systems are minimized to the greatest extent possible.

Business Continuity (BC) procedures kick in as soon as a disaster is triggered. These procedures are a progression of preordained tasks, manual or autonomic, aimed at enabling an organization to continue serving its customers during and after a disaster. It precedes, and ideally minimizes or precludes, the recovery process.

Business Continuity Planning (BCP), explained below, is an excellent starting point for a business to focus because it yields valuable input that can be used to develop cost-effective Disaster Avoidance policies. The BCP process requires the business to identify its mission-critical operations and indispensable processes and data that are essential to keeping the business functioning as a disaster is occurring. Only by knowing the value of the processes and data you are protecting can you have a basis for effectively allocating dollars towards protecting them with Disaster Avoidance methodologies.

After the BCP has prioritized the key business processes, the next step is to identify the specific and significant threats that could disrupt normal operations. And, finally, devise mitigation strategies to ensure effective and efficient organizational response to the challenges these specific threats create during and after a crisis. While there are certainly overlapping features and objectives of these three strategies, it’s important to understand and benefit from the distinctions, because they are certainly not mutually exclusive.

Disaster Avoidance policies and procedures will minimize your exposure to certain disasters. Business Continuity Planning will maximize your ability to keep mission-critical processes working as a disaster unfolds or to resume as soon as possible afterwards. And, the main objective of a Disaster Recovery Plan is to bring operations back as quickly and seamlessly as possible after they have been interrupted by an event.

Getting Back On Track

Some skills are learned from formal education and great training. Some of the more subtle skills necessary for success in business and IT are learned only through decades of hands-on experience.

The professionals at WAKE Technology Services have both. It was the application of these combined skills that enabled WAKE to successfully perform a number of key roles during a recent IT engagement involving the deployment and implementation of a new storage architecture.

Setting the Stage
In 2004, the development of an RFP to overhaul this client’s IT storage system was initiated. The project had hit a number of bumps and roadblocks along the way. Personal preferences, differing agendas, subjective evaluations of vendors, and discord among co-workers prevented the IT
team from defining clear requirements and objectively identifying and evaluating vendors. They had difficulty even agreeing on which version of the RFP should be submitted to vendors. Arequirement within one version of the RFP was the ability to support a legacy OS. That was inserted knowing that a particular vendor could not satisfy the requirement. But, as it turned out, the OS, developed by Digital Equipment Corporation back in 1979 wasn’t supported by any of the vendors.
To complicate things further, there were rumblings of questionable tactics by some of the vendors during the inital RFP process. Innuendos of kick-backs and alleged attempts to circumvent the normal “chain of command” by seeking audiences with the CIO and even the Board of Directors contributed further to the chaos. Clearly, the project was not moving in the right direction.

The Role of Mediator
As the hospital’s IT team entered the eighth month of grappling with the development of the RFP, WAKE’s Gene Kern was approached by the Vice President of IT to conduct an independent evaluation of the data center. As part of that assessment, Gene quickly discovered additional problems with the hospital’s storage configuration and the required storage maintenance procedures. He also identified numerous points of failure within the system. Since the client
was already engaged in the RFP process for upgrade of the storage system, Gene was asked to intervene, conduct a thorough review of the in-progress RFP project and ultimately mediate a resolution to the existing stalemate caused by an overall lack of consensus within the team and guide the process through completion.

The Role of Project Manager
WAKE immediately began the task of completely managing the entire RFP process. Step one involved the identification of all requirements (technical, business, and vendor) necessary for a successful implementation. Next, WAKE followed their proven methodology for completing an IT engagement, to take the project from RFP through final deployment. Specific tasks ranged from the development of a convincing business case, to initial design, to managing all vendor communications and negotiations, client conferences to evaluate and score bid responses, assessment of
final bids, contract award and ultimately overseeing the implementation.