bankingciooutlook

Scaling at Volume '11'

By Lou Senko, SVP & CIO Information Technologies, Q2ebanking

Lou Senko, SVP & CIO Information Technologies, Q2ebanking

As we just now begin to fold our security operations into our broader IT team, I’m reminded of how many times we made strategic adjustments to scale, how many times we refactored our path to success. Throughout my career, scaling has been a constant theme— either through explosive organic growth or through a series of acquisitions. In this article, I will share with you some of the narrative we are using inside Q2 IT to help us continue steering this rocket of the unbelievable growth we are on. In the four years I’ve been in my current role, the company has gone through a successful IPO and unprecedented year-over-year growth—to the point of we’re growing nearly as much in a quarter now as we did in the entire first year I joined—only 4 years ago. Here’s what that translates into for our IT department:

- 500 percent end user growth, adding more users in the next 60days than we had when I joined
- 279 percent employee growth, 7 new facilities
- New hosting environments, >20X server and storage growth
- IT team grew over 700 percent

An effective Strategic Roadmap contains these key elements:

- What ‘Good Looks Like’ at scale
- Your current state
- Your plan to get from here-to-there

Start by painting the picture of “What good looks like” at much larger volume: size, revenue, customers, end users, transactions—whatever work is in your environment. Plan higher than you think you can go, which may not be high enough; planning for 50 percent growth seems crazy until you actually hit 200 percent. You have to be able to clearly and simply explain where you need to take your business unit, which makes you the biggest challenge—your lack of experience at the larger scale, your lack of knowing how it should look, your lack of knowing where all the pieces fit into the larger puzzle. A knowledge gap could make you the wrong person to enact this transformation.

Fear not. Remember that it was the first time for everyone when ground control decided that 756.3 seconds was precisely the perfect amount of time to fire the Lunar Module’s descent engine on July 20, 1969, to land it safely in the Mare Tranquilitatis of the Moon (the Sea of Tranquility). So, the way to take your game up is to read, beg for and borrow information and best practices, and most importantly, use other companies as examples of what good looks like. It’s just another knowledge gap you can close. Find and use a framework that resonates with you (ITIL, NIST, etc.) and join the forums that can help you develop your governance (CEB – IT Leadership Council, ISF – Information Security Forum, …etc.) and your management, not just the technical cookbooks your team requires.

“ - beware of leadership terms that balk at replication because they - or the settings they are in - are so “special” or “different.” They may be suffering from delusions of uniqueness that foster misguided Buddhism. Too often, we humans convince ourselves that proven rules or technologies don't apply to us or the apparently unique place or situation we are in, when, on fact, we are fooling ourselves.”

Scaling Up Excellence: Getting to more without settling for less

Just as important as explaining and envisioning the happy future state is being very transparent of where you currently are. As uncomfortable as this can be, the whole point of the road map is to define the journey so you can instill confidence in yourself in believing that you got to where you are through the careful shedding of blood, sweat and tears, but yet still acknowledging that you still have a heck of a journey ahead. Recognize that you need more support, time and resources. This is the step in which you may also have to correct any notion that everything in your department and company is running perfectly and can continue to do so as you scale, not the perception you want to establish.

Of course you have to demonstrate that you also have ‘the plan’ to go from your current position to your state of nirvana. Again, it’s going to be a tough journey, but that’s why you should be excited for these opportunities, which are always disguised as risk and work. You can’t build the organization or the technical footprint that you’ll need two years from now—the pool of resources, the technical level of specific talent, the storage fabric–you can’t afford it, and you don’t need it just yet. However, you can work on the foundation that you will continually build on throughout the coming years. It’s important to set the expectation that it’s always three steps forward and one step back.

A key exercise is to pre-define the breakpoints that trigger reorganizing. Is it growing case backlog? Falling response time? Increased inbound workload? Whatever metric tells you that you can’t simply push harder and stretch, but have to stop, reconfigure and usually make an investment. I won’t go into technical architectures here, as there are many great technologies currently available that let you implement a foundational footprint and then add-to in a scale-up -and out-design. We use the infrastructure underpinnings as a differentiator, one that enables leading edge application functionality, performance and resiliency.

The evolution of our NOC is a great example of this growth:

• Started with three ‘part-time’ resources borrowed from Support working 17x5

• Oct/13- user growth continued - moved to small team of 5, professional system admins, developed response playbooks

• Oct/14- SLAs increased, volume at all hours increased - added to team (10), dedicat­ed management, move to 24x5 then to 24x7, new monitoring tools, new metrics

• Mar/15- SLAs continued to increase, new compliancy requirements - added more re­sources (14), added compliancy task (SOC/ SOC2/SOX404), more monitoring tools (per­formance), added a lead position

• Aug/16- adding more environments and services - now we are adding security alerting, new playbooks and escalation teams of SMEs

This roadmap outlined the need and direction, the metrics tied to business growth, wove into the budgeting process. Matching our team

growth and deepening skillset with increased in­bound workload and expanding responsibilities-the NOC now handles 25,000 alerts per month resulting in 3,000 cases which 85% case closure rate within that them.

Organizational scaling is another matter. There is nothing prescriptive, no rulebook you can tap. We have a couple of engineering architects on staff (myself included) to assist in mapping out the organizational development and the future shifts in scale required. Evolving current talent to keep the personnel you need, augmenting the existing employee base with new talent that has ‘been there’ and can help you lead the transformation are both necessary components to handle the increased workload across the company. Every new hire needs to bring more experience that closes the knowledge and scaling gaps, especially at the managerial layer. Each internal promotion needs to be heavily scrutinized and weighed against bringing in new, more experienced talent from the outside. Remember, you know what the role you’re hiring for does today, but can you really spell out all the responsibilities it will have in two years? Always hire better than you think you need.

As you give future executive updates, use the same consistent message, same goals, same journey. Everyone gets increasingly comfortable with the message and knows what to expect, making selling the future easier because envisioning the future becomes easier, expected and part of the narrative. As you evolve, it’s, “Here’s what we said, here’s what we did, here’s where we are and here’s what’s next.”

The end of the timeline will get clearer as you start unveiling it slowly as quarters come and go. This keeps your strategic roadmap relevant and allows you to course-correct with whatever new information is presented along the way. The roadmap helps you build the credibility that you know where you need to be, where you really are and how you are going to get there.

New Editions