Orion at Scale: Best Practices for the Big League
Some reports from your IT monitoring system help you do your job—like auditing, troubleshooting, managing resources, and planning. There are also reports that you create to let your boss know how the IT environment is performing, such as enterprise-level SLAs, availability/downtime, and future-proofing. But what exactly does senior management really need to know, and why?
In this session, you’ll hear from SolarWinds CIO (and others) on what information is relevant and how it’s used.
Hello everyone and welcome to today's panel, The Executive BabelFish: Reports your CIO Wants to See. You know, too often communication between executive leadership and IT on the ground, really gets in the way of smooth service delivery for users, and the CIO's ability to meet their SLA with the CEO. We've invited three technical experts and moreover, communication experts, about breaking through ineffective procedures, rejecting IT dogma and challenging habitual behavior. And while trending topics like DevOps and Agile may pop up in this discussion, we're primarily going to be focused on changing communications culture to align IT with the business, rather than being methodologically prescriptive about how to accomplish that. You've told us over and over again that you have great ideas about improving IT responsiveness to your businesses, and our panel will focus on how both sides can communicate better to get great IT ideas into production. I'm Head Geek, Patrick Hubbard, and joining me today are leaders from both sides of the conversation. First, we have two technology executives. First, Rani Johnson, Chief Information Officer of SolarWinds. Rani, thanks for joining us.
You know, it's really interesting because the THWACK community doesn't always get a chance to meet some of the senior executive staff here. And they always ask questions when we see them at trade shows or on THWACK, like, well how does SolarWinds do IT? And I guess the role is actually pretty much what you'd expect.
Maybe and maybe not. This is, first of all, my dream job. I get to provide IT services for the coolest IT software company in the world. Kind of my scope a responsibility includes development, operations, DevOps--including that-- enterprise architecture, project management and the typical infrastructure jobs.
Awesome. And then you get to deal with a little bit of care and feeding of the development team. And some snowflakes and a little bit of that.
We have a lot of unique little snowflakes.
Awesome, awesome. Well thanks for being here.
Second, we have Joel Dolisy. Chief Information--I'm sorry-- Chief Technology Officer for Kinnser Software. It's great to have you with us today.
Thanks, Patrick. Happy to be here.
Well, at Kinnser you're focused on healthcare, right? So you’re doing clinician mobile applications a little bit, but a lot of services and SaaS, and running data centers to actually enable healthcare at physicians’ offices.
So yeah. So Kinnser, basically, is a leader in the post-acute world in healthcare. Which, basically, is all around delivering healthcare at home or hospice's at home. And so basically, we provide a software for those agencies to actually run their business day in and day out. And we deliver that as a SaaS set of solutions and also mobile solutions in disconnected scenarios. So it's a really interesting world where we have a lot of challenges, a lot of compliance. But also, it's really rewarding actually, at the end of the day, you're actually helping getting people getting better. Which is really, really good.
So a chance to make the world a better place but also deal with DevOps at the same time.
Exactly. So that's a dream job.
Awesome, well thanks for being here.
And last, representing the IT side of the house, and most of you in THWACK, is Mindy Marks. She's a Senior Director of IT for Sandow.
Hey, well thanks so much for being here. You know, and it's really funny because she's developed a really great working relationship and a communications approach with Sandow's CXO team, and reports directly to the CIO. Thanks for being here today and also thanks for being the voice of people who actually have to get stuff done.
Yes, yes I'm definitely very hands on. I do have a great rapport with my CIO who I directly report to. But I am very much within infrastructure and rolling out projects. And I'm very much part of the team and communicating up and communicating down. So my role is very unique. I am very geeky too. I like to get my hands on a lot of equipment as well. But I also like the communications side and the business side, and bringing value to what we're doing.
Awesome. Well thanks again for being here. Okay, so I suspect that I'm not really going to be saying much today, and that's really a relief. And by that, I mean we all know what we need to do, right? What the opportunities are for IT to really transform out businesses. But often times it's really hard to know where to get started. So let me start with this question. Just, you know, a great big omnibus question. The point of communication-- whether it's planning, reporting, status-- is to really be able to just get stuff done, right? And increasingly for IT, and the businesses' demands on us, is to get stuff done faster. To find ways to do things faster. So the first question is, from the business' perspective, from the board's perspective, what does the business actually want from IT, right? Do they want speed? Do they want features, they want dependability, they want quality, they want cost control? What do they want? If they have to pick one thing, or at least a top couple of things, what do they want?
So, I think they want it all. And they want it without having to communicate too much to IT. They just want us to get our work done. And we have to make sure that we've set proper expectations, set specific kind of communication cadences, and just deliver. They want us to be predictable. They want us to be accurate; they want high quality. They're never going to choose one. They've invested in the cost of IT to deliver, and that's really just our jobs to do that work.
I would agree 100%. They want it all. They want it all from the top to the bottom, and I agree with you. They really don't-- they really don't care about how you're getting it done. They just want to see it done. They want to see the features there. They want to see you bringing the business quality. They want to see the new tool that's in place that we've provided. They want feedback from the people that are using it, and telling us that it's a great tool, and it's saving me time on my job, and my business is more effective; I think they want it all.
Yeah, they don't-- I don't think that they want to hear about it.
They don't want to hear about it. Which actually is right. I think they should focus on the business level constructs. And how do we make money? How do we stay in compliance? All of those are business drivers. All the tools on "Hey, can I get on the Wi-Fi? Can I get my power?" You know, all of that stuff-- Keeping the lights on. It's kind of assumed it works. And so it's kind of an unfair fight that you have to fight when you're in charge, because you have to keep those things; you have to spend money doing that, but they don't want to hear about it. They just expect it to work.
Co-equal top priorities.
Oh, we don't have that.
Yeah, that never happens.
That never happens.
Okay, so maybe that's a place to start. I'm hoping that we eventually get to some reports that are actually, for example, reports that are useful, right, with management. But that first--that first step, right-- having that conversation where you say to them: Based on this budget, with these requirements, these are the subset of things that we can actually deliver. To help them step away from that misconception that somehow you can do it all. How do you even start with that conversation?
I think it's really-- you have to have great communication with the top of the business to be able to define those needs and requirements. I think you have to really pick the top things that are the most priority to you, and things that you really think you can accomplish. So many times, you're right, the list is endless; the things that we should be doing, need to be doing, even from hands-on on the ground. But really, defining what's important to the business, and what's going to bring the most value, and where you're going to put your IT resources, I think is what--is how you define-- define what's important.
And I think that's the beauty of Agile and DevOps, is that you get a chance to go and list all of the priorities. Make sure that it's clear of backlog. Make sure that it's fully prioritized, and then you just determine what you can do against that list, with the time and the resources that you have.
Yeah, but I think it's also about alignment. I think it's-- If you're going to them with your list, and that doesn't match at all with what the business is trying to achieve, then it's going to be a tough battle. And so I think that's what it is. Sure, DevOps, Agile, and all of that can help in managing that. But at the end of the day, it's all about alignment and getting sure that everybody's aligned on the same set of priorities-- at least at the high level. You always have additional things that you have to take care of, that will never go up on that list of priorities, but it's important to actually have that.
Yes, and as the business changes fast you have to change fast. So we have to adjust in our IT. You have your set things that you put one, two, three, four, five-- and the next day, all of that is scratched, because the business comes to you with a new idea and their like, we're going to do this today. And this is what's important. And no matter all the time and money and effort you've put into this in your teams, you kind of really have to just be able to change fast and go with the business.
So you're talking about learning how to talk to the business about the business first.
So, where do you think that breaks down? Is it that the business doesn't want to explain it to IT, or they think that maybe IT won't be able to understand it or consume it? Or is it, IT-- you know, I just want to work the help desk, and deal with my issues, and try to get my new projects up and running. I just don't really know-- I want to use the requirements as a demarcation line between what I do and the business itself. Where does that first tendency not to communicate come from?
I think that your example there about the help desk that's what they really think about when they think about IT. But the problem there is IT is involving a lot of other projects-- sales enablement, you know, all of those things-- development, you're keeping the source and all of that...
Help desk is the great interrupter.
It is. It's always seen as this thing where people go when they have problems. But I think it's not just IT. It's not about answering problems. It's about actually being proactive and supporting the business and going forward. And I think that's where IT has the most actually to help, and provide value, is on that front. And not just being reactive to problems. You know, so I think that's important to actually focus on that. And I think part of it is IT needs to be part of the discussions within the different departments, and making sure that you are supporting the business.
Yeah, I absolutely agree with that. I think really a lot of times your help desk, or the people that are really seeing the problems, you know, they're helping your sales team, they're helping your HR team, and they're actually dealing with issues that your user probably has a better solution, or a solution that we can provide them to do their job better. And they're really seeing it from the bottom. And so I think having that communication, and really watching what's going on at that level, at the help desk level, really starts to show business problems. You know, things where we can bring value from the DevOps team, from an infrastructure side. From a software development. You know, we can provide things by really understanding what's going on at that bottom too. They really do bring so much value that I think kind of gets skipped over. On my team, I have two engineers that work directly with me, and we have team meetings every week. And I listen to all their problems. And a lot of times, things that we're doing in infrastructure is affecting them, and we have no idea that's even affecting them. We rolled out something without communication. We made a change, and now they're seeing the effects. By having that conversation with them, it's letting us know, "Hey look, there's an alarm that's going up here. And yeah, we've affected something somewhere and let’s fix it." Or we have a real business problem that we need to fix-- a process, a workflow, something we can do better.
One thing I'll say, what we should aim for in IT, is to not just be problem solvers, but to really understand the strategic direction of the business, that we can actually be business enablers. That's one of the, kind of, aspirational goals, and you have to, kind of, really build a mature IT department to get to that. But if we start at the problem level, we are often not invited to solution, to actually enable the business for growth and profitability.
But there's a big parallel between powerful software companies where you build products. And there's a whole set of processes around that, where you do product strategy. So you align yourself in the business with where you want to go. But you also listen to your customers. You take that feedback loop. And it's funny how-- --well, I find it funny-- how there's a dichotomy there between how we build products and how we run IT within those software companies-- where we don't do a lot of those things there. Either we are very purely reactive or we are purely strategic, but then we don't take care of the care and feed of the internal users, and then take that feedback loop and bring that back in, and make those services better, basically. We just keep answering the same questions over and over again. And I think that's where we have-- there's more to do there. And it's all about communication there.
And it just goes back to what you mentioned; we have to stay aligned. We're going to have to evolve but we have to stay aligned as a team.
Yeah, it's kind of interesting sort of the Dev in DevOps is pretty well organized and I think most people know what that is, right. Development has been development in various forms but kind of narrowly defined since the beginning. But IT is more of sort of snowflake in every environment where you find it. And that seems like one of those things to-- whether you actually have developers that are a part of operations, or at least are willingness to find people and put them in nontraditional IT rolls. Like find people inside of the business and then bring them into IT. Where you might say, why do you have a business analyst as a part of the IT team. Well, they actually know the business, and that was something that we talked about the other day.
Yeah. I've experienced that as well in my department in my current position. There was--he was working for the company-- he was super excited about the company-- really knew the processes, the business in and out. We brought him over to the IT side. He started creating applications, working with a lot of the brands. You know, building CRMs and really understanding what's going on from a com-- you know, how these people are sales driven-- and tweaking, reporting, providing so much-- and, you know, he really just came from being an executive.
The level of depth I think some of our technical team members have to have, require us to supplement them with people who actually have that business acumen or that domain expertise. So I think it's a wonderful addition to be able to bring business people over into the technology team, so that we can really make sure that we serve the business well.
I think it also goes vice versa as well. I think if you take an IT professional and you put them into the business analysis role, and put them into a department that's having problems, and you sit there and you really figure out what the problem is. You can really take your IT knowledge and help the business in any department you put it. I really think it goes both ways. Take somebody who is really great with processes and understands the business; you can flip flop both ways.
I think what matters most is for them to be driven. If they are driven, then you can put them in whatever situation and they'll be good. They'll get something done. If somebody's not driven then you're going to go nowhere with that.
So that really, kind of, brings another question, which is: Where does improved communication start? Does it start with someone who is an internal champion for aligning IT with the business, and actually seeks out an executive sponsor? Does it start with executives who nurture and develop potential champions on that team? Is it both? Do they meet in the middle? What are a couple of ways to really accelerate that?
I think you need to have both top down and bottom up. I think that if you have one of them it's going to be super hard. You're going to hit the ceiling at one point or you're going to hit the bottom if you come from the top. Because, at the end, the people in the trenches are the people who are actually doing the work, and they need to communicate back up. And people at the top they send the direction. If that is broken there, then you have a void in the middle. And so it's really both actually have to meet in the middle. And so it's extremely difficult if it's only one-sided, because it's an uphill battle. And it's the same thing for whether you start an Agile project, or you start Agile into a company that is not Agile. If you're trying that into isolation of a team, it's extremely difficult. You can be successful in your team, but it's going to be extremely difficult to scale that. It's the same thing there for communication.
Do you see that where politics tends to kind of get in the way? A lot of times, pilot projects are sort of used as that. And sometimes someone will be given extra resources, especially in a cost center, to do something a little differently, and then their viewed by the rest of the team as a cowboy, right. And that there's almost an anim-- can be some animosity there that they're getting to do something special and fun. Instead of--maybe the executive goal was, I want you all to see the opportunity for us to invest more in IT as a function of transforming the business. Is that just sort of a missed opportunity to communicate better about the goals of that? So that if you start those pilot projects, if you bring dev or a specialist into a team, to be able to say, "Hey, I'm doing this, and the rest of everyone in IT, this is an example that I would like for all of you guys to get involved with."
Yeah, I think the whole-- There was one word in there which is the 'specialist' in the team there. I think that's a dangerous word, because you know it creates-- that's how you create snowflakes. That's how you create something that's actually not scalable, because you have a single point of failure. I think we all have 'Joe' who's the guy on our team, or 'Alice' who's the girl on our team basically. And if Alice gets hit by the proverbial bus, it's pretty bad. And I think that's the danger there. I think that-- And it depends on the company culture also. I mean, you look at Cisco for instance-- a lot of their innovation came by pulling their best engineers, putting them into a spinoff, getting them to work, and then taking them back in. Well, I think that's great for the people who are in that team. I think the people who stay on the main ship, and see that other team there, it's extremely hard at that point to motivate them. And so I think that you have to be very careful about how you deal with that.
Absolutely. I encourage, kind of, transparency in our entire project portfolio. And if we take on a special project, if you will, we're really clear about why we're doing that-- what objective we're trying to meet. And so we have to encourage transparency about why we're endeavoring to move these projects forward. What is the motivation? Be it growth, revenue, productivity-- we need to be really, really clear about why we're doing these bodies of work, and make sure that we're transparent about what we're setting out to accomplish.
Yeah, I agree. I think full transparency on a team is extremely important. I think that you can have teams that do have pilot programs. But if you explain it properly, and the team is all on board with the goal at the end of the day of what they're working on, I think you're going to get the backing from the entire team. I think a lot of times, it's when they don't know about it, and they haven't heard about it. And it hasn't been brought up at the team meeting that they were going to work on this specifically, because we're going to do it fast. There's reasons, business reasons, behind why we're doing it this way. I think people are on board. I think they just want to be told the truth. I think they want to know what's going on. And I think that if you do that, you have everybody's buy in on the team.
Yeah, it's one of those things where if someone starts on a project, and let's say they're doing some automation or working on deployment or delivery, and if you say to them, why are we doing this? Where you've maybe cut my budget for storage, or I need to actually do some hardware refresh as part of a normal hardware cycle, and we're going to invest in this strange thing. And if you can say to them, well this is actually so we do deployments Monday through Friday, between ten in the morning and three in the afternoons, and not on the weekends anymore-- most people can get behind that.
Oh yeah, absolutely! 100%. And the team that I lead, I find that has worked very well. We all have different levels. We're all on different levels of our career-- from your help desk engineer to your director, to your CIO. And when you put them all in the same room-- although there's a lot of terminology they start to learn, acronyms that they maybe were not familiar with-- they start learning things about databases. And so you start having help desk people really understand what's going on. And I think it moves their careers forward. They start to find which path their going to take. And because we want to invest in your people-- and so they've been there, and they're the ones in the trenches doing the tickets every day, working with the people-- sometimes not easy people, we all know-- and you need to really find a path for them as well. Where are they going to move and where are they going to find their love? What's your next step if I push you? Do you want to go into networking? Do you like security? Because, you know, you always want your people to go to the next step in their career. I think by having them in those meetings, starts brainstorming-- it starts people really thinking.
So what's an example... You know, I think we used the word 'quality' just a minute ago. We kind of talked about that. Businesses like to use that as a metric. It's not always easy to figure out what drives that. But what's an example of maybe a quality metric that you see, when you realize, if you are communicating clearly with a team, and we're-- or it's a champion on that team that's communicating clearly with you-- what's a sign that they're beginning to get it? You're giving them a little bit more leeway; a little bit more rope and they are actually beginning to be more self-sufficient. They're better aligned. What are the sort of the tea leaves that you read to determine that communication is actually improving the quality of service delivery?
That's a great question.
On my team, I've actually got a great example of this. So we're a very small team, and we all wear many hats, and we all are very hands on at the end of the day as well. And so I've had to kind of step out of that roll, and really give them the tools to say, I'm empowering you to move forward. You may make mistakes. That's not a problem; we'll deal with them. We all do, and it's a learning curve as you get there. So I've seen major growth, and being able to step away, step away a little bit. And sometimes you have to go back and hold their hand again. Step away, step away, step away. Defining quality can be in so many different ways. I mean, that's such a vague term in saying how do I define quality. The definition would be something without defect, right. What am I producing without defect? What am I putting into place that I don't have to redo? How many times am I revisiting the same problem without really fixing the problem? Those things can be all defined as, what do I define quality?
I think to paraphrase that for me, it's about rework. Because there's a beautiful book about that.
You know, so. But it really is about that. It is about that. Because if you look at it, if we build the wrong thing, or if we deliver the wrong service, then that means that upstream we actually got the wrong requirements. If we delivered it 'buggy,' then that means that we did a bad job in terms of implementation. And that means that communication there was not happening. And so I touch every single one of those phases. Those are the more of the-- the KPIs, you know that I'm tracking-- what is the amount of rework that we have to do all along the deliver chain?
I look at quality as an expectation. And you have to communicate and get really clear on what that expectant of that defined expectation is. And so within--let's say it's a project delivery expectation-- Quality is scope, schedule, budget. If you look at it from a coding perspective, it's defects. But it's setting an expectation. You may have a project where some certain level of bugginess is okay. But as long as you define that objective, kind of that service level expectation, to me that equals quality. And if you make that, if you hit that, then you delivered, I guess, the quality expectation.
Right. And not to get stuck in sort of defect-analysis paralysis. Every time we get on an airplane, sometimes you think, how many bugs, or known bugs, are being tracked in the flight-- in the flight management software?
I'm getting on a flight at six. [Laughing] Thanks for me thinking about it before I fly out.
Okay. Well, you talked about being able to communicate and try and fail. I think we hear the term 'fail fast' a lot, right. And I think most senior executives-- a lot of CEOs talk about we want to be able to enable 'fail fast.' And they have peers who are using it in their organization and it's really effective. Only, other parts of the business maybe can respond a little bit more quickly. The runway and, sort of, the factory floor time for a lot of what we do in IT is a little bit longer. What are some of the ways that you can use communications and reports with senior execs to kind of nerf 'fail fast' a little bit? To give yourself-- To kind of create a little bit of a buffer to improve delivery, reliability and quality. And let them see that occasionally people make mistakes, but you're mitigating risk.
We try to deliver minimum viable products. And so we make sure that within each sprint, or within each quarter, that we're actually delivering something. It doesn't mean that it's going to be defect-free. It doesn't mean it's going to be something we can actually, necessarily put into production. But we want to show that we-- we want to demonstrate some level of value to get the investment, to continue that work either into the next sprint or into the next quarter for work. And so I think there is a level of grace we get in delivering small work packages versus delivering larger efforts.
I think that that's where, in the past, projects get lost. You know, it take six months, nine months, twelve months and there's nothing. Then you're over budget or it doesn't deliver the value that actually was expected. I think that's where Agile, DevOps, you know, all of those things actually are bringing or leaning-- whatever process you want to use. It's all about showing increment of value to the end user and customer very quickly. And then you learn from that. And then you rinse and repeat and continue. The only thing is you can't just stop at the first MVP. [Laughing]
Yeah, I think you lose a stakeholder's attention, at some point, if you don't deliver. If they give you their expectations and they don't get it fast enough, they've already moved on to the next project, and completely forgot about it. And now you don't have their buy-in. And now you're banging on their door to try to get them to do it. You know come back and revisit what you've just put into place. So I think being Agile and using DevOps is really important when it comes to this.
I think, in the spirit of reporting too, you have to actually show that you did what you said you were going to do. You have to actually be able to have something that's demonstrable, not just that MVP, that says, here's what we said we're going to do, look, we have done it. And make sure that you're tracking and reporting on that. And also being transparent when you don't hit all of those particular tasks, or the little, smaller objectives within that.
So the point that you made about dependability-- it's something that management actually does care about--or reliability. So you're suggesting that it's not-- Rather than just give a traditional report of availability, or outage, right-- actually overlaying that with a metric for how quickly, or the number of deployments that you've done, and where they happen in there. So that you can actually show sort of continued reliability in spite of something that looks traditionally risky with waterfall, right.
Okay, so here's a question-- because you brought up Agile, and of course, I'm going to want to talk about that-- Because, you know, DevOps fixes all the things.
Time to leave. [Laughing]
DevOps are--Right! Right. You gave the example before of doing production deployments during the workday and not on the weekends, or during maintenance windows. I think that's something we can all agree, as an outcome of an Agile technology, is something that we want. Why is it difficult to even get started having that conversation? That seems like a good indicator of communication between executive and IT. It's like if you even express interest in Agile sometimes; does IT actually get pretty much panicked by that?
I think we just made the first edit there about what you mean by Agile. Because I think, for me, Agile is all about delivering customer value very quickly. Okay, that may mean that you may want to deploy during the day, if it actually is not. It's a zero downtime deployment; it doesn't hinder them. But again, Agile by itself is all about customer value, quickly. And show increments of progress there. And so I think that's. I think, what I've seen in a lot of these is that gets lost into a lot of organizations. And they start focusing on techniques for this, for that. And they forget the big picture. Which is, I just need to provide value quickly to the customer-- whoever that customer is. It's inside your enterprise. It's outside the enterprise. It's product customers. It doesn't really matter. At the end of the day, if you're not focused on those end users then you're going to have a really hard time actually pushing Agile. Because it's going to focus on the wrong things; it's going to focus on techniques.
Yeah. The methodology, not principles.
Absolutely. I find organizations are pretty receptive to Agile, and they understand that this means that, let's say, the finance team is not going to have to, over the weekend, test something too. When we deliver value or change to a group, they have to make sure that they've kind of accepted it, even post production. And so if that means that, hey, we're in our seats at the time that most people are available to actually do some of that tests, or that final production validation, it's actually--it's convenient for everybody.
I see that more and more as well.
Now, if we do something risky, I do prefer that on the weekend. [Laughing]
Or at least in a lab.
Oh yeah, in a lab.
That's sort of those technical debts that a lot of times we don't think about. It's like we want to experiment. Great. Let's do it in the lab. We don't have budget for lab. Okay, well, that's something to communicate about up front, right? One of the things that's kind of interesting, and we talked about it earlier, was the types of reports that you would see. Like, one of them would be adoption rate reports for new features, and new technology that might be part of an application that's been around for a long time. Does that mean that IT really needs to know more about how the application works? Or the user expectations of that application? To be able to figure out how to create those reports in the first place?
Absolutely. If you don't, then you're flying blind and that means you're not aligned with what the business wants.
Right. And that and you have a chance to turn users into instrumentation as well.
Yeah, it's so easy nowadays to actually collect a lot of data. You have a company that builds software for this. But you have to know how your end user is actually experiencing the product. You need to know that live and you need to be able to act on that. So it's not just about collecting the data. If it's collecting for the purpose of collecting it and not doing anything with it, then don't do it. But you need to be able to react quickly. Like, if you deploy a new UC communication, for instance, in the company, and you wait, you know-- and if you do a big bang deployment, and you wait two months after that to figure out if people are happy, then I think you've lost. Because you waited too long. You need to do that in steps, you know, in increments-- get feedback directly, you know, a few days after. You have expectations about 50% of the users will actually start using this actively. And if you're not on that, then you need to react to that. If you're below that adoption rate, then you have to think about what do we need to do to actually get that back. Did we get it wrong? Do we have the wrong solution? Do we have the right solution but we implemented it wrong? There's all of those questions that you have to answer. And if you're above, then great! Then obviously, you hit the mark even better than what you thought, and then you can continue.
Absolutely. And I think part of that is setting that success criteria, and being willing to, kind of, roll back or call that back if that's not the right solution.
And I think that reporting can be defined in so many different ways. We're using the word 'reports' and 'reporting.' But there's so many different types of reports that people use to create value for a company. You mentioned anomalies. You know, we're getting a report every day that we're looking at. But really, it's not the report that's bringing a lot of value it's when we have that anomaly and there's something that we really need to address and take care of, that's affecting the business. It's affecting budget. It's costing us money. That can be a problem if we go down, such as you know.
Absolutely. I think IT leadership is not looking for like, are we doing what we say we're going to do every day? It's more so, we set a service level objective; we set an expectation with our IT leadership teams. And we want to be alerted if there's any kind of leading indicators that we're off track, and as soon as we, kind of, basically failed against that service level objective.
Yeah, and I believe that's where reporting really becomes helpful for IT. It's bringing up those things that are really going to affect us, and how are we dealing with it, and are we communicating-- For me, I would be communicating to my CIO letting him know that this is the risk that we're looking at. He may not have a physical report or an email, or an excel sheet or a gantt chart-- well, not so much for a report, for a gantt chart-- but he really wants me to come in and communicate that with him. We're a small IT team so I'm not sure how it works with your communication...
Have you seen instances where you bring a novel report and they actually get excited about what's going on in IT? An example would be, to your point about not just showing availability, but actually your ability to remediate novel issues, right-- so almost a responsiveness report.
I do have one example of a report. Maybe kind of technical but... So our company has a lot of cloud services. We run everything in AWS, heavily. And so I know anybody who has an AWS environment, cost is a big part of that. And controlling costs, and not having your DevOps team throwing hardware when you have software problems, or problems that actually exist in code. So we have a company, they have their own account, and I've kind of handed it off to the development team to kind of do their ops as well. And I got a report, and this report was probably one of the most valuable ones because they had decided to make changes, and they didn't communicate, right. So the biggest key is communication. The bill went from $2,000 to $10,000 in about 30 days. So it's a huge red flag and if no one had caught that, and that anomaly had not been brought up, the company would have paid $120,000 for an environment that they had no idea. So you know having that type of report to me is very important, if you're talking about a physical report.
I'll say my two favorite reports to receive are reports of we saved money, or reports of our information security vulnerability is low. [Laughing]
Okay so, now you've brought up sideways risk. All right, so risk reporting is something that the business really does care about. And I think maybe in IT we think it's always unfortunate that sort of security comes almost last. It's sort of monitoring second to last, and then security last. What are the types of security-related reports that you see from IT that gives you confidence that there's a reasonable assurance that nothing terrible is going to happen in the business? I mean start with that, right? That thing that keeps the CIO awake at night and not able to turn in.
It's usually the vulnerability-- basically, our Vat Scan results. And it's a vulnerability assessment tool that provides a result that says how we compare to other systems in our infrastructure and other companies in our industry, with regards to how many vulnerabilities we either haven't patched or we're currently exposed to. And so when I see our scores-- they actually do score them with the A to F-- We want to see our scores, as great passing grades. I can sleep a little better at night, because there's always a new vulnerability. And we have to regularly assess our environment against those things. And so that's the one thing that I do actually, absolutely read.
So along with that report, do you really show the CIO's office something special if you marry that along with a report that's actually like a service level objective? Like, not only what is the risk but how effectively can we recover in the event that something unknown or a new issue pops up?
Absolutely. So what helps me sleep well at night-- if any CIOs sleep well at night-- is knowing that I've got...
Maybe that's the ultimate metric.
That really is it. [Laughing] That's what we measure our, kind of, our general life success with is, "Can we sleep?" And are we really serving the company well. I'd like to know that our patching is current, our backups are current, and that our 'vulnerability assessment tool' scores are within a comfortable risk range. And so when you marry that with, "Hey, we've met all of our service level objectives, and for the most part, our security throughout that risk is low"-- I can at least put in a few hours at night.
I think the only part for me is making sure that we have the capacity, that we have the headroom for the services that we are delivering, that I know that we can recover. I know we have backups; I know we have that. But am I sure that, for the next quarter, I can take on the next 100 customers that we are bringing on. And do we have the plan in place to actually get all of that. You know it's that. And for that I need to-- I want to know about the metrics about utilization, right-- about the database level, about the CPU-- you know, about all of those server metrics-- I need to know, at least at the high level, where we are at. Are we at 80% or 90%, 95% of database? Do we see transaction time spiking up over time? If we do, then what are we doing about that? It's really about that also.
And I think that I'm completely on the opposite end where you guys have all these great tools and sometimes IT departments don't have these tools. So it makes us proactive-- not proactive, reactive. I would love to have a report like that that came to me and said, "Hey, my backups are 95% and we would be able to restore in this amount of time." Where I don't really have that luxury in my area, which I wish that I did. So having that tool that you both explain would be wonderful for my world. But we're so small and it's just reactive, you know. I get a phone call that there's a website down or I have a monitoring tool that's giving me maybe an email that we jump in and we start to dive into where I don't get the statistics first, because we're so small; we don't have time.
That make sense. And I think that's why you have to, kind of, tailor your report to the needs and the size and the maturity of your business. And so maybe it's not something that actually a tool gave you this particular result, but that your team has done this body of work so that you have a lower risk.
Yeah, I agree 100%.
So going back to sort of 'nerfing fail fast.' So you're saying that it's actually easier to reassure the CIO with a good mitigation and recovery report, than saying I have eight nines or a million nines of availability, and just them knowing that that's impossible?
For me, the way that we communicate is not so much a report that's telling us that we're meeting our sale leads, or giving me statics on servers, or being able to prep for onboarding a hundred customers, and making sure that I have enough infrastructure in place to handle it. We're a small IT department, and my direct relation is with my CIO. And I walk in his office, and we basically have conversations once a week, if not almost every day, on where I see risk, and things that I know that are risk, and I kind of say that in a verbal way. And we have a list of things we know in our head that are projects that need to be done. But the business has other priorities that come before those sometimes. And you're not given the resources. Not that they don't want to give you the resources. They just really don't understand what's going on on the back end. And things that really have to happen from security. I mean security is one of the hugest ones, one that will come and get you if you don't try to mediate before. And so our department works a little bit differently. I have worked for a retail company that was very good with the vulnerability reports. They were under compliance and so they were required to do a lot of that. And when you're under compliance, I think it forces you to have those reports and SLAs and more statistical planning than what we have. But I see both sides. I see both sides.
So basically you're saying one of the best ways of being noticed by leadership-- and noticed in a good way instead of the way that sometimes IT gets noticed-- is to have a 'it's okay I've got this report?'
That's the secret to getting promoted.
I don't know if it's just the report. I think it's also your capability of reacting quickly, and solving the problem, and getting back your customer in shape very quickly. I think that's for me, at Kinnser, that's what my CEO and the rest of the executive team really care about. They know that the system is not perfect. But what they'll count on is the fact that we have a problem, we react, we go back to work.
It builds trust too. I mean they have to trust the IT department is doing everything that they possibly can to make sure the business is going to run fine.
Okay, so this is that tough part with the panel, where I get to be dad and look at my watch and say we're about to run out of time. [Groaning] Let's do this. Let's have each of you-- when I ask a question, let each of you take a chance to kind of think about this. And the idea here is I want you to sum up your thoughts on how the IT and the CIO's office can really improve communication. And especially to remind the business how indispensable IT truly is. So considering everything that you know now-- what you've learned over the years, what has worked and what hasn't worked-- if you could go back in time to a moment early in your IT career, where you were successful but you knew there was an opportunity to really accelerate your career, what would you tell yourself? What advice would you give the younger version of you on how to improve communication and even manage up?
Wow, that's a great one. So I'm an introvert. I am an extreme introvert, and I kind of grew up with the philosophy in my head that IT should just be seen and should work, but not really be heard. And so I shied away from opportunities to meet with our leadership and to share with them what we were focused on and how we were doing. And again just kind of shadow in the background, it just works. And so I would tell myself to take that opportunity to meet with leadership, to share the good work that our team has done. Because what I learned when I did start doing it is that it gave me an opportunity to better align our priorities and our resources with their objectives, and actually end up serving the company better. So I tell myself to take those opportunities to highlight and celebrate the great work that our teams are doing in IT. Because there's no parades that are running in the background when IT does their job. We're kind of like, unfortunately, like plumbers.
It's a utility.
It's just supposed to work. And getting those opportunities in front of the leadership to share the great work that our team has done. It's really rewarding for all of us. And so that's what I would say--take those meetings.
I think for me it goes along those lines. It's about, thinking about, making the step to really understand who's on the other side. That you may be a great technologist, but technology doesn't happen just by itself. It has to have a purpose-- being applied and solving problems that people are experiencing. And the closer and the sooner you can actually close that gap, and understand how it can make that person happier, in day to day life--better, because you apply what you know-- then you're going to be that much more powerful. And you're going to be much more successful, faster-- versus living in your own bubble, being the greatest technologist of that tiny piece of technology that nobody really cares about from a business perspective-- very interesting-- but you're in your own bubble. And what really matters is for you to bring that bubble and really kind of reach out to the other people-- So I think empathy, and closing the gap there.
So, if I could go back and tell myself, I'd tell myself a lot of things at that age. But from a career perspective, like we spoke before, I started as a field technician, working in a warehouse fixing computers. And one thing I would probably tell myself is to start networking more-- networking up--because I think you naturally progress from that to your love, which is you start working into networking. You get projects. You get involved. You start communicating. So, just networking with your peers-- people in your department-- I think starts bringing you into projects when you start to bring things up that you're seeing, and I think I would tell myself 'network.'
It's interesting. The words that we end up using. Network, trust, empathy. It really, it seems to be most about finding ways and creating, not just ways of communicating, but actually some reports on paper, that allows your management-- that uses the same language which the senior leadership expects to see-- to trust us to do the right thing. To trust us to deliver on time. To trust us when we say that's too much-- that's too many requirements for this sprint. Let's deliver this on time and then add that to the next one. Well, this has really been fantastic to have all of you with us today. And I hope all of you got a lot out of this and a lot of ideas about how you can communicate with your senior leadership better-- to get your really fantastic ITs to transform, and really push your departments forward. Thanks again so much Mindy Marks, Joel Dolisy and Rani Johnson. It's been great to have you here. I kind of feel like we're going to have to do a podcast. I think all the cool kids are beginning you to do that. You can let us know in chat if we should do that or not. So, thanks again. For SolarWinds, I'm Patrick Hubbard. I can't wait to see you on chat and enjoy the rest of THWACKcamp.