cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Tell Us How You Manage Cyber Monday Insanity & Get 250 THWACK Points!

Level 15

The ghosts and ghouls of Halloween are behind us, and the beginning of November means that Thanksgiving is fast approaching. As the leaves continue to change and the holiday season ramps up, shopping is at the top of everyone’s to-do list. And what better way to take advantage of the holiday sales than from the comfort of your own home on Cyber Monday?

The rise in online activity stemming from these sales, however, will certainly create costly bottlenecks and downtime, and an increased need for optimized application performance. How can we best handle and manage this increase in demand and activity to provide end users with the best holiday shopping experience?

We want to hear your post-Thanksgiving advice for peers and colleagues managing the Cyber Monday madness. Help us all prepare this year by sharing your best practices for application performance management and maintaining uptime by Thursday, November 16, and we’ll give you 250 THWACK points in exchange.

284 Comments

Wendy, you make me worry I'm not worried enough.

;^)

Frankly, I've spent the last year getting an even more solid base for our network, adding VNQM, HA, IPAM, and UDT to our environment, along with more pollers that can handle another 60,000 elements.

Keeping a sharp awareness of our firewall performance with the new Net Insight that shows useless or shadowed ACLs is something new through Solarwinds NPM 12.2 and NCM 7.7, and my team is excited about having a tool that shows that to us.

Frankly, Cyber Monday isn't a factor for incoming traffic in my health care employment environment, and it's only a slightly increased amount of internal-to-Internet traffic as people shop from work.

The biggest impact I've seen this year came from moving tens of thousands of Exchange accounts (along with all their archived e-mails) from internal servers to the Cloud.  That caused our firewalls to experience internal self-created Denials of Service as those huge files exceeded the IPS inspection capacities of HA 5555 ASA's.  As a result, customer and internal data transfer between internal and DMZ resources was significantly slowed until the e-mail migration were postponed to off hours, and IPS inspection was removed from those specific transfers.

The second biggest impact to performance came when Apple released an iPhone upgrade last year.  The Engineer's Toolset set to poll every ten seconds and display Historical Graphs clued me into the situation in real time, and I watched our Guest Internet traffic quadruple in bandwidth for nearly two days as people took advantage of our internal Public access WLAN.

Knowing what's coming has proven more important (so far) than worrying about what might happen, but we're working on the bad possibilities now as well.  In fact, we're preparing a mock Ransomeware exercise to show our C-Level and health providers how that kind of malware can impact their work lives and their ability to treat patients.  It's an exciting opportunity to explain to people what the risks are, while giving my Network Team and our InfoSec and SA's directives to plan to isolate our data centers for protection.  Having a worst case game plan ready, hoping we'll never use it, will give us confidence and a game plan to follow in case the worst might happen.

How would you layer the varying actions and opportunities your network team can or should (or should not!) take?

  • Shut down Internet pipes at the firewalls
  • Shut down all Public/Guest Internet access
  • Disable internal corporate SSID's by risk factor, given some are dedicated to medical equipment, others for general security, others for personal security products like Vocera and Pin-Point. 
  • Disabling SVI's on Distribution Switches, shutting down physical links between switch blocks and core routers that support data centers.
  • Shutting down regional and core WAN routers, and having the right plan in place to do things in the right order so we can recover communications intelligently and by documenting the right interfaces to shut first, and which ones to no-shut in the right order later.

These are some fun, and scary, things to think about and document.

So Cyber Monday isn't a worry point for my organization at all.  Maybe for Amazon or Google, but not so much for us.  We have bigger fish to fry.

This year we have really taken steps to improve our SAM monitoring. We now use a custom JMX template to get all the application statistics we need. That being said cyber Monday isn't really an event we notice application wise.

Level 13

Don't usually get Cyber Monday insanity. Get Cyber Monday, Tuesday, Wednesday, Thursday and Friday Insanity.

Level 8

We don't really have any ecommerce platform, so we don't notice

Well actually last year traffic was lower then average

Level 8

I am new and still learning, so right now I have all the worries in the world

Level 10

Our websites are mostly Business to Business, so fortunately did not see much impact of Cyber Monday etc.

MVP
MVP

Cyber Monday doesn't really happen in our firm as we do have flexi working ways, but yes if its the shopping sites and online sales that you are talking about, these india online sites always open big offers or discounts in odd hours where most of them are off to bed and the remaining are sufficient to close down the stocks

MVP
MVP

Being a rural hospital we get our bandwidth through the New England Telehealth Consortium so the pricing is deeply cut below "retail." That gives us the ability to have a Gig to the internet. So far shopping, browsing, streaming hasn't been a real impact to us. If we find it to be an issue our first step would simply be to limit the amount of traffic allowed for such things at the firewall leading to the internet.

Level 10

What Cyber Monday insanity?

Not an e-commerce business, hence traffic remain relatively the same, business as usual.

Level 13

Sorry I should have answered this correctly. Our company is business services so we don't tend to get hit by Cyber Monday. We do have a customer that does do e-commerce but I don't tend to get involved too much with that customer. I just have to contend with insanity in general - a hypnotherapist has been suggested to me that I might give a try.

Level 10

We tend to notice a good uptick in shopping website traffic, but with 1Gig to the internet we aren't in any trouble.

Level 10

Our firewalls block users from doing non-business related shopping, so no worries concerning outbound traffic.  We don't have an online store, so no worries concerning inbound traffic.  Cyber Monday is just another day for us.

Level 8

Working in retail, IT during holiday is very conservative on changes here.  That being said we also recently just added all or our stores' registers to Orion and have a much more in depth look at performance and application issues.  Our Ops team actually uses it more often now which is a plus as well.

Level 16

I used to work at a retailer that had both brick and mortar and online sales so had to get ready for both Black Friday and Cyber Monday. They also sold food so the entire week leading up to Thanksgiving was a critical time for the online systems.

Change moratoriums and much scrutiny was put into any work that needed to be done. We would beef up the monitoring by running our 'robots' at a higher frequency as well as had more eyes dedicated to watching all the systems.

There were rarely any issues caused internally by the systems but they did get burned one year by some external systems that they relied on to do tax calculations for online sales. That was a capacity issue with the external vendor that was eventually fixed.

Fortunately for my company we don't have a large internet presence. Being a wine & spirits distributor we do not do direct sales. However, like most companies we rely heavily on internet for business operations. This year my security engineer and I took on the project of wrangling all 30+ firewalls across our enterprise to better position IT for days like Cyber Monday, opening rd. of the NCAA, the Olympics, etc. We have seen instances where users, even on days not listed prior, have sucked up huge loads of bandwidth streaming.

We are now near the end of our project and we have consistency across all of our firewalls. Rules are in place to block known offenders. Monitoring and alerting is set so when certain threshold are exceeded which allows us to be proactive. Regular meetings with HR allows them to take action on negative trending for surfing activity on non-work related websites.

Given that we don't have a NOC and we don't have anyone dedicated to firewalls this appears to be a workable solution. Our internet usage is WAY down while we've instituted very few rules (HR has been our hammer!). We read reports everyday and address any anomalies. As we go through our busy season with this implemented it will be curious to see what we learn.

Level 13

Being a draconian IT manager, we block all shopping sites from everyone. Only a select few are allowed to visit shopping sites based on their job functions (i.e. procurement dept.). We do have a guest wireless network that allows shopping and social media traffic, but we've throttled down the bandwidth to 1 mb / s.

The only insanity I face on cyber Monday are the complaints that "but I can't buy that great <insert product here>!" - to which I reply, "you should be working. We don't pay you to shop."

Level 12

Every day is Cyber Monday (or so it seems.). 

Seriously though, we typically start with a non-technical solution.  Education!. We have informal meetings with various Departments to provide details of the impact and risk to our Network, during this "Holiday" as well as a focus on the upcoming Christmas season. We use this as a time to also address the risks of increased malware, virus, Social Engineering and Non-Technical scam's that come with the season.

We of course also harden our Firewall and remain vigilant with our NPM etc...

It actually is a perfect opportunity to engage Staff about their impact and responsibilities to their Employer and to fellow staff. 

cyber monkeys.jpg

Level 20

In preparation for cyber Monday and the holiday season we upgrading our Orion to NPM 12.2 and related modules on all new W2K12R2 and SQL2016 on newly built brand new VM's!  We've also moved our VM's off old Dell blade centers to all Cisco UCS!  Both of these things are great but finally updating to the latest Orion modules will be the best.  We'll have all the new features we've been missing out on.  My biggest fear is that our pollers might not handle what they handle today!  I sure hope that doesn't happen.

Level 9

Extra coffee. It's about all I can do.

Level 8

While we don't host an e-commerce system, we have been affected adversely by cyber Monday in the past thanks to some "slight" misuse of company resources.  We found the best way to handle it is with our web filtering solution.  We don't block shopping sites entirely, but there is a 15 minute per day quota that covers access to shopping, sports and certain other non-business related sites.  Our users can still find a great deal when they're on break, but work is still able to get accomplished with no competition for resources.  We have also encouraged more and more personnel to work remote so they can shop on their personal PC and keep the company PC connected to only business critical sites.

**EDIT**

We also have our BGP set to prefer certain providers over others that have caused us problems in the past.  So unless a destination site is hosted on that provider's network, we're doing everything we can to bypass ASN's like the one that took down a good portion of the internet very recently.

Level 8

We have our end users on a cable modem so they don't interfere with business network traffic. If they flood the pipe it only affects their own shopping.

Level 10

Change freezes. Big change freezes.

Everyone ignores them, but still. *grinds teeth*

Level 13

Cyber Monday Insanity!!!!! How about Cyber everyday Insanity.  Cyber Monday is just another day from hell.

Level 9

Cyber Monday is actually a slow day for us so we don't have to deal with bottlenecks that day.

Monitoring our Internet traffic is always important. CyberMonday isn't a big deal for us. Not as much as March Madness - Iowans are crazy about college sports. We watch the traffic using both SolarWinds NTA and the Dashboards on our Palo Alto Networks Panorama - firewall management platform. We typically contact the member and then escalate to their manager if there is chronic abuse of the Internet. Being a World Wide Architectural, Engineering, and Design company we get a lot of outbound traffic with myriad types. So it is hard to justify a lot of the outbound initiated connections. Many of our small offices are in rural areas so Internet shopping is how all the urban transplants survive.

In bound traffic is tightly controlled and monitored. So no worries there.

Level 10

I've never worked at any kind of E-Commerce site that might be impacted by Cyber Monday, so I'll just say how I deal with it.

Wirecutter and Slickdeals are good sites to see what good deals are popping up.   You can kind of watch both all day.  Otherwise I'll watch the timed deals on Amazon especially if there's something I'm interested in that might be limited quantities.

Amazon is like a "normal" retailer so they'll sometimes raise prices to show a larger "sale" price that might actually be higher than usual (I'm looking at you, Kohls, as the leader in this space).  You can use Traktor to see if the price is actually good or not.

Steam will probably have good sales, but they frequently do.  Just add whatever you're interested in and get alerts on low prices on games (or apps, I'm watching some Stardock stuff right now) you're watching.

I'm honestly not much of a shopper, so it's not a huge shopping day for me, but I love seeing what all is out there and on sale.

Wirecutter | A New York Times Company

Slickdeals: The Best Deals, Coupons, Promo Codes & Discounts

The Tracktor - Amazon Price Tracker

http://store.steampowered.com

Level 8

Being and educational institution we monitor our bandwidth and use policies that if we implement manually if our connection is maxed out.  We implement polices to block non educational sites until our connection is below 90% utilization.

Level 9

Well wabbott​, I suggest planning how much you want to limit your spending to. Then set your alarm for an extra 30min to get those early Emails and get the early online specials because they always run out of the best deals.

Bonus points for buying something for yourself that you can re-gift at Christmas a month later.

Level 15

lol well that's one way to prepare!

Level 9

We use the SAM to find those applications that end-users usually have to notify us of problems on. Now we're ahead of the alerts.

Level 9

Our strategy is to look at last year's performance logs and see if there was anything needing upgrading.

We can easily do this with Solarwinds NPM and NCM.

We have all the data for the last year and more.

We wondered if our 10G links were maxed out and fortunately no so our core network is good.

We recently bought a Dell Hyperconverged solution and started managing it with Solarwinds as well.

We can see the storage utilization as well as memory and CPU util as well.

Everything looks good and we event tested our failover solution just in case.

We are ready for Thanksgiving!

Level 10

We understand that Cyber Monday is going to happen; regardless of device users access it with. Smartphones, Laptops, you name it they will find a way to watch the timer tick down to Zero before they break the point-and-click device they have in use. Our policy is simply "Do not let indulgences interfere with work, if you cannot resist, then take the day off (with Manager approval of course)".

Level 9

Cyber Monday is for amateurs. Every seasoned geek can make any day a Cyber-Mon-Day.

In any case, we'll spend ours on building a brand new Solarwinds environment and porting the data over from the Dev environment (that weirdly enough became our Prod somewhere along the way). And then adding additional pollers in DMZ & AWS.

Should be done with NPM migration before the turkey is done and the rest (VIM, NTA, DPA) will follow right after the Monday craze - most of Monday is dedicated to slowly and peacefully digesting Thanksgiving dinner, consuming leftovers dipped in gravy and lazily watching web stores selling all these "last year" gadgets at a ludicrous rate.

Level 9

One strategy is to implement a busy season freeze.   Only security related changes in order to ensure availability isn't impacted by unanticipated code changes that can occur even with a dev, test, uat and prod planned environment.

Level 12

Some issues are best solved through chemistry, such as copious amounts of the molecule CH3-CH2-OH.

Level 9

All heavy traffic days are handled by automation based on a combination of triggers from NPM and SAM. Thresholds specified through automated templates would trigger web/app/db tiers to pin to hot tiered storage until thresholds fell to acceptable levels for sustained periods of time. Once the lower thresholds were met the tiers would level themselves back out. All of this made possible via NPM and SAM as well as the Orion syslog.

Level 7

Our students are gone by then, so we have all of the bandwidth of the university for Cyber Monday. Malware Tuesday is what blows.  Everyone gets click happy and happily installs loads of Malware.

Level 8

Wendy,

Working in the financial sector, our company really doesn't have any insanity (other than the normal insanity) due to Cyber Monday. We have restricted some site for many years and only some managers other than the IT Dept. has full internet access.So for us Cyber Monday is a non-event.

Level 9

We aren't an eCommerce group and for employees hogging bandwidth, well they just have to deal with no network for the day . Really haven't ever had issues.

Level 14

We've got an unusual way of coping with Thanksgiving in the UK. We don't have a Thanksgiving.

However, we have had a Black Friday for the last few years and the Boxing Day sales have been going for ever and are really something else. I've worked with on-line betting companies where the Grand National horse race was the one huge event of the year and I've worked with a few other clients that have one mega-day in the year.

It's all down to preparation, the platforms must scale, the bandwidth must be more than adequate, all the application monitoring must have been proven. Test it and test it again, do it at expected load, then try to break it with load. I had one system up to 400% of anticipated load without problem & then it hit 350% on the day.

And then you get the pizzas in and wait - - -

Level 12

Pizza's, yes mandatory!  🙂

Level 7

we don't have ecommerce, so no worries here

Level 9

We manufacture plastic irrigation tubing and recycled garbage bags.  Not a big Cyber Monday item.  Doesn't effect us that much.

Level 9

In our small company, most of our users are on the road, so they won't have time doing any shopping.  This helps our network's bandwidth.

One of the perks of being a small company.

Level 7

We're not offering discounts on Health Care plans on Cyber Monday, so we're not affected in that way.

We're probably more affected by employees shopping from our systems.

Level 12

Not worried about cyber Monday as the Internet connection is monitored and an email has been sent out to all users letting them know that we're watching all Internet activity.   We did catch a person watching Netflix (thanks to NTA and UDT).

Level 10

I am in Canada so it does not really apply.

Level 14

UK based, but this buy-a-thon has wandered across the globe, paydays before Christmas and all that still counts here.

Luckily, my bit of the company operates almost as a MSP, so we have 10Gbps internet pipes a plenty.

Level 11

what is cyber Monday? I am usually hopped up on coffee...

Level 9

Fortunately for us, our biggest issue with Black Friday/Cyber Monday is making our users do their shopping on their own time   We're filtering internet usage - we aren't completely blocking "shopping" sites, we have a splash screen before the requested page coming up to ask the user if this is actually work related.  We are mandated by management to "not inconvenience" our users (lol).  The splash screen helps largely because most users will double think going to the site...others will go regardless of what we do anyway.

Good luck to those delivering service to merchants.