Geek Speak

5 Posts authored by:

Most enterprises rely on infrastructure and applications in the cloud. Whether it’s SaaS services like Office 365, IaaS in AWS, PaaS in Azure, or analytics services in Google Cloud, organizations now rely on systems that do not reside on their infrastructure. Unfortunately, connectivity requirements are often overlooked when the decision is made to migrate services to the cloud. Cloud service providers downplay connectivity challenges, and organizations new to cloud computing don’t know the right questions to ask. 


SaaS: It’s just the Internet

When organizations begin to discuss cloud infrastructure, an early assumption is that all connectivity will simply happen via the internet. While many SaaS services are accessible from anywhere via the internet, large organizations need to consider how new traffic patterns will affect their current infrastructure. For example, Office 365 recommends you plan for 10 TCP port connections per device. You can support, at most, 6,000 devices behind a single IP address. If you have a large network and a small PAT pool for client egress, PAT exhaustion will quickly become a problem.


Internet-based SaaS applications make hub-and-spoke networks with centralized internet less efficient. Many WAN solutions use local internet connections to build encrypted tunnels to other sites. You can dramatically reduce network traffic by offloading SaaS applications to a local internet connection instead of backhauling traffic to a centralized data center. However, be mindful of the impact of your security footprint as you decentralize internet access across your organization.


But What About the Data Center?

Invariably, as teams begin to build IaaS and PaaS infrastructure in the cloud, they need access to resources and data that live in an on-premises data center. Most organizations begin with IPSec tunnels to connect disparate resources. Care must be taken when building IPSec tunnels to understand cloud requirements. Many cloud teams assume dynamic routing with BGP over VPN tunnels. In my experience, most network engineers assume static routing over IPSec tunnels. Be sure to have conversations about requirements up front.


When building VPNs to the cloud, throughput can be an issue. Most VPN connections are built on underlying infrastructure with throughput limitations. If you need higher throughput than cloud VPN infrastructure will support, you will need to consider a direct connection to the cloud.


Plug Me In to the Cloud, Please

There are several options to connect directly to the cloud. If you have an existing MPLS provider, most offer services to provide direct connectivity into cloud services. There are technical limitations to these services, however. Pay special attention to your routing and segmentation requirements. MPLS connectivity will likely not be as simple as your provider describes in the sales meeting.


If you do not want to leverage MPLS service to connect to the cloud, you can provision a point-to-point circuit from your premises to a cloud service provider. Cloud services publish ample documentation for direct connections.


Another option is to lease space from a co-located provider who can peer with multiple cloud service providers (CSPs). You provide circuits and hardware that reside in the co-lo, and the co-lo provides peering services to the one or more cloud providers. Be aware that each CSP charges a direct connect fee on top of your circuit costs. There may also be data ingress and egress fees.


You Want to Route What on my Network?

Cloud service providers operate their networks with technologies similar to service providers. Many SaaS services are routable only with public IP addresses. For example, if you want to connect to SalesForce, Office365, or Azure Platform Services, you will need to route their public IP addresses on your internal network to force traffic across direct connect circuits. Network engineers who have always routed internet-facing traffic with a default route injected into their IGP will have to rethink their routing design to get full use of direct connectivity into the cloud.


I Thought the Cloud was Simple

The prevailing cloud messaging tells us that the cloud makes infrastructure simpler. There is some truth in this view from a developer’s perspective. However, for the network engineer, the cloud brings new connectivity challenges and forces us to think differently about how to engineer traffic through our networks. As you look to integrate cloud services into your on-premises data center, read up on the documentation from your cloud service provider and brush up on BGP. These tools will position you to address whatever challenges the cloud throws your way.

IT organizations manage security in different ways. Some companies have formalized security teams with board-level interest. In these companies, the security team will have firm policies and procedures that apply to network gear. Some organizations appoint a manager or director to be responsible for security with less high-level accountability. Smaller IT shops have less formal security organizations with little security-related accountability. The security guidance a network engineer receives from within their IT organization can vary widely across the industry. Regardless of the direction a network engineer receives from internal security teams, there are reasonable steps he or she can take to protect and secure the network.


Focus on the Basics


Many failures in network security happen due to a lack of basic security hygiene. While this problem extends up the entire IT stack, there are basic steps every network engineer should follow. Network gear should have consistent templated configuration across your organization. Ad-hoc configurations, varying password schemes, and a disorganized infrastructure opens the door for mistakes, inconsistencies, and vulnerabilities. A well-organized, rigorously implemented network is much more likely to be a secure network.


As part of the standard configuration for your network, pay special attention to default passwords, SNMP strings, and unencrypted access methods. Many devices ship with standard SNMP public and private communities. Change these immediately. Turn off any unencrypted access methods like telnet or unsecure web (http). If your organization doesn't have a corporate password vault system, use a free password vault like KeePass to store enable passwords and other sensitive access information. Don't leave a password list lying around, stored on Sharepoint, or unencrypted on a file share. Encrypt the disk on any computer that stores network configurations, especially engineer laptops which can be stolen or left accidentally.


To Firewall or Not to Firewall


While many hyperscalers don't use firewalls to protect their services, the average enterprise still uses firewalls for traffic flowing through their corporate network. It's important to move beyond the legacy layer 4 firewall to a next-generation, application-aware firewall. For outbound internet traffic, organizations need to build policy based on more than the 5-tuple. Building policies based on username and application will make the security posture more dynamic without compromising functionality.


Beyond the firewall, middle boxes like load balancers and reverse-proxies have an important role in your network infrastructure. Vulnerabilities, weak ciphers, and misconfigurations can leave applications and services wide open for exploit. There are many free web-based tools that can scan internet-facing hosts and report on weak ciphers and easy-to-spot vulnerabilities. Make use of these tools and then plan to remediate the findings.


Keep A Look Out for Vulnerabilities


When we think of patch cycles and vulnerability management, servers and workstations are top of mind. However, vulnerabilities exist in our networking gear too. Most vendors have mailing lists, blogs, and social media feeds where they post vulnerabilities. Subscribe to the relevant notification streams and tune your feed for information that's relevant to your organization. Make note of vulnerabilities and plan upgrades accordingly.


IT security is a broad topic that must be addressed throughout the entire stack. Most network engineers can't control the security posture of the endpoints or servers at their company but they do control networking gear and middle boxes which have a profound impact on IT security. In most instances, you can take practical, common sense steps that will dramatically improve your network security posture.

Most network engineers enter the profession because we enjoy fixing things.  We like to understand how technology works.  We thrive when digging into a subject matter with focus and intensity.  We love protocols, acronyms, features, and esoteric details that make sense only to a small group of peers.  Within the ranks of fellow geeks, our technical vernacular becomes badge of honor.  However, outside of our technical peers, we struggle to communicate effectively.


Our organizations rely on technology to make the business run.  But the deeper we get technically, the wider the communication gap between with IT and business leadership becomes.  We must learn to bridge the gap between technology and business to deliver the right solutions.


I was reminded of this communication disparity when working on a circuit outage recently.  While combing through logs and reviewing interface statics, a senior director asked for a status update.  I told him, “We lost BGP on our Internet circuits".  He responded with a blank stare.  I had failed to shift my communication style and provided zero helpful information to my leadership.  I changed my approach and summarized that we lost the logical peering with our provider.  Although the physical circuit appeared to be up, we could not send Internet traffic because we were no longer receiving routing information from them.  My second response, though less precise, provided an understandable picture to my senior leadership and satisfied his question.  He had more confidence that I knew where the problem was and it helped him understand what the escalation point should be.


When communicating with leadership about technical projects and problems remember these things.


  1. Leadership doesn’t understand your jargon and they don’t need to.  I’ve heard many network engineers decry the intelligence of their leadership because management doesn't know the difference between an ARP table and a MAC address table.  This line of thinking is silly.  Management’s role is to understand the business, build a healthy organization, manage teams effectively, and provide resources to accomplish business goals.  Some degree of technical knowledge is helpful for front-line management, but the higher in the organization an individual is, the less detail they will need know about each technical arena.  This is as it should be.  It’s your job to know the difference between an ARP table and a MAC address table and to summarize technical detail into actionable information.
  2. Management doesn’t always know the exact right question to ask.  I once had a manager describe a colleague as an individual who would provide only data, never analysis.  My boss felt as though he had to ask 30 questions to get a handle on the technical situation.  My colleague thought his sole responsibility was to  answer the question precisely as asked — regardless of the value of that answer.  Don’t be that guy or gal.  Listen carefully and try to understand what you manager wants to know instead of parsing their words precisely.  Answer their question, then offer insight that you believe will help them do their job better.  Be brief, summarize, don’t include so much technical detail that they check out before you get to the punchline.
  3. Effective communication is an art, more than a science.  At the end of the day, great communication happens in the context of strong professional relationships.  You don’t have to be best friends with your manager and you don’t need to spend time with them outside of the office.  However, you should work hard — as much as it depends on you — to build trust and direct, respectful communication channels with your leadership.  Don’t dig in your heels unnecessarily.  Give when you can and hold firm when you must.  If you develop a reputation as a team player, your objections will be taken more seriously when you must voice them.


Strong communication skills are the secret weapon of truly effective network engineers.   If you want to grow in influence within your organization, and you want to truly affect change, you’ll need to sharpen your soft skills along with your technical chops.

If you work in an organization that is publicly traded, or you are subject to other government regulations, you will be required to perform a periodic network audit. Even if an external entity doesn’t require an audit, it's a good idea to review your network to ensure your environment meets current standards.


In a large network, a full audit may seem like an overwhelming task, but a few regular practices can help at audit time. As with most things, a successful audit begins with planning. The overall health and manageability of your network will dramatically reduce your audit and remediation efforts.


Let’s get a few basics out of the way. You will need an up-to-date inventory of all the devices on your network. There are many ways to maintain this inventory, ranging from a low-tech spreadsheet to a fully automated inventory tool. Regardless of the solution you use, you should track details like hostname, management IP, serial number, and code version. You should keep basic configuration standards and templates to which your device configurations can be compared. Proper standards, correctly defined and applied, will enforce a minimum standard and will help you stay compliant throughout the year.


As you consider your networking standards, appropriate logging is a must. You will need a centralized logging server which will collect syslog from your devices. There are many open source and commercial logging options. Minimally, you will need to log invalid login attempts, hardware failures, and adjacency changes. Firewalls should log ACL denies and other security events. As you build a logging infrastructure, you can add tools to parse logs and alert on relevant events. You will need to carefully control who has write access to your log data. Check internal and external adit standards for retention requirements. Many organizations need to save audit logs for 7 years or more.


In recent years, wireless networks have become a sticking point in network audits. As compared to wired networks, wireless infrastructure has been changing rapidly, the number of devices have proliferated, and vulnerabilities for legacy wireless abound. Many organizations implemented wireless quickly with a pre-shared key without considering the security or audit consequences. In order to meet current wireless security audit requirements, your wireless network will need to authenticate via 802.1X with user credentials or a unique certificate managed by PKI. You will need to log all access to the wireless network.  In addition, some standards may require wireless technology to perform roque detection and mitigation.


Its important to remember the purpose of a network audit. Typically, an audit measures how well you comply with established controls. Even if your network is, in your estimation, well run, secure, and well documented, you can fail an audit if you do not comply with established policies and procedures. If you have responsibility for ensuring your network audit, get clear guidance on the audit standards and review any policies and procedures that apply.


Network audits can be challenging and labor intensive. However, with careful planning, the right tools, and diligence over time, the task will become much simpler.

On modern enterprise networks, 100% uptime has become table stakes. Most organizations can no longer rely on a single circuit for internet connectivity. We look to carrier circuits for the redundancy and guaranteed uptime that our organizations need. When carrier outages occur, network engineers find themselves in a hot seat they can do little about. However, if we do our homework, we can improve our organizations' uptime by taking care as we provision connectivity.


Causes of carrier outages

Most network engineers have experienced the rapid-fire text messages and flurry of questions when the Internet stops working.  It’s important to understand the upstream causes of these outages so we can work with our carriers to mitigate them. The first, and most common, is the dreaded fiber cut. Regardless of the cause, a fiber cut in the wrong location can have widespread impacts. Second, an upstream provider issue can interrupt service. While less frequent than a fiber cut, these outages can be frustrating because, although your circuits and peerings are healthy, traffic does not flow properly. Third, DDoS attacks, whether directly targeting your organization or another customer on your provider’s network, can have a crippling impact on service availability. 


Managing Around a Fiber Cut

A few different approaches can help mitigate the impacts of a fiber cut.  Your organization can purchase circuit diversity from a single carrier. In this scenario, your carrier will engineer multiple circuits into your facility.  As part of this service, they will evaluate the physical path each circuit follows and ensure the circuits do not ride the same cable or poles. For true diversity, you’ll need to be certain that circuits take different paths into your facility. And, if circuits terminate into a powered cabinet, you must verify the reliability of the power source for that gear. Ask lots of questions and hold your carrier accountable. Be certain that they are contractually obligated to provide diversity because there are penalties if they fail to do so.  Work with an engineer for your carrier; don’t take the sales rep’s word for it.  A single provider should have a complete view of the physical path for your circuits and be able to guarantee physical diversity.  Unfortunately, however, using single carrier puts you at a higher risk for an upstream configuration our routing failure with that provider.


The Multiple Carrier Route

Instead of ordering path diversity from a single carrier, you can order two circuits from different providers. This option reduces your reliance on a single carrier, but makes it more difficult to ensure full path diversity. You will need to talk to your carriers about sharing the physical path information for the circuits with you or with one another. You’ll still want to be certain the circuits enter the building via a different conduit and terminate into properly powered equipment. If you use different carriers, you will need to pay special attention to your BGP configuration to verify that the path in and out of your network is what you expect.


An Important Note about Grooming

Even if you do everything right — you validate proper path diversity when you order a circuit, you pay special attention to the entrances into your building, and you verify that all vendor equipment is properly powered — things can change. Carriers will periodically groom circuits to change the path they follow through their network.  An industrious provider engineer may see that a circuit follows a less-than-optimal path through their network and then diligently re-engineer it to be more efficient. You will not be notified when the grooming takes place; it will be transparent to you, the customer. The only way to prevent grooming is to communicate clearly with your carrier and ask that they mark circuits that have been carefully engineered for path diversity to prevent them from being groomed.


As with most topics is networking, there are many factors to consider and tradeoffs to be made when ordering connectivity for your organization. You cannot have complete control over carrier-provided connectivity, but you can be diligent throughout the process, communicate the challenges clearly with your leadership, and be clear with your service provider about your expectations and the level of service being provided.

Filter Blog

By date: By tag:

SolarWinds uses cookies on its websites to make your online experience easier and better. By using our website, you consent to our use of cookies. For more information on cookies, see our cookie policy.