Programming for the NOC: Building Utilities for Career Progression

One could be forgiven for believing that the Network Operations Center (NOC) is a boring place to work. Outsiders often imagine techs mindlessly staring at a bunch of screens looking for some kind of blip in the numbers. However, staying one step ahead of trouble isn't the snoozefest you might imagine.

In reality, something is always happening. There's an ever-evolving set of metrics and endpoints to analyze, and NOC itself needs to constantly adapt. Out of the box (so to speak) the NOC doesn't do it by itself (yet).

Tools of the NOC Trade

NOCs rely on a slew of utilities to sniff out abnormalities and take corrective action when trouble arises. The thing that makes the NOC tech's job interesting is that there's no one-size-fits-all NOC utility. Every organization has a unique configuration, so the NOC ends up relying on a mashup of utilities that don't cooperate with each other. Organizations choose from a slew of options, including:

• NetIQ Operations Center for monitoring applications, services, and SLA's

• ManageEngine to Keep Tabs on Windows Servers

• Wireshark to analyze network protocols and discover erroneous or unauthorized traffic

• SpiceWorks for ticketing, asset tracking, and the knowledgebase

• Solarwinds for server security scans and backups

• Cisco Prime to configure and troubleshoot network equipment

The NOC tech is perfectly positioned to identify the weak spots in network monitoring, diagnostics, and control. They are, after all, the closest point of contact with the data.

A resourceful NOC Tech may wish to subdue this mash of datastreams by automating the trivial stuff. That's a great idea and it just takes some training in a high-level programming language.

How to Get More From Your NOC Utilities

For a smoother running NOC, monitoring tools should be integrated with each other, as well as tie into your alert and ticketing systems. Unfortunately, grabbing a tool off the shelf isn't enough to accomplish this. Most NOC utils feature a GUI-based configuration, but this will only give you minimal and independent functionality.

Most of the software you'll find in a NOC comes with some kind of application program interface (API) or command line interface (CLI) designed to extend the product into your particular workflow. However, many NOCs lack the dedicated engineering talent needed to keep up with an ever-changing set of configuration and integration parameters. So, why not do it yourself?

There are several ways that a little programming can create a smoother running NOC. One of the best uses is to implement triggers. A trigger can automatically take a certain action based on monitored criteria. Starting and stopping servers, bringing additional capacity online, and rerouting network traffic no longer need to be manual operations.

It's also an easy task to write some simple scripts to automate log searches. While your NOC probably uses a GUI tool to generate reports on log files, these tools often lag a step behind the hackers. It's easy to identify a pattern common to the latest exploits and create a script to locate these patterns in near real-time.

In addition, most NOC software sports a GUI interface that can actually be customized to eliminate unused features or add your own tools, for instance on a context (right-click) menu. Before you decide on a particular piece of NOC software, ensure that it contains an API that can tie directly into the GUI.

Today's enterprise networks are, more often than not, based on a hybrid cloud. This can make network-wide operations difficult because of the interaction among differing architectures. Fortunately, cloud providers integrate their own API and CLI into their instances, which can be exploited to automate tasks and streamline administration across both local and remote networks.

Lastly, a bit of programming can integrate your monitoring systems with ticketing and alert software so that automated alarms can be created that know who to contact when certain events occur.

You may notice that the main idea here is automation of manual processes. The more scenarios we can spec and implement with automation, the more that NOC techs are free to dive into the details, discovering previously unknown insights, allowing the NOC to be more proactive.

Best Languages for NOC Programming

You don't need a master's in comp sci to write some simple utilities for the NOC. The best languages for NOC programming are fairly simple to learn and have a syntax similar to natural language.

Python takes top honors for its ease of learning and overall usefulness. As an example of Python's capability, the Azure CLI is written in Python! If you thought you might have to learn C# to integrate with your Windows cloud, you'll be happy to know that Python is a much simpler option. Because of Python's extensibility, it's likely that an open source library already exists for pretty much any application you can think of. Of all the general-purpose programming languages, Python does the best job of bridging the barrier between IT and Development, making it the perfect choice for creating custom NOC utilities.

JavaScript takes a close second for the simple fact that so many APIs are built around the JavaScript language. For instance, NetIQ can be customized and extended using their JavaScript-like NOCscript. Add ManageEngine to the JavaScript party as well, since it can be tailored using Microsoft's Jscript.

Lastly, no guide on IT programming would be complete without mentioning Perl. Although the overall popularity of Perl has waned in recent years, there are some applications for which there is no substitute. In the NOC, Perl is most useful for its syntax shortcuts that allow for incredibly powerful one-line commands. For example:

1                    perl –F: –Lane 'Print $F[0]' /Etc/Passwd

That command is a simple one-liner that could be used to scan for unauthorized user accounts on your Linux servers.

Scripting NOC Solutions with AWK

High level programming languages work great for creating custom NOC monitoring and management utilities. And also for deepening the connection between certain off-the-shelf tools and your particular configuration. In some cases, though, a simple script can be created to do the same. For this reason, it makes sense for the aspiring NOC Engineer to get familiar with scripting.

Your particular environment will dictate the overall scripting language that makes the most sense. It will most likely be PowerShell or Bash. The syntax between most script languages is more similar than different. As a result, learning any one of these scripting environments will make it easy to use another if the need arises.

This is good news because there are some specific scenarios that may lock you into a particular script language. For example, the LUA language is used by WireShark to create packet dissectors, but you're not likely to find applications for LUA elsewhere within IT. Another example is BeanShell, which is used by Cisco Prime to interact directly with Cisco devices.

Beyond these general purpose script languages, there are also purpose-built scripting systems, many of which make sense in the NOC. Probably the most useful of these is AWK, which is strictly designed for parsing text files. This is most useful in searching for particular patterns in log files as well as searching web site files for malware injections. For example, the AWK command:

1                    awk 'Length($0)>100' *.Php

That command will search web site files for the long strings typical of most malicious web shells.

While all of this may seem like it would require a lifetime of learning, the truth is that the knowledge required to come up to speed on one of these languages spills over into all the others, so that there is very little learning curve after you master the basics.

In fact, once you've got the basics down, you will find that there is a multitude of scripting communities that publish nearly ready-to-use scripts and programs for common IT needs. One of the most useful is the SpiceWorks script community, where users publish PowerShell scripts for just about any application you can imagine.

The Bottom Line

As you can see, by honing your programming and scripting skillsets, it's easy to take the initiative and optimize your NOC's off-the-shelf software for your particular needs. Even though your NOC's diverse range of monitoring and management utilities may not work well together by default, they can reach their full potential with some custom code.

With your unique insight and a bit of programming, a good NOC Tech can earn the respect of workmates and superiors, rapidly transitioning to a top-tier NOC Engineer.