Tools and Skills for Information Security Practitioners: Python Part 1

This series, Tools and Skills for Information Security Practitioners will touch on what I have discovered over the years that can take a career in security to the next level.

There are a ton of general skills that someone in security should have and there are a ton of other resources out there to help someone get into the security field. I want to help take someone to the next level. Going to the next level could mean a couple of things - first, it is my opinion that the best security people have found a niche or two within security that they really enjoy and second is to continue to hone the basic skills to be able to do in your sleep.

About me: see my previous blog post about how I got to where I am in general.

The reason I am starting this series is that I see way too many people wanting to get into security to be a pen tester or, to a lesser extent, to do forensics. While I will never begrudge a person for wanting to do something specific, many of these folks seem to think that pen testing is what the cool kids are doing and that is what they see in the media as being what security is...

However, if you just concentrate on what everyone wants to do, getting into or progressing in the security field is going to be difficult as the pen testing discipline becomes saturated.

Don't get me wrong, there are a bunch of great pen testers and pen testing is a core tenant of a secure organization and needs people to perform these tests.

But, there are probably dozens of other disciplines in security that are not as visible or as "cool" as pen testing, but are equally, if not, in my opinion, more important. A pen tester can tell an organization where their weaknesses are which may be the way a bad guy got in, but the pen tester can't tell you that there is a bad guy already in the network taking advantages of those weaknesses and stealing your data... And there are already a ton of pen testing resources out there.

I want to concentrate on what I feel are some areas that are not as discussed or tutorials provided for the new or junior security practitioner.

In this first post, I want to concentrate on Python from a security practitioner's stand point. This will absolutely not be an all-encompassing Python tutorial or step-by-step learning Python post. This will concentrate on skills, modules and techniques that I have found helpful in the security monitoring, response, threat intel and threat hunting disciplines.

I made a decision about a year ago, I made the move to Python 3 from Python 2 - although there is no true technical reason for using 3 instead of 2, version 3 tries to improve on the basic programatic features - such as making the print command a function, so version 3 uses print("Hello World") vs. version 2's print "Hello World". Python 2 has some modules (such as __future__) that can help in the transition or help run version 3 scripts in version 2. I try in my scripts that I make public to put in checks for 2 vs 3, but in some cases, if my scripts do not run - it's probably this...

I am not a Programmer or Developer

I have found a ton of uses of Python in what I do. Let me caveat all of this with I do not consider myself a programmer or developer, I create scripts to solve common tasks or to provide integrations with other systems or data. I AM NOT A DEVELOPER, so if you bitch about my code formatting, the efficiency of my code or anything that would make a developer a developer - I will ignore it. Although, constructive comments on making something more efficient, I may at least read the comment, I do hate slow scripts... You have been warned. :)

Same with the modules I talk about below - there are hundreds of Python modules in the community, which I absolutely love - and as I am trying to solve a problem, if I find a module that works and seems to be pretty robust, I will likely stick with it. Prime example - requests vs. urllib2 - urllib2 may be a bit more powerful than requests and you may see me use it from time to time, but for general web or API calls, I like requests a bit more, so that is what I use...

Technical Setup

• Python 3.x: to put it bluntly - make the move to Python 3, it has been out for years, is stable and is the future. Especially if you are just learning - learn on v3. I try to make my scripts backwards compatible, but I do not test my scripts in v2, so be aware if you are looking at my code or scripts
• iPython: I do a lot of my initial coding of a new script in iPython - I find it a heck of a lot more powerful and better for initial development and testing. The basic python shell works perfectly well, but I do not find it easy to use when doing hard core scripting before putting it into a .py file.
• Mac OS X vs. Linux vs. Windows: I have become a Mac guy (but don't call me an Apple fan boy - I still have an Android phone...), so most of my scripting is in OS X, but everything will work in Linux or Windows - there may just be some different setup steps - there are many resources that talk about running Python on Linux and Windows...
• Atom Editor: I have fallen in love with the atom.io editor that is put out by Github. There are a bunch of great packages that I use (base64, language-email, file-icons, etc). There are a ton of other good editors (sublime, CodeRunner) or IDEs (PyCharm), but if you use Github to store and release code, Atom has built-in support for pushing commits to Github - I absolutely love that and stopped even looking at other editors.

Reasons for the skill

• Security Monitoring: Creating scripts to interact with applications and services' APIs
• Security Operations: Automate common tasks
• Threat Intelligence: Develop scripts and interpretations for commonly used web or other services
• Threat Hunting: consolidating and analyzing large amounts of data
• Data Analysis: finding trends, creating visualizations, and interpreting indicators

Most used skills

• Moderate level of understanding of general Python scripting
• Loading and using modules
• Populating and using lists and dictionaries
• Required Modules
• requests
• json
• xmltodict
• re
• datetime
• hashlib
• os
• sys
• Optional Modules that can help a lot
• configparser
• pymongo
• argparse
• BeautifulSoup
• pandas
• numpy
• wget
• socket
• Cool modules that may or may not be used
• flask
• arrow

Basic usage of some of the modules

I'll touch on some of the usages of the modules and concepts noted above and give a couple of examples.

requests
The requests module is an alternative to urllib2 and for my purposes requests does just fine and is what I have become accustomed to. Documentation: http://docs.python-requests.org/en/master/ Basic Usage: Here, I will make a request to the site Unshorten.me, which takes shortened links and displays what the full URL is. Why don't I just make the request to the shortened link instead of pushing through http://unshorten.me? My IP address will not show in the shortening service's interface; thus, not alerting the bad guy of an IP that is potentially investigating the link...

import requests
s_url = 'http://bit.ly/2GmLHzg'
url = 'https://unshorten.me/json/'
r = requests.get(url + s_url)
print(r.json()['resolved_url'])

json
Many APIs or automated systems use JSON to return data (as you can see in the Requests section above). Some modules, such as requests, do have built-in recognition of JSON, so the additional json module is not necessary. However, if you want to do a bit more with the returned data or save it out to a file, the json module becomes a robust way to deal with JSON data.

Documentation: https://docs.python.org/3/library/json.html

Basic Usage: Below is a snippet of code used to post a message to a Hipchat server.

import requests
import json
host = "mysite.hipchat.com"
token = "ThisIsMyHipchatAPIToken"
room = "Threat Intel Notifications"
message = "Test from Python"
url = "https://{0}/v2/room/{1}/notification".format(host, room)
headers = {'Content-type': 'application/json'}
headers['Authorization'] = "Bearer " + token
payload = {
   'message': message,
   'notify': False,
   'message_format': 'text',
   'color': 'green'
}
r = requests.post(url, data=json.dumps(payload), headers=headers)

In this example, when you dig into the Hipchat API documentation, it will indicate that the post must be sent in JSON format; however, although JSON is, for all intents and purposes, a dictionary in Python, but you cannot send a Python dictionary across to Hipchat, it must be sent as a string. So, to send over a string that inside of it is a JSON object, you must utilize the json.dumps(payload) function in order for requests to be able to send it across.

xmltodict
I cannot stand XML, while XML may be perfectly fine for some applications or scripts, for some reason I cannot fully grasp handling XML in Python (if you have to handle XML in a more pure manner, the Element Tree module [https://docs.python.org/2/library/xml.etree.elementtree.html] is what has worked for me in the past).

So, instead, I will convert XML to a dictionary using xmltodict.

Again, I am more about getting a job done that being the 'leetest scripter...

Documentation:  https://github.com/martinblech/xmltodict

Basic Usage: Fortunately, most APIs are using JSON now, so I do not have a specific security related example right now, so here is something that I used xmltodict for with baseball data.

Until recently, very detailed MLB game data was available from http://gdx.mlb.com/components/game/mlb/ however, something happened recently and now there is a 'The specified key does not exist' error.

If you are wanting MLB data, Retrosheet is current the next best source, just not as detailed. Anyway, while some of the MLB data is available in JSON, most of the more detailed data is only in XML (for example, the individual pitch data in the inning_all.xml file), so needing to start with XML data.

import xmltodict
import request
game_url = "http://http://gdx.mlb.com/components/game/mlb/year_2017/month_09/day_11/gid_2017_09_11_colmlb_arimlb_1/inning/inning_all.xml"
r = requests.get(url)
data = r.text.encode('utf-8')
doc = xmltodict.parse(data)

The doc variable will be a dictionary of the values from the innings_all.xml file. Note: the xmltodict module does some unique things with the dictionary - any key that is a value and not another list or dictionary is prepended by an '@'. For example, within the doc variable, to access the batter's player id would be: doc['game']['inning'][0]['top']['atbat'][0]['@batter']

re
The re module is another highly used module - it can perform regular expression searches on variables. Documentation: https://docs.python.org/3/library/re.html Basic usage: Here we will search a list of log entries for a specific IP address in a list of generic traffic data.

import re
test_log = ["192.168.0.4:44352 -> 192.168.1.6:80", "192.168.0.7:34323 -> 192.168.1.7:443"]
for log in list:
    if re.search('192.168.0.7', log):
        print("Found 192.168.0.7")
    else:
        print("192.168.0.7 not in log")

Conclusion

That is about all the time I have for today - I will continue this series in the near future.

Please let me know what you think of the post and feel free to contact me at paul[@]ir4n6[.]com.

About Me and Why Do I Want to Start Yet Another Security Blog

Why am I starting yet another security blog? There are dozens and dozens of great blogs that I respect and visit on a regular basis. So I thought a lot about it and decided that while there are many great blogs out there - most of them are either extremely technical, which is too much for many just starting out in security, or the blogs are just too high level and don't really provide an explanation on a topic - it seems they are saying "here are the top 10 things that a security team should do" but they fail to actually explain how to implement these good practices.

I think that is the direction I will go with this blog - while I do work on things that are trying to being innovative and bring new concepts into security, this blog will be more about how to implement some of the key security concepts. At least that is what I am trying to do :)

Why me? Why do I think after not being very visible in the security community do I now decide to jump into the fray? That is a good question, let me try to explain.

I have been recently seeing more and more posts, tweets, articles, and presentations on how to get into the security field and at this stage in my career and life, I am wanting to give back - both in knowledge and mentorship. So here it is - how I got into security and some of the things that I think have helped me succeed in the field.

Key take aways for getting into and succeeding in the Information Security field:

• Don't just stick to your job description, be willing to do things that you have never done - just make sure you are still doing what the job dictates
• Always be willing to ask for help
• Create your own systems and network environments - and break them, then fix them, then break them again
• Learn on your own - you were likely hired for your current skill set, although you should be willing to step outside your box at work when everything else is done - you are definitely going to learn more without any of the guard rails that work would have
• If your job provides a training budget, take full advantage of it - go to classes, conferences, lectures, seminars, whatever
• Always bring solutions with you when presenting a problem
• Get out into the security community and meet people, network, share ideas, whatever - having connections who know you and what you can do is probably one of the biggest advantages you can have when looking for new positions.

A long, long time ago I joined the US Navy's Nuclear Power field, eventually becoming one of the chemists of the nuclear plant, maintaining the water chemistry to prevent failure of components and monitoring radiation levels to ensure we were not receiving to much exposure. Did that for 8 years, got out, moved back to the midwest and was going to college for chemistry and that would be that. So I got a job as a bartender and started looking around to schools - but a funny thing happened, I had gotten a computer when I got out of the Navy and I found that when I got home from bartending, I spent the next 6 to 8 hours playing around on the computer. I thought, I really should look to get paid for doing this...

So my first job in the computer field was as a delivery person for a small consulting company. Because of the company's size, there was a lot of other work to be done - building systems, maintaining the office network, troubleshooting systems and networks, and a number of other things that the small staff of techs could not keep up with - so I started chipping in troubleshooting, fixing, building, you name it. That was one of the first things I learned about the computing field - don't just stick inside your preassigned guard rails and job description - make sure you are still doing all of the appropriate tasks for what you got hired for, but don't be scared to dip your toe into deeper water and figure something new out.

Don't worry, this is not just going to be a rehash of my resume - but a couple of positions I have had do have a good story behind them.

Although one of the best things about consulting/external support positions is you get exposed to a lot of different networks, business processes, and strange issues to fix. But I started getting a bit antsy with bouncing from one client to another, so decided to move to a single company and maintain a single environment. This position was pretty routine, but I did learn one very valuable lesson - it just so happened that I was on-call the week I was to give my notice of leaving for another company. On the exact day I was going to give my notice, there appeared to be an issue with the Exchange server - at the time I had far more experience with Groupwise and Lotus Notes (don't laugh...) and those systems were far less temperamental if rebooted, so, you can see where this is going, I ended up rebooting the Exchange server as everyone was coming into the office. So what was the lesson - first, the easiest solution is most likely not even close to the best solution and second, do not be afraid to ask for help if you are in over your head. I was very junior at the time and the more senior folks were only a phone call away...

After that I started on a journey that, I believe, helped me exponentially expand my skills and find that true passion I wanted to have within security. Over the next decade or so I went through a decade or so at a handful of companies - each had their good and bad, each gave me a chance to build security programs and ultimately incident response programs, which has been one of those passions of mine. But, through all of that time, there was one constant, I did a lot of learning, troubleshooting, experimenting, playing around, and breaking things outside of work. Although my wife will disagree, it is far better to break the internet at your home than it is at the office. And break things I did... That is the next thing that I learned - if you really want to expand yourself in the security field, you have to have a home lab (now I guess that would be a few cloud servers) and do not be afraid to break things - you really do learn a heck of a lot more when you have to fix something that breaks than you do just setting something up.

There was one other constant through the first part of my career - I always performed in my position such that my peers and superiors knew they could count on me, trust my judgements, and listen to my recommendations. That does not come easy... Sometimes you just have to sit back and listen, sometimes even if you know the way something is done or architected is not the best or even right way - you have to learn there is a time and a place to question or comment on a solution. If you know someone is wrong, unless it is going to cause significant damage or lead to a compromise, never criticize or question anyone's ideas in a public setting. I am not saying that you cannot have a differing opinion and bring it up in a meeting or other public setting - but just be diplomatic about it - don't degrade the other person's idea and absolutely never think that your way is the only way.

You should never think you are the smartest person in the room - if you are, you really need to surround yourself with people who can challenge you and make you better.

But back to security specifically... if you are really passionate about security, really want to make a difference, even if that difference is just within your organization - you absolutely have to be willing to learn outside of work. You can take class after class, go to conference after conference on the company's dime, but trust me, you will only get better if you create opportunities outside of the office to learn. As I mentioned before, create a home lab or put virtual machines on your system, what ever - then go break them. Even if you could careless about actually troubleshooting and fixing things as a profession - you get a heck of a lot deeper into config files, log files, and learn much more about how something functions than by just viewing the web console.

Never bring just problems, always bring solutions. Ever since I became a manager, I have always told my people to bring me solutions, not problems. Trust me, I have been around the block a few times and while problems happen all the time, don't just come to me with the current problem, bring a solution with you as well. The people I have trusted the most and had done the best in their careers come up with solutions as well. Anyone can detect a problem, heck in today's Machine Learning/Artificial Intelligence buzzword security industry - even the machines can find the problems. Coming up and presenting solutions will get you points in pretty much anyone's book.

Don't feel guilty, if your company offers a training budget, use it! While you can see I am a huge proponent of learning on your own outside of work - there is no reason you cannot learn somethings while on the job as well. Most companies have a training budget set aside for continued improvements - use it to the fullest and perhaps take some course that you perhaps could not afford yourself or attend a conference.

Lastly, I would like to touch on certifications... I have had my share of certifications - my first being Certified Novell Administrator (CNA), then MCSE, CISSP, etc. To be completely honest, I do not know of a single job that I got because I had a certification - I got the job because I knew what I was doing and could provide the company with a distinct set of skills to help protect the company. Did it get me past the HR recruiter because I used to have a CISSP - perhaps, I honestly will never know... However, when I am looking for people to come work for me, I never judge a candidate on his or her certifications. In fact, if I see too many certifications, that tells me that you are too generalized and more learning what is needed to pass a certification test rather than really being good at security. Take that for what it is - I am just one director at one company who is not a huge fan of certifications; however, if my people want to get certifications, as long as I think they are not just taking the course and test for sake of getting the cert, I will support them 100%, because I know they have the background.

Oh, one more lastly, lastly - get out an meet people in the security industry. It is one thing to follow respected security people on Twitter, read their blogs, etc, but it is more to actually meet people in the industry. Go to security related events and groups. ISSA, ISACA, and OWASP are all national level organizations that have chapters in all major cities - even if you are a ways into your career and you think something like ISSA is too high level - do not let that keep you from attending the meetings and meeting people. Heck, if you think the presentations are too simple or high level - propose to speak at an upcoming meeting - just make sure you are presenting on a valuable topic and it will be of interest to the audience.

OK, that's about it. Honestly, if you made it this far, thank you. While it is very easy to write about yourself, knowing that someone read until the end is cool. As I said starting off, I am at a point in my career and life where I want to give back to more than just the company I work for and just to those who work for me and who I work for...

Thank you, I hope this helps someone and please do not hesitate to reach out to me at paul[@]ir4n6[.]com