Table of Contents

• Windows Text Editors
• Linux (Ubuntu) Text Editing Tools
  • Sed
  • Grep
  • Awk
  • vi
• linux, osx cron, windows schedule
• LDAP search, LDIF data format, LDP and Softerra Tools, and examples
• Understanding X.509 Certificates and asymetric cryptography
• Using OpenSSL to acquire PEM (Base 64) encoded pulic keys
• API calls, Keys, and tools for getting data out of web front-ended services
• JSON, XML,  data
• Networking
  • OSI
  • TCP packet analysis
  • Firewalls (G1-3, NG)
  • Layer 3 Routing, Routes, Routing protocols 
• Log data, SPL and Splunk
• SMTP  how to read email headers
• Statistical modeling basis of AI and machine learning

Intro

This is document is broken down into tools and skills for getting, manipulating, or analyzing data associated with security although there are a lot of just basic IT skills including networking, protocols, services data models, and of course tools.

There are a number of tools and skills that can help you get your job done when the vendor-provided tools just can't.  Applications are rushed to market and the latest and greatest tools generally with Web  UI's that are less than helpful operationally.  Many times the applications are good for working on individual units, but when you need to make wide-sweeping changes to data or just get data, the tools are not available.  I put skills above tools in the order of requirements.  Getting and manipulating data is required, but an understanding of the basic technology is a requirement to be able to "see" what is happening and which tools to apply to get either the fastest or the best results.   This is an attempt to catalog some of the skills and tools that I have found useful over the years.  Some of these will certainly fall to the wayside over time, but I have found them to be reliable and useful.

Windows Text Editing Tools

The tool of choice is not necessarily dependent on a specific vendor.  Use anyone you want for data manipulation, but here are the features that I find invaluable.  I use Notepad ++ for the following reasons in no particular order

• It is open-source and well-maintained by the community.
• It is 64-bit and works on large data sets.
• It just works and does not "enhance" your text with hidden metadata.
• It has a feature called column mode edit that allows you to separate data or embed data vertically across all rows.
• The search replacement includes standard matching, extended matching, and regex and can span multiple documents.
• There are plugins for JSON, python, (MIME tools) Base 64, SAML, (HASH support) MD5, and SHA generators.

Linux editing Tools

The first challenge for any windows only user with Linux is you have to figure out how to run Linux.  I have done this many different ways in the past from VMS to dual boot and compiled unix tools for windows but Microsoft has fixed this issue by allowing you to run any flavor of Linux embedded in Windows as a subsystem or (Windows Subsystem for Linux).  If you are running MacOS there is no need to do this as the underlying OS in Mac OS is Darwin Linux, so most of the tools are already available.  I generally run Ubuntu for its ease of use and tools, online examples, and docs are readily available.  Remember GOOGLE IS YOUR FRIEND.  Google (WSL Ubuntu) to find out how to install, start and update Ubuntu as an application on your Windows system.

Now to the editing tools.  (Disclaimer:  There are hundreds of different ways to use these tools.  I am not a guru, and I don't care too much for elegance as long as it works.  If you want to spend the time to make it prettier or more efficient please share it with your teammates)

What:  SED {Stream Editor} is a good tool for scripting repetitive changes throughout a file or stream of data.

Why: Installed tools for LDAP such as Softerra or LDP are great for querying based on filters or looking at individual entries, but when you need to get bulk data for a specific group of people it is easier sometimes to point at a file for names, uids or email addresses.  I sometimes need to use this to identify faculty or students and differentiate them as users.

How:  Generally installed in most Linux distros by default.  I use it to convert LDAP output in LDIF format to CSV so that I can load it into Excel

Example: When querying an LDAP database using the command line tool ldapsearch the output is formatted in an LDIF output.  Here is an example of output

# extended LDIF
#
# LDAPv3
# base <ou=people,dc=rice,dc=edu> with scope subtree
# filter: mailalternateaddress=bribbeck@rice.edu
# requesting: uid sn given name rice class ou mail
#

# bribbeck, People, rice.edu
dn: uid=bribbeck,ou=People,dc=rice,dc=edu
uid: bribbeck
sn: Ribbeck
given-name: Barry
rice class: Staff
ou: Information Security Office
mail: Barry.R.Ribbeck@rice.edu

# search result
Search: 3
result: 0 Success

# numResponses: 2
# numEntries: 1

If I were to search for a lot of people and want to pull this data into a CSV, I need a way to run it through a parsing routine that will take out all of the lines that I don't want and the attributes such as "dn: " or "uid: " ,  the extra spaces and such then put all of the data into a single line per user with just the data delimited by commas.

So I wrote a simple BASH shell script using the command line editor  VIM or vi and the SED command to modify the file.  Here is the SED bash script.

 Replaced lines beginning with "result:" in file $1
sed -i '/^result:/d' $1
# Replaced lines beginning with "" in file $1
#sed -i '/^dn:/d' $1
# Replace given name: with ""
sed -i 's/givenname: //g' $1
# Replace RDN in UID: with ""
sed -i 's/,ou=People,dc=rice,dc=edu//g' $1
# Replace uid: with ""
sed -i 's/uid: //g' $1
# Replace sn: with ""
sed -i 's/sn: //g' $1
# Replace mail: with """
sed -i 's/mail: //g' $1
# Replace ou: with ""
sed -i 's/ou: //g' $1
# Replace riceuserstatus: with ""
sed -i 's/riceuserstatus: //g' $1
# Replace riceclass: with ""
sed -i 's/riceclass: //g' $1
# Remove all CRLF and replace them with  | as a delimiter
sed -i ':a;N;$!ba;s/\n/|/g' $1
# Replace dn: uid with CRLF to get one user data per line
sed -i 's/|dn: uid=/\r\n/g' $1
# Remove any extraneous "|"
sed -i 's/||//g' $1
# Replace "|" as delimiter with ","
sed -i 's/|/,/g' $1

Copy this into a file "vi parse"

and then change the permissions to make it executable  "chmod 755 parse"

To run execute the script called parse ".\parse ldif" where ldif is the name of the output file from your ldap search