Formatting JSON with the Python JSON Tool

When working with web server JSON responses, if they span several lines with no ‘new line’ formatting they can be difficult to read. When JSON responses end up on the command line, it is possible to format them for easier reading.

Here I use a short output from the jsontest.com site as an example.

Copyable commands:

curl http://ip.jsontest.com/
curl -s http://ip.jsontest.com/ | python -m json.tool

For this, you will need python and mjson installed. Assuming you have python, you can check and install mjson as follows:

~$which mjson
~$

~$pip install mjson
--snip--

~$which mjson
/usr/local/bin/mjson

Hope this is of use!

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Building a UK Street / Road Name Wordlist

If you need to create custom dictionaries or wordlists, there are many options, in this case we were looking for UK specific street / road names.

You do not need to ‘scrape’ open street map for data, it is not ideal from their point of view or yours. As the data is open, they do actively provide downloadable exports (i.e an ‘.osm.pbf’ file) for different data sets.

Looking at a page of export data such as (http://download.geofabrik.de/europe.html) will enable you to find and download the correct file for your needs, and it can then be processed offline.

On Kali (or similar) you should have the osmosis package available which can be used to process Open Street Map data exports, filtering out only highways. Then with a little grep, these can be used to create steet / road name lists.

The same idea can be used for counties, towns, citys etc.

#Install tools
apt-get install osmosis

# Download a '.osm.pbf' file for your area: http://download.geofabrik.de/europe.html
wget http://download.geofabrik.de/europe/great-britain-latest.osm.pbf

# use a program like Osmosis to filter out only highways
osmosis --read-pbf file.osm.pbf --tf accept-ways highway=\* --write-xml myfile.osm

# From the resulting XML file, extract all names
grep 'k="name"' myfile.osm | cut -d\" -f4

You will likely find some slightly poor data within the file (at the time of writing), this is essentially down to user supplied data to the open street map project.

This however does not worry me overly for this usage case. You will find that some extracted data files are quite large, so remember to clean up when your done.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Monitoring network traffic with port mirroring

If you would like to monitor the traffic between two network devices or network segments, you will need to obtain access to the packet flow between the two hosts or network segments.  In a virtual lab environment we have a number of options, and normally the opportunity to reconfigure things to get access.

In a production or more a physical environment, we may need to insert a network tap (however this would require disconnection of the link to connect the tap, and also an expensive tap) or configure the production switch to mirror traffic to an available socket on the switch.

Alternatively it is possible to configure a spare low cost switch such as the Netgear ProSAFE GS105E v2 to mirror a port, and then insert it like a tap.  This process is also similar to what you would need to complete on a production switch.

First find the IP address of the switch.  If you use a new ProSAFE GS105E, you may need to use the utility on a new switch, or configure your machine to access the default subnet 192.168.0.0/24 and connect to http://192.168.0.239 (the default ‘off network’ address of this switch).

Next, navigate to the “System” > “Monitoring” > “Mirroring”.  Here we can define a ‘source port’ (1) where we want to monitor all traffic, next we select a destination port (5) that all traffic on source port (1) will be mirrored to.  On the Netgear we also need to change the ‘Mirroring’ drop down to ‘enabled’.

If this was a production switch, we would be monitoring an already active port and using a spare (5) as the mirror for monitoring, in this case, I will connect ports 1 and 2 to a test network and an internet of things (IoT) device bridge that we are interested in monitoring.  I will then connect my laptops network interface to port 5, the mirrored interface.  (Remember, you are not connecting to the network here, you are monitoring a copy of the traffic flowing through port 1.

We therefore fire up Wireshark, selecting the correct physical interface that connects into port 5, and then monitor the network link with the promiscuous option selected.

Wireshark should show you the packets passing through port1, in this test environment you can see source address x.x.79.35 with packets heading to 192.168.3.10, and also 192.168.3.10 sending packets to x.x.79.35.

We can now happily inspect the traffic between this device and the network, we can see traffic flowing in both directions and at a high level see the two main IP addresses of interest, along with a data flow on UDP/2011 that can be further inspected.

Enjoy.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Firefox Captive Portal Detection and Burp Suite

If you get requests like this in Burp, and you don’t need or want them…… you need to disable captive portal detection.

Eg:
GET /success.txt HTTP/1.1
Host: detectportal.firefox.com

There isn’t a easy checkbox to configure this, however it is possible to disable using about:config.

  1. In a new tab, type about:config in the address bar and press Enter.
  2. In the search box above the list, type captive.
  3. Double-click the network.captive-portal-service.enabled preference to switch the value from true to false.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

How to list a range of IP addresses with the Linux seq command

Ever needed to convert IP addresses that someone has written down into a list?
eg: 10.0.0.47 to 52 and 192.168.243.5 to 12

Try seq, as follows:

$seq -f "10.0.0.%g" 3 6
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.6

Or for my example:

$seq -f "10.0.0.%g" 47 52 ; seq -f "192.168.243.%g" 5 12
10.0.0.47
10.0.0.48
10.0.0.49
10.0.0.50
10.0.0.51
10.0.0.52
192.168.243.5
192.168.243.6
192.168.243.7
192.168.243.8
192.168.243.9
192.168.243.10
192.168.243.11
192.168.243.12

You could then redirect the output too:

$seq -f "10.0.0.%g" 47 52 ; seq -f "192.168.243.%g" 5 12 > ips.txt

Hope this is useful.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Viewing HTTP Headers on Linux with curl

If you want to identify the configuration options for STS etc, you will need to look at HTTP Headers. From Linux you can use curl. For HTTPS sites, you may want to consider the –insecure option if you have other cert issues to contend with.

~$curl -I https://jervis.ws/
HTTP/1.1 200 OK
Date: Tue, 26 Apr 2016 20:43:16 GMT
Server: Apache
Link: ; rel="https://api.w.org/"
Content-Type: text/html; charset=UTF-8
Facebooktwittergoogle_plusredditpinterestlinkedinmail

Browser Separation with Integration using VirtualBox

seamless-modeSeamless window integration of two separate operating systems, with isolated network stacks.

Ransomware poses a real risk to every day web surfers. There are multipule attack vectors, drive by downloads, compromised sites and malvertising to name just a few are all huge areas of risk. This leads some to want seperation or sand boxing of their web browser, either for some of their browsing or all of it.

The main motivation here is separation for risk reduction and security, not privacy/anonymity.

With browser security under screwtany, things are improving however with problem after problem with 3rd party addons and browser plugins, scripting issues, 3rd party code includes and so on, the web is full of risk areas before you even consider malvertising compromised sites or cross site scripting.

Moving beyond ad or script blocking

sandpitBy utilising VirtualBox, you can build a (mostly) separate computer for your browser. Minimising risk through separation using a Virtual Machine Sandbox.

Whilst using a physically separate computer provides better security to your ‘main’ system, it is impractical in the majority of cases. Creating a Virtual Machine sandbox, with some clearly understood elements of integration between the two systems provides much better functionality and end user experience, whilst maintaining a significantly higher level of security.

Why not other approaches?

As a former user of No Script, I became a little tired of picking through the many scripts to try and unpick what web developers were thinking in a drive to make the sites function again. I was also very aware of a quick ‘allow all on this page’ one lazy day could undo years of time consuming script inspection. This is not the solution for the faint hearted and not something I could recommend to may.

Ad Blocking? Ok so this protects against some threats such as Malvertising, however it does not address any other risk factors and therefore leaves massive areas for bad actors within the browser. We are also now in a world of Ad Blockers, Ad Block Blockers and Ad Block Blocker Blockers.

Sandboxing via Sandboxie? This does not stop the browser reading files on your system.

Physically separate system? Ok, but lets face it, who runs a separate PC for ALL of their browsing…. not many. If you are doing any amount of browsing from your PC then this option could help secure the browsing.

What do I need?

You will need to download and install VirtualBox. You will also need either a Windows Licence and media, or a copy of Linux. I have used both Windows and Ubuntu in this browser isolation technique successfully. I recommend having an ISO image of the disk for easy build and rebuild should you wish.

Configuring Virtual Box

Once you have created a basic Virtual Machine selecting either the Windows or Linux templates, you will need to make some changes. These adjust the security and functionality. Some improve security and separation, whilst others create openings between the Virtual Machine and your host computer.

Which of these changes you make is personal preference, and down to your use case and requirements. For example, if your worried about malware within the browser reading clipboard content on the host OS, or if you are worried about file encryption malware.

When you access web content, you are likely going to want to download files. If your then going to need these on your ‘main’ system, you will need a seamless way to export these files. Consider setting up a share between the VM and your host system. You should create a sub folder for this, and share this to protect your other files and the rest of your host PC from any security risks. Eg: have a temporary downloads folder, accessible from both systems.

Create Share with VM

When configuring your networking, you have several options.

NAT – The VM will share your PC’s IP address. However your PC and the Browser will each have an IP address on an internal NAT network. You will need to factor in host firewall security.
Bridged – Your host PC will push the browser VM to the main network (either wired or wireless depending on your configuration). You will then be able to protect the two systems from each other as if they were separate network devices. Review your hosts Windows Firewall security configuration. (Likely the best option)
Internal – Not appropriate.
USB – See below, under “Looking for better network separation?”

Configure Virtual Networking

You will then need to install either Windows or Linux. eg: Ubuntu. In this case, I have installed Windows 7.

Install Windows 7

Whichever OS you install, you can install the guest tools to unlock integration including clipboard and folder sharing. Even if you do not want these features, you can switch them on and off, the tools will help ensure your guest VM behaves correctly.

Install VirtualBox Tools

Consider how you wish to interact with your browser, eg: you may wish to copy links from the host to the guest… or you may wish to copy webpage text from the guest to the host. Set this to your requirements, just remember that if the guest can read your host clipboard, then it can read your clipboard all the time. This type of risk is reduced later with read only disks and snapshot reverts.

Clipboard Settings

As per the above, configure a set sub folder to be shared with the VM if you want to seamlessly move files. However if you share too much, the guest browser VM will have access to it.

Test Folder Share

Once you have your system installed, updated and correctly configured it is best to take a snapshot. This will save the state of the system by creating a differencing image file, where any new disk writes are stored. Your live VM will behave normally but you will be able to revert to the snapshot.

Take new snapshot

Once you are up and running you will find you need to patch / update your browser and browser guest operating system. I would recommend when you need to do this, you revert the VM to a known clean snapshot, perform the updates and take a new snapshot. Once you are happy all is well you should delete any snapshots that are no longer required as managing these will be come more complex and slow performance over time.

Delete unneeded snapshots

When you want to refresh your browser, eg: clean the session, drop any malware stuff etc from the guest OS, then revert to a known clean snapshot. You may choose to do this daily, and/or after visiting untrusted content etc etc.

Revert to or restore snapshot as needed

Investigate the display modes to play with seamless window integration.

seamless-mode

Desktop 1

Looking for better network separation?

If you would like even clearer network separation, then consider this. VirtualBox is loading a USB kernel driver into your host OS. So remove all the network cards from the VM, now the system does not share the host systems network adaptor at all. Now connect a USB Ethernet Adaptor to the PC, and create a USB filter.

USB Ethernet

USB Ethernet

Immutable images vs Snapshots

My original option was to use the VirtualBox immutable image setting for the main guest drive, and then the differencing file would be automatically thrown away when the system went through a full power off and power on cycle. (All changes are lost when the virtual machine is powered on the next time, as this is when the temporary differencing file is removed). I noticed issues however where this was not removing the differencing files correctly and I was ending up with several, or I would revert the image to ‘normal’ mode to install updates, and these updates would go into a differencing image. I therefore recommend snapshots at this time, which also helps keep this straight forward as you will need to be used to reverting and creating new snapshot for the patch/update process.

Thoughts?

Would like to here your thoughts and ideas, please comment below.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

VeraCrypt takes a minute to pre-boot authenticate

VeraCrypt PIM 2VeraCrypt is a free disk encryption software, that is based on TrueCrypt 7.1a.

If you are suffering slow boot up with VeraCrypt, you password takes ages to be accepted or slow pre boot authentication then read on…

When I first tested VeraCrypt with Windows 10, I had problems with it taking around a minute to process the password in the pre-boot environment. After a couple of boots, I wondered if this was really a workable potion at this time.

I then found this was due to the “Personal Iterations Multiplier”, PIM. It is a parameter that was introduced in VeraCrypt 1.12 and whose value controls the number of iterations used by the header key derivation function.

As shown in the screenshots, it is possible to set this value as you wish, some quick tests showed setting the value to 1 took less than a second. Therefore, I recommend reviewing the documentation and then selecting an appropriate value if you are suffering from this problem.

https://veracrypt.codeplex.com/wikipage?title=Personal%20Iterations%20Multiplier%20%28PIM%29

VeraCrypt PIM 1

Facebooktwittergoogle_plusredditpinterestlinkedinmail

letsencrypt quick setup – Ubuntu and Apache

Note to self following a quick setup on a development box….

Setup:

apt-get install git
git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt
cd /opt/letsencrypt
./letsencrypt-auto --apache -d example.com -d www.example.com --email admin@example.com --agree-tos

To renew:

/opt/letsencrypt/letsencrypt-auto renew

Quick cron hack for renewals:

echo "/opt/letsencrypt/letsencrypt-auto renew" >> /etc/cron.daily/letsencrypt-cron.sh
chmod +x /etc/cron.daily/letsencrypt-cron.sh

Test with:

/etc/cron.daily/letsencrypt-cron.sh

Thanks to Adam for the head start: https://www.adamcouch.co.uk/2016/02/20/lets-encrypt/
More here: https://letsencrypt.org/

Facebooktwittergoogle_plusredditpinterestlinkedinmail