Sponsored by

Conference notes: Modern Pentest Tricks For Faster, Wider, Greater Engagements (Area 41 & HITB 2018)

Posted in Conference notes on July 12, 2018

Conference notes: Modern Pentest Tricks For Faster, Wider, Greater Engagements (Area 41 & HITB 2018)

Hi, these are the notes I took while watching the “Modern Pentest Tricks For Faster, Wider, Greater Engagements” talk given by Thomas Debize on both Area 41 & HITB 2018 conferences.

What changed and why you need to adapt?

In 2018 it is easily possible to:

Why do you need to adapt your techniques?


  • More & more security tools are writing more & more good quality & reliable tools
  • You will be asked to cover wider scopes faster
  • It has already changed

Current pentest assessment require pentesters to SCALE!

Modern tricks for modern pentesters

Data analysis & processing: CSV

  • Pentest involves a lot of iterative work: scan & exploit targets, harvest data & use it on new & old targets
  • So to scale, you need a good data format to quickly process new data on all pentest phases (recon, exploitation, post-exploitation & reporting)
  • Use CSV for inputs & outputs. It is the best format for data analysis & processing
  • It’s a simple human-readable format but there is no standard for CSV
  • CSV just means separate some stuff with another stuff (a delimiter like a comma)
  • Best practices:
    • Encoding: Use utf-8
      • Python 2 “csv” module doesn’t support utf-8, use unicodecsv instead
    • Quoting & escaping: choose to have all fields quoted to prevent CSV injection (Example)

Pentest/infosec tools offering CSV output

Tools to handle CSV

  • Microsoft Excel
    • Use the “Text to Columns” function to convert text to Excel columns & choose your delimiter, then use the “Filter” function
    • Con: Max number of lines is 1 million (~10 to 30 MB file, easily attainable, commonly encountered issue)
  • csvkit
    • Free, open source, one of the best tools for CSV
    • Command-line suite
    • csvkit tools in a nutshell:
in2csv, sql2csvconvert anything to csv
csvclean, csvformatensure your inout or output files are correctly formatted
csvcutjust like UNIX “cut”
csvgrepnot just like UNIX “grep”, allows to search regex/patterns only in desired columns
csvjoinexecute an SQL-like join to merge CSV files on a specified column or columns
csvsortnot just like UNIX “grep”, allows to sort desired fields
csvstackconcatenate/merge multiple CSV files
Output & Analysis
csvjsonconvert a CSV file into JSON
csvlookjust admire the beauty of a CSV file in your interpreter
csvpyload a CSV file into a CSVKitReader object and then drop into a Python shell
csvsqlperform SQL queries on a CSV file
csvstatprint some statistics per columns
  • Dataiku Data Science Studio (DSS)
    • Free & Enterprise editions, not Open Source
    • The free edition is enough
    • Allows to perform the same kind of processing than Excel but without size limitation
    • Intuitive, user-friendly & efficient
      • Only 4 hours on a 4 cores + 16 GB RAM machine to join the “hash” column a 30 GB uncompressed DB dump with a 4 GB “hash : cleartext” file
    • Official tutorials

Parallel execution: GNU Parallel

  • Pentest involves a lot of parallel work:
    • Extracting the results of a tool output on multiple targets
    • Launching the same bruteforce on multiple targets
  • Parallel execution is crucial to scale on wide scopes
  • GNU Parallel is a Perl script to parallelize any command and maximize I/O & CPU usage
  • Useful options:

| — | — | | --progress | a percentage of done/to be done | | --bar | a nice progress bar | | --joblog | a log of executed tasks, allows resuming | | -- resume | resume to your current execution status | | --sshlogin | distribute the tasks on remote computers through SSH |

Example 1: Directory bruteforce
  • Run wfuzz on target_list.txt
$ parallel -a target_list.txt --joblog joblog --progress --bar 
		-f 'results/result_wfuzz_{= s/[:\/]/_/g =}.json',json
		--filter 'c<403'
		-R 3 -Z -c
		-z file, '/usr/share/dirbuster/wordlists/directory-list-lowercase-2.3-small.txt
- `{= s/[:\/]/_/g}` is a Perl/sed expression to remove bad chars in filename
- Format of *target_list.txt*:
  • Then filter results by 200 HTTP response code & get a screenshot of all responding URLs with webscreenshot
$ ls ./results/result_wfuzz_*.json | parallel
"cat {} | jq '.[] | if (.code == 200) then .url else empty end' | sed 's/"//g' >> list_to_webscreenshot.txt"
Example 2: DNS enumeration
$ parallel -a subdomains.txt --joblog joblog --progress --bar
	"dig +noall +answer {}.<target domain>"
- Format of subdomains.txt:
Dig with GNU Parallel (cheap method) vs Gobuster (proper & optimized tool)
  • Gobuster: Directory/file & DNS busting tool written in Go
$ git clone https://github.com/OJ/gobuster.git && cd gobuster
$ time go run main.go -u google.com -w ~/subdomains.txt -m dns -q -o google.com_subdomains_by_gobuster.txt

=> ~22s

  • Dig: Native tool for DNS enumeration
$time parallel -a ~/subdomains.txt "dig +noall +answer {}.google.com | tee -a google.com_subdomains_by_parallel.txt"

=> ~27s

  • Dig with GNU Parallel is almost as perfomant as Gobuster
  • Very useful when you cannot install any tool on your attack machine (for example if provided by the client). You only have to install GNU Parallels

High-level scripting languages for easier static & dynamic analysis: Jython & Frida

  • Pentest sometimes involves reversing “custom-wtf” obfuscation or encryption
  • Usually during Android applications & Java thick clients engagements
  • Why use a high-level scripting language for instrumentation?
    • If you don’t want to go down the rabbit hole to figure out how it works
    • If you can’t replicate/rip the code into your favourite language
      • E.g.: Oracle WebLogic Server encrypts local passwords with a cryptosystem that’s not implemented in any Python module
  • Use Jython for static analysis of Android & Java apps
    • Writing Java code in Python… that can use Java classes… and Python libraries in the same snippet
  • Use Frida for everything else
    • Writing Python or JS or QML or Swift or .NET… injecting C++ scripted in JS (Google v8)… to instrument ASM, Objective-C or Dalvik… on Windows, Mac, Linux, Android, iOS
  • Example: WebLogic Server password decryptor in Jython

Compile Python scripts on-the-fly: PyInstall

  • Pentest sometimes involves the need to have compiled version of tools
  • Because:
    • The target you are onto does not have the proper execution environment (dependencies, interpreter) & you can’t install it (no root, no outgoing connection, laziness…)
    • You can’t just have a proper reverse shell or Meterpreter
    • You need to evade antivirus
  • Compile Python tools with PyInstall
  • PyInstaller bundles a Python script with a Python interpreter
  • You can compile a script for Windows on Windows
  • You can also cross-compile for Windows from Linux with wine (Tutorial)
  • Useful options:
    • --onefile: creates a standalone executable file which is a self-extracting zip payload
    • --onedir: creates a single directory with everything inside if you don’t want a standalone executable file, as large standalones (> 18 MB) take time to unzip before execution
    • --key <key>: a specific key to encrypt the zip payload, of course included in the executable (Tutorial)
      • Encrypt the payload & include the key in the executable (no obfuscation used), useful for anti-virus bypass
    • --icon <icon_file>: for visual fanciness

Examples of Python scripts compiled with PyInstaller

  • Impacket examples
    • Impacket example scripts compiled for Windows
    • Practical Usage of NTLM Hashes
    • High-value utilities in the Impacket tool suite: mimikatz.exe, ntlmrelay.exe, psexec.exe, samrdump.exe, secretsdump.exe, smbexec.exe, smbrelay.exe, ticketer.exe, wmiexec.exe, wmipersist.exe, wmiquery.exe
    • Real-life tricks
      • Impacket tools are gaining popularity and getting flagged by AV. Just use the --key option to bypass them!
      • If you cannot grab the Ntds database (Ntds.dit) of a domain because of the network connection or not enough space on your computer, just take the secretsdump.exe executable & put it on the target or on the domain controller wherever you want & you’ll be able to process the Ntds file remotely & grab password hashes from it
  • Patator compiled for Windows
  • CrackMapExec compiled for Windows
    • Thomas is the unofficial official maintainer of the compiled version 2
    • Version 2 is old but is UTF-8 compatible (useful for pentesting sites containing accents like French sites)
  • jdwp-shellifier compiled for Windows
  • Compile rpivot from Linux for Linux
git clone https://github.com/klsecservices/rpivot.git
cd pivot
pyinstaller --clean --onefile server.py
./dist/server -h

Searching code: Stop using grep

Taking a step back


  • CSVKit
  • Dataiku
  • GNU Parallel
  • Jython
  • Frida
  • PyInstall
    … all the things!

See you next time!