Conference notes: Modern Pentest Tricks For Faster, Wider, Greater Engagements (Area 41 & HITB 2018)

Hi, these are the notes I took while watching the “Modern Pentest Tricks For Faster, Wider, Greater Engagements” talk given by Thomas Debize on both Area 41 & HITB 2018 conferences.


What changed and why you need to adapt?

In 2018 it is easily possible to:

Why do you need to adapt your techniques?


  • More & more security tools are writing more & more good quality & reliable tools
  • You will be asked to cover wider scopes faster
  • It has already changed

Current pentest assessment require pentesters to SCALE!

Modern tricks for modern pentesters

Data analysis & processing: CSV

  • Pentest involves a lot of iterative work: scan & exploit targets, harvest data & use it on new & old targets
  • So to scale, you need a good data format to quickly process new data on all pentest phases (recon, exploitation, post-exploitation & reporting)
  • Use CSV for inputs & outputs. It is the best format for data analysis & processing
  • It’s a simple human-readable format but there is no standard for CSV
  • CSV just means separate some stuff with another stuff (a delimiter like a comma)
  • Best practices:
    • Encoding: Use utf-8
      • Python 2 “csv” module doesn’t support utf-8, use unicodecsv instead
    • Quoting & escaping: choose to have all fields quoted to prevent CSV injection (Example)

Pentest/infosec tools offering CSV output

Tools to handle CSV

  • Microsoft Excel
    • Use the “Text to Columns” function to convert text to Excel columns & choose your delimiter, then use the “Filter” function
    • Con: Max number of lines is 1 million (~10 to 30 MB file, easily attainable, commonly encountered issue)
  • csvkit
    • Free, open source, one of the best tools for CSV
    • Command-line suite
    • csvkit tools in a nutshell:
in2csv, sql2csv convert anything to csv
csvclean, csvformat ensure your inout or output files are correctly formatted
csvcut just like UNIX “cut”
csvgrep not just like UNIX “grep”, allows to search regex/patterns only in desired columns
csvjoin execute an SQL-like join to merge CSV files on a specified column or columns
csvsort not just like UNIX “grep”, allows to sort desired fields
csvstack concatenate/merge multiple CSV files
Output & Analysis  
csvjson convert a CSV file into JSON
csvlook just admire the beauty of a CSV file in your interpreter
csvpy load a CSV file into a CSVKitReader object and then drop into a Python shell
csvsql perform SQL queries on a CSV file
csvstat print some statistics per columns
  • Dataiku Data Science Studio (DSS)
    • Free & Enterprise editions, not Open Source
    • The free edition is enough
    • Allows to perform the same kind of processing than Excel but without size limitation
    • Intuitive, user-friendly & efficient
      • Only 4 hours on a 4 cores + 16 GB RAM machine to join the “hash” column a 30 GB uncompressed DB dump with a 4 GB “hash : cleartext” file
    • Official tutorials

Parallel execution: GNU Parallel

  • Pentest involves a lot of parallel work:
    • Extracting the results of a tool output on multiple targets
    • Launching the same bruteforce on multiple targets
  • Parallel execution is crucial to scale on wide scopes
  • GNU Parallel is a Perl script to parallelize any command and maximize I/O & CPU usage
  • Useful options:
--progress a percentage of done/to be done
--bar a nice progress bar
--joblog a log of executed tasks, allows resuming
-- resume resume to your current execution status
--sshlogin distribute the tasks on remote computers through SSH
Example 1: Directory bruteforce
  • Run wfuzz on target_list.txt
    $ parallel -a target_list.txt --joblog joblog --progress --bar 
          -f 'results/result_wfuzz_{= s/[:\/]/_/g =}.json',json
          --filter 'c<403'
          -R 3 -Z -c
          -z file, '/usr/share/dirbuster/wordlists/directory-list-lowercase-2.3-small.txt
    • {= s/[:\/]/_/g} is a Perl/sed expression to remove bad chars in filename
    • Format of target_list.txt:
  • Then filter results by 200 HTTP response code & get a screenshot of all responding URLs with webscreenshot
    $ ls ./results/result_wfuzz_*.json | parallel
    "cat {} | jq '.[] | if (.code == 200) then .url else empty end' | sed 's/"//g' >> list_to_webscreenshot.txt"
Example 2: DNS enumeration
Dig with GNU Parallel (cheap method) vs Gobuster (proper & optimized tool)
  • Gobuster: Directory/file & DNS busting tool written in Go
    $ git clone && cd gobuster
    $ time go run main.go -u -w ~/subdomains.txt -m dns -q -o google.com_subdomains_by_gobuster.txt

    => ~22s

  • Dig: Native tool for DNS enumeration
    $time parallel -a ~/subdomains.txt "dig +noall +answer {} | tee -a google.com_subdomains_by_parallel.txt"

    => ~27s

  • Dig with GNU Parallel is almost as perfomant as Gobuster
  • Very useful when you cannot install any tool on your attack machine (for example if provided by the client). You only have to install GNU Parallels

High-level scripting languages for easier static & dynamic analysis: Jython & Frida

  • Pentest sometimes involves reversing “custom-wtf” obfuscation or encryption
  • Usually during Android applications & Java thick clients engagements
  • Why use a high-level scripting language for instrumentation?
    • If you don’t want to go down the rabbit hole to figure out how it works
    • If you can’t replicate/rip the code into your favourite language
      • E.g.: Oracle WebLogic Server encrypts local passwords with a cryptosystem that’s not implemented in any Python module
  • Use Jython for static analysis of Android & Java apps
    • Writing Java code in Python… that can use Java classes… and Python libraries in the same snippet
  • Use Frida for everything else
    • Writing Python or JS or QML or Swift or .NET… injecting C++ scripted in JS (Google v8)… to instrument ASM, Objective-C or Dalvik… on Windows, Mac, Linux, Android, iOS
  • Example: WebLogic Server password decryptor in Jython

Compile Pythin scripts on-the-fly: PyInstall

  • Pentest sometimes involves the need to have compiled version of tools
  • Because:
    • The target you are onto does not have the proper execution environment (dependencies, interpreter) & you can’t install it (no root, no outgoing connection, laziness…)
    • You can’t just have a proper reverse shell or Meterpreter
    • You need to evade antivirus
  • Compile Python tools with PyInstall
  • PyInstaller bundles a Python script with a Python interpreter
  • You can compile a script for Windows on Windows
  • You can also cross-compile for Windows from Linux with wine (Tutorial)
  • Useful options:
    • --onefile: creates a standalone executable file which is a self-extracting zip payload
    • --onedir: creates a single directory with everything inside if you don’t want a standalone executable file, as large standalones (> 18 MB) take time to unzip before execution
    • --key <key>: a specific key to encrypt the zip payload, of course included in the executable (Tutorial)
      • Encrypt the payload & include the key in the executable (no obfuscation used), useful for anti-virus bypass
    • --icon <icon_file>: for visual fanciness

Examples of Python scripts compiled with PyInstaller

  • Impacket examples
    • Impacket example scripts compiled for Windows
    • Practical Usage of NTLM Hashes
    • High-value utilities in the Impacket tool suite: mimikatz.exe, ntlmrelay.exe, psexec.exe, samrdump.exe, secretsdump.exe, smbexec.exe, smbrelay.exe, ticketer.exe, wmiexec.exe, wmipersist.exe, wmiquery.exe
    • Real-life tricks
      • Impacket tools are gaining popularity and getting flagged by AV. Just use the --key option to bypass them!
      • If you cannot grab the Ntds database (Ntds.dit) of a domain because of the network connection or not enough space on your computer, just take the secretsdump.exe executable & put it on the target or on the domain controller wherever you want & you’ll be able to process the Ntds file remotely & grab password hashes from it
  • Patator compiled for Windows
  • CrackMapExec compiled for Windows
    • Thomas is the unofficial official maintainer of the compiled version 2
    • Version 2 is old but is UTF-8 compatible (useful for pentesting sites containing accents like French sites)
  • jdwp-shellifier compiled for Windows
  • Compile rpivot from Linux for Linux
    git clone
    cd pivot
    pyinstaller --clean --onefile
    ./dist/server -h

Searching code: Stop using grep

Taking a step back


  • CSVKit
  • Dataiku
  • GNU Parallel
  • Jython
  • Frida
  • PyInstall
    … all the things!

See you next time!