Thursday, 24 December 2020

Postgresql Database Starts with 100% CPU processes ?

If you have a problem with PostgreSQL, it starts some processes that consume lots of resources. You kill those processes but they will restart after a few minutes.

Later you check the /var/log but there is nothing about this. What does that mean?😟


Or sometimes admin user gets created suspiciously and Database Log shows as below

2020-12-20 06:51:17.801 IST [521] postgres@postgres ERROR:  role "admin" does not exist



You have been hacked..!!😓

This is weird and you own a configuration that might be "weakly secured" and probably you might have been hacked if it was exposed publically.  Compromised systems may experience increases in CPU usage, system lag, and business service failures. To remain on servers for a long time, mining trojans use a variety of methods to compromise server security, such as modifying scheduled tasks, firewall configurations, and system dynamic-link libraries, which may cause service interruption on the target server.

Check What are they doing? Check the Postgres log, check any slow running queries. 

Read More: https://www.zdnet.com/article/pgminer-botnet-attacks-weakly-secured-postgresql-databases/



Is really that easy to hack PostgreSQL?

Not with a default installation, But if expose the Postgres server to the internet (so anybody can see and connect) allow superuser access from the outside (i.e. not limited to "localhost") or didn't choose a secure password. Each of those things alone is already a security risk but combined, they are simply asking for trouble. Looking at the log file it is apparent that someone (or something) tries various attempts before succeeding to log in as Postgres. You should immediately disconnect that server from the internet and only allow superuser access through localhost and nothing else. Or better: only allow the user for your application to connect remotely, deny all other users remote access.

One should include the remote IP address in the log output so that you can maybe trace back the attacker. Then investigate what kind of damage the attacker did, before you allow remote connections again.

The option left with you is to...

  1. Dump your database.
  2. Read through the dump. Make sure that the commands to set up the exploit aren't contained in the dump.
  3. Create a completely new server instance from scratch, secure it, and restore the database from the dump.
Also, remember that once a machine has been hacked it cannot ever be truly trusted until it has been absolutely wiped and reinstalled. Don't just think that wiping PostgreSQL and starting again is sufficient, the attack might have been a shell-level hack and a root-kit installed. That root-kit could have replaced any system binaries and any part of the running kernel and left itself a backdoor for when it's discovered.

There have been quite a lot of cases where Postgres installed in/via docker contained some insecure settings and possibly backdoor so that hackers can install bitcoin (or other alt-coin) mining software.

PgMiner marks the second time a coin-miner operation targets PostgreSQL databases, with similar attacks seen in 2018, carried out by the StickyDB botnet. Mining trojans first appeared in 2012. The price of encrypted digital currencies has soared since 2017 and the computing resources of servers can be put to better use due to the emergence of anonymous coins such as Monero, which are untraceable and immune against ASIC miners. Mining trojans have been a major security threat on the Internet since 2018. 


How the BotNet Works?

The attacks follow a simple pattern.

  • The botnet randomly picks a public network range in an attempt to perform RCE(Remote-Code-Execution Attack), iterates through all IP addresses part of that range, searching for systems with PostgreSQL. The attacker scans port 5432, commonly used by PostgreSQL of the hosts in the private/local network exposed online.
  • Once it finds an active PostgreSQL system, the botnet moves from the scanning phase to its brute-force phase, With the user "postgres", which is the default user of the database, the attacker performs a brute-force attack iterating over a built-in list of popular passwords such as “112233“ and “1q2w3e4r“ to crack the database authentication.
  • Once the malware successfully breaks into the database, it uses the PostgreSQL "COPY FROM" program feature to download and launch the coin mining scripts. Further, escalate their access from the database app to the underlying server and take over the entire OS. The SQL COPY FROM feature has been controversial since its debut in PostgreSQL 9.3. The feature allows the local or remote superuser to run shell script directly on the server, which has raised wide security concerns. However, there’s no risk for RCE as long as the superuser privilege is not granted to remote or untrusted users, and the access control and authentication system is properly configured

  • The miner takes a file-less approach, deleting the PostgreSQL table right after code launch. PGMiner creates a table with a text column, saves the malicious payload to it, executes the payload on the PostgreSQL server, and then clears the created table.

  • Once installed, the malware uses curl to carry out tasks. Curl is a command-line tool to transfer data to or from a server. If curl isn’t available on the victim’s machine, researchers found that the malicious script tries multiple approaches to download the curl binary and add it to the execution paths, including Direct installation from official package management utilities like apt-get and yum.

  • The next step, researchers said, is environment cleanup: It removes cloud security monitoring tools and monitor utilities; checks for virtual machines; kills all other CPU intensive processes such as system updates, and kills competitor mining processes.

The last task of course is to begin stealing CPU processor power to mine for Monero. The operators have been controlling infected bots via a command and control (C2) server hosted on the Tor network.


How authentication works in Postgres?

This part of Postgresql is a bit complex, but very powerful. In summary, Postgresql will allow connection following these criteria :

postgresql.conf: listen_adresses and port

pg_hba is then read from top to bottom and - very important - postgresql will stop at the first matching line (match on type of connection, db, ip and user) and use this line to determine connection method. You'll get used to it.

Then check user privileges (LOGIN, then CONNECT to the db, then individual privileges depending on the query)  

Rules of thumb: Never, ever, ever use trust in your pg_hba.conf. Even for test. Even with an IP. Never. It's too easy to mess up (the mask for instance), or to forget a test line.

 



No comments:

Post a Comment