Reducing server load

In this article, we will delve into why increased server load occurs and discuss various ways to optimize high-load processes. Special attention will be given to code optimization in Apache/Nginx and MySQL, we will talk about caching as an auxiliary tool, and also consider possible external threats, such as DDOS attacks, and ways to prevent them.

Why Server Load Occurs

Before proceeding to server optimization, it is necessary to conduct a thorough analysis of the current load on resources. This includes measuring CPU load, RAM usage, network activity, and other key parameters. Understanding the dynamics and peak loads allows identifying bottlenecks and optimizing resource allocation, thus increasing the stability and performance of the server infrastructure.

For initial troubleshooting of high server load, we recommend conducting a general server diagnostics. If this is insufficient, a more detailed analysis of resources is necessary. As an auxiliary tool, exploring the logs of the Linux server can be helpful, as this is where the source of the problem is found in most cases.

Optimizing Apache/Nginx Server

Increased Server Load Due to Indexing

Increased load due to indexing on the server can occur, for example, when search engines scan a large number of pages on your site. This can lead to increased use of server resources and, consequently, slow down the site's performance. Identifying the cause is relatively simple; you need to open the file located at:

/var/www/httpd-logs/sitename.access.log

When indexed by search engines, the user will see entries of the following nature:

11.22.33.44 - - [Date and Time] "GET /your-page-path HTTP/1.1" 200 1234 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

As a first solution to reduce the load, you can use the setting of meta tags "noindex" and "nofollow" on pages that do not need to be indexed. The second solution is the .htaccess file, where entries corresponding to specific search engines need to be added, for example, to hide from Yandex and Google:

SetEnvIfNoCase User-Agent "^Yandex" search_bot
SetEnvIfNoCase User-Agent "^Googlebot" search_bot
Order Allow,Deny
Allow from all
Deny from env=search_bot

Similarly, edits need to be made for other search engines. It should be noted that the capabilities of .htaccess are not limited to just blocking indexing. We recommend getting more familiar with its main features in the article.

Using Caching Settings

Incorrect caching settings on the server can also lead to high load. To optimize this parameter, corresponding changes need to be made in the configuration files or .htaccess. In the case of Apache, the latter option is preferable, for Nginx – the former.

On an Apache server, you need to open the .htacess file and insert the following code:

<FilesMatch ".(flv|gif|jpg|jpeg|png|ico|swf|js|css|pdf|doc|docx)$">
Header set Cache-Control "max-age=2592000"
</FilesMatch>

Then, enable the Expires module using the command:

sudo a2enmod expires

After which, restart the web server:

sudo service apache2 restart

And activate the module by specifying:

ExpiresActive On

On a Nginx server, it is sufficient to add the following code to the configuration file:

location ~* .(jpg|jpeg|gif|png|ico|css|swf|flv|doc|docx)$ {
root /var/www/yoursite.com;
}

And perform a service reload:

sudo service nginx restart

Note that with these settings, the Allow and Deny directives will be bypassed.

Using Data Compression

Enabling data compression using Gzip on Apache and Nginx web servers helps reduce the amount of data transmitted between the server and the client, which improves performance and reduces web page loading time.

To enable Gzip on Apache, you need to activate the mod_deflate module:

sudo a2enmod deflate

Then, restart the web server:

sudo service apache2 restart

And finally, add the following block to the configuration file or .htaccess:

<IfModule mod_deflate.c>
# Configure compression for specified file types
AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css text/javascript application/javascript application/x-javascript application/json

# If the browser matches the specified pattern, apply compression only to text/html files
BrowserMatch ^Mozilla/4 gzip-only-text/html

# If the browser matches the specified version patterns of Mozilla 4.0.6, 4.0.7, 4.0.8, disable compression
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# If the browser is MSIE (Internet Explorer), disable compression for all files except text/html
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

# If the request contains the specified pattern (extensions of image files), disable compression
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip
</IfModule>

This configuration enables compression for certain types of files and disables it for images.

In the case of Nginx, configuration occurs in the http block of the configuration file. The following code needs to be added:

gzip on;
gzip_disable "msie6";

# Adds the Vary header, indicating that the response may change depending on the Accept-Encoding header value
gzip_vary on;

# Enables compression for any proxy servers
gzip_proxied any;

# Sets the compression level. A value of 6 provides a good balance between compression efficiency and resource use
gzip_comp_level 6;

# Sets the size of the buffer for compressed data (16 buffers of 8 kilobytes each)
gzip_buffers 16 8k;

# Specifies that data compression should be used only for HTTP version 1.1 and higher
gzip_http_version 1.1;

# Sets the file types that can be compressed
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

Similar to Apache, here the compression parameters for certain types of files are set. After making changes to any of the web servers, a service reload is required:

sudo service apache2 restart

Or

sudo service nginx restart

DDOS Attack on the Server

High server load can occur as a result of a DDoS attack. Identifying the presence of a DDoS attack can be done through monitoring a sudden increase in traffic, abnormal requests, and server performance drops. Reviewing logs for repeated requests from one IP address or port scanning can also indicate a possible DDoS attack. There are many protection measures, but we will only discuss the basics.

Using a CDN (Content Delivery Network). A CDN can serve as an intermediary between your web server and users, distributing traffic and caching content to mitigate the impact of a DDoS attack. CDNs can also have built-in DDoS protection mechanisms, including load distribution and traffic filtering.

Configuring firewalls and intrusion detection systems (IDS/IPS). Firewalls can be configured to filter traffic based on various criteria, such as IP addresses and ports. IDS/IPS can detect abnormal traffic behavior and block suspicious connections. These tools can be effective in tracking and blocking potentially malicious traffic.

Configuring Apache and Nginx web servers to mitigate the impact of DDoS attacks.

As a solution for Apache, we enable the mod_evasive module. To do this, uncomment or add the following line in the httpd.conf or apache2.conf configuration file:

LoadModule evasive20_module modules/mod_evasive.so

In the same file, you need to add a settings block:

<IfModule mod_evasive20.c>
# Hash table size for storing request information
DOSHashTableSize 3097

# Number of requests to one page before activating protection
DOSPageCount 2
DOSPageInterval 1

# Number of requests to all pages before activating protection
DOSSiteCount 50
DOSSiteInterval 1

# Blocking period in seconds for IP addresses
DOSBlockingPeriod 10
</IfModule>

Similarly, we activate the mod_ratelimit module:

LoadModule ratelimit_module modules/mod_ratelimit.so

And add the configuration:

<IfModule mod_ratelimit.c>
# Setting the output filter for rate limiting (Rate Limit)
SetOutputFilter RATE_LIMIT

# Beginning of the settings block for the location "/login"
<Location "/login">

# Setting the environment variable rate-limit with a value of 1
SetEnv rate-limit 1

# Ending of the settings block for the location "/login"
</Location>
</IfModule>

The configuration for Nginx is similar to Apache. In the nginx.conf configuration file, the following directives need to be utilized:

http {
...
# Defining a zone for connection limits
limit_conn_zone $binary_remote_addr zone=addr:10m;

# Defining a zone for request limits
limit_req_zone $binary_remote_addr zone=req_zone:10m rate=1r/s;

server {
        ...
        # Configuring connection limits
        limit_conn addr 10;

        # Configuring request limits
        limit_req zone=req_zone burst=5;

        ...
    }
}

After making changes to each of the services, they need to be reloaded:

sudo systemctl restart apache2

Or:

sudo systemctl restart nginx

These examples provide only a basic configuration, which can be further adapted depending on specific requirements and the nature of attacks.

Optimizing MySQL Queries

Optimizing MySQL database queries on a web server can be achieved in various ways, and one of them is the proper configuration of the configuration file. Typically, this file is named my.cnf or my.ini and is located in the /etc/ or /etc/mysql/ directory. You need to open it and make the following changes:

[mysqld]
# Location of the file for recording slow queries. Be sure to replace it with your path
log-slow-queries = /var/log/mariadb/slow_queries.log

# Threshold time for considering slow queries (in seconds)
long_query_time = 5

# Enabling recording of queries that do not use indexes
log-queries-not-using-indexes = 1

# Disabling query caching
query_cache_size = 0
query_cache_type = 0
query_cache_limit = 1M

# Size of temporary tables
tmp_table_size = 16M
max_heap_table_size = 16M

# Size of the thread cache
thread_cache_size = 16

# Disabling name resolving
skip-name-resolve = 1

# Size of the InnoDB buffer pool. Set to 50-70% of available RAM
innodb_buffer_pool_size = 800M

# Size of the InnoDB log file
innodb_log_file_size = 200M

Let's also consider additional recommendations that can facilitate interaction with the server database:

1. Use the EXPLAIN command before an SQL query to analyze its execution. This allows you to get an execution plan for the query and determine which indexes are used, which tables are scanned, etc.
2. Indexes speed up data search, so properly designed indexes can significantly improve query performance. Pay attention to columns that are frequently used in WHERE or JOIN conditions.
3. Avoid using SELECT *. Specify only those columns that are truly necessary for your query, instead of selecting all columns in a table.
4. Avoid using functions in WHERE conditions. Using functions (such as LOWER, UPPER, LEFT, RIGHT) in WHERE conditions can make indexes useless. Try to avoid their direct use in conditions.
5. Use INNER JOIN where possible, as it is usually more efficient. Also, ensure that the corresponding columns for joining have indexes.
6. Use LIMIT to restrict the number of returned rows if you need to get only a certain number of results.
7. Consider caching query results, especially if they rarely change, to reduce server load.

The Mail Server Creates High Load on the Server

In this section, we will explore how to determine that the mail server is experiencing high load and what steps can be taken to optimize its operation, including checking the message queue and configuring server parameters. Start with checking the message queue. The mailq utility can help with this, to activate it, enter the corresponding command in the terminal:

mailq

This will display a list of messages in the queue, if any. Each message will be displayed with its unique identifier and information about the sending status. A similar result can be obtained by reviewing the mail client logs.

In most cases, high load occurs in the event of server compromise when it starts sending spam. However, if after checking the administrator is confident that the server has not been attacked from the outside and users are not neglecting spam, it's time to move on to optimizing the mail server. Here are the steps that will help:

1. Ensure that your domain's DNS records are correctly configured, including SPF, DKIM, and DMARC records to improve mail delivery and protect against spam. The correct configuration of parameters can be found in the article on mail server diagnostics.
2. Check network settings, including firewall configuration and routing rules, to avoid blocks and speed up mail delivery.
3. Configure message queue parameters according to server load. This may include setting the maximum queue size and timeouts.
4. Consider the solutions we discussed in this article earlier. Periodically optimize the mail server database to improve performance, use caching mechanisms to speed up data search and processing, such as DNS queries.
5. If the mail server still regularly encounters high load, consider scaling options, such as using a cluster of mail servers or cloud solutions.

Conclusion

Increased server load directly affects website loading speed, ultimately impacting user experience and reputation in search engines. Thus, effectively managing this load plays a key role in ensuring continuous functionality of the resource and increasing its accessibility for visitors.