Queues

Receive And Index Queue Intro

When MailArchiva receives an incoming mail from a mail server such as via an incoming SMTP connection, or manually via an import operation, the mail is assigned a randomly generated filename with a .blob extension and immediately written to disk in a receive queue. The receive queue is located in [application data]\queue\receive. At this point, the mail is not yet visible in the search results; it needs to be archived and indexed first. After writing the incoming mail to this temporary location, for tracking purposes, MailArchiva inserts the randomly generated ID into the database of the internal queuing engine database, whose support files are located at [application data]\queue\receive\ctl.

Once the message is queued, the queueing engine schedules the delivery of the item in the receive queue for archiving. After archiving has taken place, the data is inserted into a separate index queue for indexing. The index queue is located at [application data]\queue\index, and similarly, its associated database is located at  [application data]\queue\index\ctl. Since both indexing and archiving operations involve a considerable amount of parsing and computation when compared to writing raw incoming data to the receive queue, the archiving and indexing operations typically take comparatively more time to perform. Assuming the server receives a sudden spike in traffic, data can build up quickly in the receive and index queues. If the number of items in the queues exceeds the value for the max queue sizes specified in Configuration->Archive and Configuration->Index, for archive and index queues, respectively; upon processing a new archive request, the server will in most cases wait for at least two minutes for a slot to become available, failing which, it will reject the incoming SMTP connection with a retry later SMTP code such as 421, or 451. If the source of data is a manual data import, the import operation will typically be blocked until a slot in the queue becomes available. 

In unfortunate circumstances, the queuing engine may not be able to process items in the queue for archiving. This can happen if there is no available volume configured, the volume store happens to be offline due to an unstable mount point or volume is assigned incorrect permissions. In such situations, emails will simply build up in the queue until it is full, after which time, if the data is received via SMTP, the server will begin to reject traffic, usually in a manner that causes the sender to automatically retry later.

Adjusting Queue Sizes

Reducing the size of the receive (in Configuration->General->Archive) and index (specified in Configuration->General->Index) queues, limits the total amount of space that the queues may consume on disk. Given that the speed of reading source data is typically faster than the speed of encryption, compression + write, especially, with regards to imports, it is crucial to set an appropriate queue size, taking into account the amount of disk space available on the queue partition such that all available disk space cannot be consumed during an import operation.

Changing Queue Paths

It is good practice to locate queues on a separate partition on disk. The reason being, assuming queues share a partition with other MailArchiva program files, and the server suddenly receives a huge rush of traffic (e.g. DOS or spam attack), items may build up rapidly in the queue. If storage space becomes exhausted, the entire system may become unuseable as there are many components of MailArchiva, such as writing of temps files, that are reliant on a healthy supply of free disk space. To change the location of queues, edit the bootstrap.conf file.

How To Recover Lost Queue Items

In rare situations, the database associated with the queuing engine may get corrupted due to out-of-disk-space on the queue partition, or due to sudden power loss, or otherwise. In this eventuality, orphaned items may be present in the receive and index folders but cannot be processed, due to the fact that the queuing engine is not aware of them. To resolve this problem, stop the server and rename the directory [application data]\queue\receive to restore_receive and [application data]\queue\index to restore_index. When MailArchiva starts up, it will see that there are messages available in the restore_receive and restore_index folders, and will automatically process them. The same logic also applies to the smtp client queues too!

Restore Queues (Windows)

Restore Queues (Linux)

How To Preprocess Items in Err Folder

MailArchiva moves messages it cannot process due to non-compliance with the rfc-2822 spec to the queue/err folder. When processing messages, the system tries to be as lenient as possible. To reprocess the items in the err folder, take the following steps: