Overview of System Architecture

The CIYAM sytem consists of several key applications and tools. Most of the tools are not relevant for use in a production environment although the bundle and unbundle tools should be installed for usage with backing up and restoring application data.

The essential components of the system are as follows:

Name Role/Description
ciyam_client Single-threaded console client.
ciyam_interface Multi-threaded FCGI CIYAM client.
ciyam_server Multi-threaded application server.
upload Multi-threaded FCGI file uploader.
xrep Expression replacement tool for expanding templates.

The following components are optional for production but are required for development:

Name Role/Description
bundle Archiving tool used for packages/backups.
unbundle Archiving tool used for packages/restores.

The following components are required for development installations only:

Name Role/Description
Meta Modelling/Application Generation module.
check_source Tool used for pre-commit source code format checking.
construct Used to construct template file outlines from packages.
diff Text file diff tool (similar to standard *nix tool).
diffp Diff patching tool (to patch one or merge two diffs).
extract Tool used to extract manual code from generated source.
generate_commands Tool to transform a .cms source file into .cmh include.
modeller Legacy modelling tool used to maintain the Meta model.
ods_dump Object Data Storage hex data dump tool.
ods_fsed Object Data Storage file system editing tool.
test Basic regression test utility.
test_cache Testbed for the generic cache implementation.
test_fcgi Test FCGI application (to check if FCGI is working).
test_hash_chain Testbed for the hash chain class.
test_numeric Testbed for the numeric class.
test_ods Testbed for the Object Data Storage system.
test_parser Testbed for the (RPC) command parser.
test_sio Testbed for Structured I/O files.
test_sql Test tool for issuing SQL queries.
xvars Tool used by the make system.

The following bash script files are required for development installations:

Name Role/Description
config Generates config.h from config.h.xrep (and config.info).
create_db Creates an empty DB.
drop_db Drops an existing DB.
genmake Generates the project makefile.
genmodule Used explicitly to generate the Meta model.
tests Runs basic tool and system regression tests.

Various other bash script files that are used internally by development installations are not listed here.

Module and Storage Concepts

An application instance consists of one or more modules (which are shared/dynamic libraries) plus one storage (which is one SQL and one ODS DB). The web interface (if present) is stored in an application directory below that nominated by the WEBDIR environment variable.

As storage and modules are physically separate items modules can be shared by other application instances but it should be noted that a storage can only be used by one application instance.

Application Server Basics

The application server is a multi-threaded program that concurrently handles clients that connect in order to invoke RPC commands. Multiple storages can be used and shared by the various connected clients, however, each client connection can itself only operate with the one storage at a time.

As well as direct storage operations via RPC commands the application server provides a framework that allows class objects to be created/destroyed and sent commands. In this manner actual DB operations are performed by clients without invoking any direct SQL (i.e. automatic object to relational DB mapping).

In line with the CIYAM methodology all SQL queries are generated (via object methods) and therefore “raw” SQL should never be found coded in any module (excluding DDL/SQL generated via the class template itself).

Storage Log File

An essential part of the storage approach is the log file named <storage>.log which is a transaction log that contains every write operation performed with the storage. Immediately before a transaction is committed to the storage the log file is appended to with the relevant protocol command(s).

It should be noted that storage operations arising from objects being used by code within the modules (unless being performed via asynchronous script) will not appear in the storage log which helps reduce the file size. When a storage is being “restored” the log is processed from the last known transaction id (or from the first if performing a “rebuild”) with each protocol command logged being repeated.

This may seem unnecessary when using a SQL DB but the approach actually plays an important role for upgrading applications in that fields that depend upon operations with other fields (or with the user and the date/time the operation occurred at) can be introduced at any time and behave as though they were there since the first transaction was occurring (e.g. you can add audit fields to a class that didn’t have them and get the correct values for user and date/time with objects that were created/changed prior to the upgrade).

This also implies that the SQL transaction log itself is not essential and indeed it’s recommended that RDBMS settings be altered in order to automatically truncate the log (and for best possible performance tuning even switch it off altogether).

Most essentially the storage log needs to be backed up regularly and although log truncation is permitted the entire log must be able to be made available in order to do a “rebuild” (which is required for an upgrade).

Uploaded and Rules Created Files

Files that application users upload are not kept in the storage (only there existence is) so another key part of backing up is to ensure all of the application files have been periodically backed up. The provided backup bash script files can be used to include these external files, however, this is not recommended in situations where the amount or size of these files is expected to be significant (for such applications it would instead be recommended to use a more specialised backup implementation).

All such storage files are kept in subdirecties below the $WEBDIR/<storage>/files directory and so any backup software needs to be configured to backup that directory branch (excluding $WEBDIR/<storage>/files/tmp branch which is used for creating temporary links/files).

Files can be created in this directory branch even for applications that do not allow end users to upload any files as they may be created via application rules. For example it is essential that Email applications store copies of the original emails that were fetched so that the application objects will be correctly constructed when a “restore” is in progress.