Overview of System Architecture
-------------------------------
The CIYAM sytem consists of several key applications and tools. Most of the tools are not relevant for use in
a production environment although the bundle and unbundle tools should be installed for usage with backing up
and restoring application data.
The essential components of the system are as follows:
Name Role/Description
--------------- --------------------------------------------------------
ciyam_client Single-threaded console client.
ciyam_interface Multi-threaded FCGI CIYAM client.
ciyam_server Multi-threaded application server.
upload Multi-threaded FCGI file uploader.
The following components are optional for production but are required for development:
Name Role/Description
--------------- --------------------------------------------------------
bundle Archiving tool used for packages/backups.
unbundle Archiving tool used for packages/restores.
The following components are required for development installations only:
Name Role/Description
--------------- --------------------------------------------------------
Meta Modelling/Application Generation module.
check_source Tool used for pre-commit source code format checking.
construct Used to construct template file outlines from packages.
diff Text file diff tool (similar to standard *nix tool).
diffp Diff patching tool (to patch one or merge two diffs).
extract Tool used to extract manual code from generated source.
generate_commands Tool to transform a .cms source file into .cmh include.
modeller Legacy modelling tool used to maintain the Meta model.
ods_dump Object Data Storage hex data dump tool.
ods_fsed Object Data Storage file system editing tool.
test Basic regression test utility.
test_cache Testbed for the generic cache implementation.
test_fcgi Test FCGI application (to check if FCGI is working).
test_hash_chain Testbed for the hash chain class.
test_numeric Testbed for the numeric class.
test_ods Testbed for the Object Data Storage system.
test_parser Testbed for the (RPC) command parser.
test_sql Test tool for issuing SQL queries.
xrep Expression replacement tool for expanding templates.
xvars Tool used by the make system.
The following batch/script files are required for development installations:
Name Role/Description
--------------- --------------------------------------------------------
config Generates config.h from config.h.xrep (and config.info).
create_db Creates an empty DB.
dotests Runs basic tool and system regression tests.
drop_db Drops an existing DB.
genmake Generates the project makefile.
genmodule Used explicitly to generate the Meta model.
Various other batch/script files that are used internally by development installations are not listed here.
Module and Storage Concepts
---------------------------
An application instance consists of one or more modules (which are shared/dynamic libraries) plus one storage
(which is one SQL and one ODS DB). The web interface (if present) is stored in an application directory below
that nominated by the WEBDIR environment variable.
As storage and modules are physically separate items modules can be shared by other application instances but
it should be noted that a storage can only be used by one application instance.
Application Server Basics
-------------------------
The application server is a multi-threaded program that concurrently handles clients that connect in order to
invoke RPC commands. Multiple storages can be used and shared by the various connected clients, however, each
client connection can itself only operate with the one storage at a time.
As well as direct storage operations via RPC commands the application server provides a framework that allows
class objects to be created/destroyed and sent commands. In this manner actual DB operations are performed by
clients without invoking any direct SQL (i.e. automatic object to relational DB mapping).
In line with the CIYAM methodology all SQL queries are generated (via object methods) and therefore "raw" SQL
should never be found coded in any module (excluding DDL/SQL generated via the class template itself).
Storage Log File
----------------
An essential part of the storage approach is the log file named <storage>.log which is a transaction log that
contains every *write* operation performed with the storage. Immediately before a transaction is committed to
the storage the log file is appended to with the relevant protocol command(s).
It should be noted that storage operations arising from objects being used by code within the modules (unless
being performed via asynchronous script) will not appear in the storage log which helps reduce the file size.
When a storage is being "restored" the log is processed from the last known transaction id (or from the first
if performing a "rebuild") with each protocol command logged being repeated.
This may seem unnecessary when using a SQL DB but the approach actually plays an important role for upgrading
applications in that fields that depend upon operations with other fields (or with the user and the date/time
the operation occurred at) can be introduced at any time and behave as though they were there since the first
transaction was occurring (e.g. you can add audit fields to a class that didn't have them and get the correct
values for user and date/time with objects that were created/changed prior to the upgrade).
This also implies that the SQL transaction log itself is not essential and indeed it's recommended that RDBMS
settings be altered in order to automatically truncate the log (and for best possible performance tuning even
switch it off altogether).
Most essentially the storage log needs to be backed up regularly and although log truncation is permitted the
entire log must be able to be made available in order to do a "rebuild" (which is required for an upgrade).
Uploaded and Rules Created Files
--------------------------------
Files that application users upload are not kept in the storage (only there existence is) so another key part
of backing up is to ensure all of the application files have been periodically backed up. The provided backup
batch/script files can be used to include these external files, however, this is not advisable for situations
where the amount or size of these files is expected to be significant (for such applications it would instead
be recommended to use specialised backup software).
All such storage files are kept in subdirecties below the WEBROOT/<storage>/files directory and so any backup
software needs to be configured to backup that directory branch (excluding WEBROOT/<storage>/files/tmp branch
which is used for creating temporary links/files).
Files can be created in this directory branch even for applications that do not allow end users to upload any
files as they may be created via application rules. For example it is essential that Email applications store
copies of the original emails that were fetched so that the application objects will be correctly constructed
when a "restore" is in progress.
