Changing you life, one Prz at a time...

Codename: ChiiServer

ChiiServer is the project codename for, well, a long long long time (>3.5 years) ago an abortive attempt to create an IRC server from scratch using Object Pascal (Delphi). Though a little ashamed, I have to mention that, at that time I did not even know how to properly read and write to a socket, not to mention synchronizing memory access multiple threads.

But, I saw the light of the all-mighty goddess Chii and came up with this great idea of "Process Level Application Server": a program that provides the basic infrastructure to accommodate multiple general purpose building block modules, and to accomplish one or several big tasks. ChiiServer has since become the codename for this SOE (Solution of Everything) project.

The major motif of this project is simply the result of my painful discovery from the textbook-like software engineering disaster of the aborted IRC server project: For a leisure-time programming project, once its size reaches certain "critical mass", it can never be done without a very very careful arrangement.

For one instance, you can spent a good chunk of 20 hours in one weekend and code 30% (say 5000 lines) of the program, then for the next a month and half, you got stuck dealing with something random in life. Then, by the time you came back to the project, the code already looks somewhat strange to you. You have to spend another 3-5 hours reading them until your "code cache" is warm again. With another 20 hours you manage to get 50% code done, however next time (God knows how many weeks later), it will take you longer to warm your cache, and give you less time to make progress. So on and so forth.

Sometimes even worse things can happen. If (and very likely, for the nature of leisure programs) you are experimenting with some new features / new styles / new design principles of programming, your code base is not stable. If not careful enough, you will end up with spaghetti-like code. Now after weeks, you have gain more knowledge / thought the problem more thoroughly / simply changed state of mind, and decide to modify something in your code. However, after five minutes, you discovered changing one place dictates changes to another place; after twenty five minutes, you found yourself making changes all over your code and your brain is cooking; after three and a half hours, you decided to scrap part of your scrambled code and rewrite... By the end of your second workshop with the project, your progress of the project ended up rolling back a couple of percent. (Now it's the perfect time to Prz...)

The holy water the ChiiServer project is designed to bring is the ability to divide an arbitrarily large project into small, leisure-time manageable modules, and interconnect them tightly to form a large project. The advantages are obvious: first, with simple functionality, fine-grained modules can each be coded and fully tested within a couple of hours, and, once they are done, they are done! No code re-reading necessary in the future as long as you do not intend to modify the functionality; since each modules are designed to be decoupled from the other, changing code of one module will not cause sources of others' to go unstable. In other words, you can keep almost constant speed in making progress, and never have to suffer from the painful progress roll-back phenomenon.

Here is an example to make the point obvious: let me re-imagine by failed IRC server project with the help of ChiiServer. Firstly, I need to write a server socket module, which creates listening sockets and signals messages when connection requests are received. Simple enough to code and test in an hour, right? A connection manager module is needed to associate each client socket with a IRC user, by processing the server socket module message. Easy task huh? Then, a client socket module is needed to establish connection and create (line based) client sockets, which again signals message when a line of text is received. Old school programming. An RFC compliant IRC server protocol module is needed to convert the raw text from client socket module messaged into meaningful operations and messages. It is probably the hardest module to code in the entire project, but the coding is almost mechanical - translating RFC into code, no-brainer. Finally, a channel module maintains record of all opened channels and presence of users in each channel; in addition it forwards public char messages to proper receivers in each channel, all according to the messages sent from the RFC module. Not a hard task to accomplish in 2 hours. These 5 modules accomplish the basic functionalities of an IRC server, and by easily writing and testing one module per week, you will have a prototype IRC server in 5 weeks. (Last time, without ChiiServer, mine was lurking at 30% by the end of the forth week, and I gave up... and of course, Prz).

But wait, there is even more that ChiiServer can provide!

  • Code reusability: well, you don't actually reuse code, but you reuse compiled and tested modules. For example, in the above imaginary ChiiServer based IRC server, the server and client socket modules are general purpose modules that can be used by other applications.
  • Online program update: since each modules are weakly coupled (by module name, and interfaced components), one can potentially unload a module while queuing the pending messages to that module, perform a binary patch, and then load back the module and let it continue processing messages. To other modules in the same application, this process is transparent and seamless. This means, you don't have to shutdown the entire program to update its functionality and/or fix errors.
  • Efficiency: Since all modules load under the same process, while messages are sent between threads, in essence only pointer exchanges are performed. Thus unlike inter-process communication that involves data copying and/or heavy user-kernel context switching, communication latency / cost in ChiiServer is minimal, which makes all modules tightly interconnected.

With all the foreseeable potentials, I started coding this great project. Ironically, what this project is set to resolve becomes the obstacles that prevents this project get finished. (Again... Prz) through over three years, I have officially recorded 6 failed attempts on this project. Now, it is the 7th take on it, and I am at the closest point ever of having a working core components. Looking back to all the abandoned source files, I can see significant improvements on both the design and the code maturity - indeed, I have learn more thing in attempting this project that in all other finished projects added together. The definition of message shifts from pure class to class base with interface support then to interface base with optional class implementation and delegation; the message queue evolves from critical section protected to lock-free; the message bus advances from linear array lookup and string match to hash lookup with GUID match. Thinking about all these improvements, had I finished coding the ChiiServer three years ago, it will be hundreds of times slower than the current implementation and I will have to recode them anyway...

As I have just said, right now I am very close to being done with the ChiiServer. According to my plan, this first modular application utilizing the ChiiServer architecture will be a tiny DNS server. Once finished, it will be used as the domain server of this domain for test. (And that will be the time I take off the FORSALE page ^_^) Then, maybe after I add the logging module, a tiny HTTP server will be built, and as well, it will be used to host WebPages for this domain. Then, my dream IRC server, and a media streaming cast server, etc. I believe with a strong and flexible ChiiServer, anything is possible!