Ever so often I get to set up hosts for running PHP. When running a load balanced solution, you have more hosts and reading logs gets complicated, development gets tedious. So what helps, is a central logging server. This is pretty easy to set up with syslog-ng, however PHP has a annoying habit of logging everything with the NOTICE error level.
There are times, when we get an application and need to find out what it does fast. We don’t have the time to read the source code. Fortunately there are multiple tools to our rescue, one of which is the strace Linux utility. strace means system call trace, it shows us every system call the application does, such as opening or reading a file, writing data to a network socket. It’s not a magic pill, it won’t show the internal working of the application, but it’s still very useful to find out what it does externally (IO operations and such).
During my time as a sysop and later as CTO I had quite a few e-mail servers under me. Over 50 to be exact. These servers were not standalone ones, but passed on e-mails to each other. We designed the system to avoid bottlenecks and make it easily extendable. It fully met the expectations in this respect, it was very easy to plug additional nodes into the system.
There was however an other aspect, where the system was not so great. And as a matter of fact, I haven’t seen any e-mail system, that was great at this. What I mean is tracking e-mails, debugging problems. Having to SSH into just a fraction of this many servers or reading logs from this amount of servers is a really, really painful way to do it.