How long is a piece of string

Sunday morning I was asked by an IRC regular: "Where does the engine parse quoted strings?". Being a sunday morning, I began to launch into a sermon on the distinction between CONSTANT_ENCAPSED_STRING and the problems which befall a single-pass compiler when you start to introduce interpolation. Not what he asked precisely, but an important component in answering his question. Unfortunately, at the time I was busy watching the Brasil-Australia game so I didn't go into the kind of detail I would have. Now, some 12 hours later, since Angela is off buying toe-socks in Santa Cruz, I'll bore anyone with little enough life to read my blog by explaining the pitfalls of using PHP's string interpolation without using an optimizer.

To start things off, let's take a page from my earlier discourse on Compiled Variables and look at the opcodes generated by a few simple PHP scripts:

<?php
echo "This is a constant string";
?>

Yields the nice, simple opcode:

ECHO            'This is a constant string'

No problem... Exactly what you'd expect... Now let's complicate the expressions a little:

<?php
echo "This is an interpolated $string";
?>

Yields the surprisingly messy instruction set:

INIT STRING  ~0
ADD_STRING   ~0 ~0 'This'
ADD_STRING   ~0 ~0 ' '
ADD_STRING   ~0 ~0 'is'
ADD_STRING   ~0 ~0 ' '
ADD_STRING   ~0 ~0 'an'
ADD_STRING   ~0 ~0 ' '
ADD_STRING   ~0 ~0 'interpolated'
ADD_STRING   ~0 ~0 ' '
ADD_VAR      ~0 ~0 !0
ECHO            ~0

Where !0 represents the compiled variable named $string. Looking at these opcodes: INIT_STRING allocates an IS_STRING variable of one byte (to hold the terminating NULL). Then it's realloc'd to five bytes by the first ADD_STRING ('This' plus the terminating NULL). Next it's realloc'd to six bytes in order to add a space, then again to eight bytes for 'is', then nine to add a space, and so on until the temporary string has the contents of the interpolated variable copied into its contents before being used by the echo statement and finally discarded. Now let's rewrite that line to avoid interpolation and use concatenation instead:

<?php
echo "This is a concatenated " . $string;
?>

Which yields the significantly shorter and simpler set of ops:

CONCAT       ~0 'This is a concatenated ' !0
ECHO            ~0

A vast improvement already, but this version still creates a temporary IS_STRING variable to hold the combined string contents meaning that data is duplicated when it's being used in a const context anyway. Now let's try out this oft-overlooked use of the echo statement:

<?php
echo "This is a stacked echo " , $string;
?>

Look close, there is a meaningful difference from the last one. This time we're using a comma rather than a dot between the operands. If you don't know what the comma is doing there, ask the manual then check back here. Here's the resulting opcodes:

ECHO            'This is a stacked echo '
ECHO            !0

Same number of opcodes, but this time no temporary variables are being created so there's no duplication and no pointless copying (unless of course $string wasn't of type IS_STRING, in which case it does have to be converted for output, but don't get picky now). Think this is bad? Consider the average heredoc string which spans several lines of prepared output embedding perhaps a handful of variables along the way. Here's one of several such blocks found in run-tests.php within the PHP distribution source tree:

<?php
echo <<NO_PCRE_ERROR

+-----------------------------------------------------------+
|                       ! ERROR !                           |
| The test-suite requires that you have pcre extension      |
| enabled. To enable this extension either compile your PHP |
| with --with-pcre-regex or if you've compiled pcre as a    |
| shared module load it via php.ini.                        |
+-----------------------------------------------------------+

NO_PCRE_ERROR;
?>

Notice that we're not even embedding variables to be interpolated here, yet does this come out to a simple, single opcode? Nope, because the rules necessary to catch a heredoc's end token demand the same careful examination as double-quoted variable substitution and you wind up (in this case) with SEVENTY-EIGHT opcodes! One INIT_STRING, 76 ADD_STRINGs. and a final ECHO. That means a malloc, 76 reallocs, and a free which will be executed every time that code snippet comes along. Even the original contents take up more memory because they're stored in 76 distinct zval/IS_STRING structures.

Why does this happen? Because there are about a dozen ways that a variable can be hidden inside an interpolated string. Similarly, when looking for a heredoc end-token, the token can be an arbitrary length, containing any of the label characters, and may or may not sit on a line by itself. Put simply, it's too difficult to encompass in one regular expression.

The engine could perform a second-pass during compilation, however the time saved reassembling these strings will typically be about the same amount of time spent actually processing them during runtime (if one assumes that each instance will execute exactly once). Rather than complicate the build process (potentially slowing down overall run-times in the process), the compiler leaves this optimization step to opcode caches which can achieve exponentially greater advantage cleaning up this mess then caching the results and reusing the faster, leaner versions on all subsequent runs.

If you're using APC, you'll find just such an optimizer built in, but not enabled by default. To turn it on, you'll need to set apc.optimization=on in your php.ini. In addition to stitching these run-on opcodes back together, it'll also add run-time speed-ups like pre-resolving persistent constants to their actual values, folding static scalar expressions (like 1 + 1) to their fixed results (e.g. 2), and simpler stuff like avoiding the use of JMP when the target is the next opcode, or boolean casts when the original expression is known to be a boolean value. (It should be noted that these speed-ups also break some of the runtime-manipulation features of runkit, but that was stuff you....probably should have been doing anyway)

Can't use an optimizer because your webhost doesn't know how to set php.ini options? You can still avoid 90% of the INIT_STRING/ADD_STRING dilema by simply using single quotes and concatenation (or commas when dealing with echo statements). It's a simple trick and one which shouldn't harm maintainability too much, but on a large, complicated script, you just might see an extra request or two per second.

Extending and Embedding PHP

It's official!!!! After a year in development Extending and Embedding PHP is now shipping from fine book stores everywhere.

I've gotten good reviews from the half dozen people I know who've gotten their hands on it, and I am really satisfied with most of it. Do I think it could be better? Of course I do, but I don't think I was ever going to be completely satisfied.

I've learned a lot through this process and while I don't see any more titles in my immediate future (there are things coming down the pipe which are likely to change my availability), I do expect that my next book, should it materialize, will be even better.

If you pre-ordered it, you should see it soon, if you've already got your copy, let me know what you think! Did I skim over some topic too quickly? Did I belabour something else? If this book eventually finds it's way into a 2^nd edition (no promises mind you), are there topics you want to see added? Tossed out? Expanded/Compressed?

What the heck is TSRMLS_CC, anyway?

If you've ever worked on the PHP internals or built an extension, you've seen this construct floating around here and there, but noone ever talks about it. Those who know what this is typically answer questions from those who don't with "Don't worry about what it is, just use it here here here and here. And if the compiler says you're missing a tsrm_ls, put it there too..." This isn't laziness on the part of the person answering the question (okay, maybe it is a little bit), it's just that the engine goes so far out of its way to simplify what this magic values does, that there's no profit in a new extension developer knowing the mechanics of it. The information is like a cow's opinion, it doesn't matter, it's Moo.

Since I love to listen to myself rattle on about pointless topics (and I havn't blogged much this month), I thought I'd cover this topic and see if anyone manages to stay awake through it. You can blame Lukas, he got me rolled onto planet-php.net...

Glossary

TSRM

Thread Safe Resource Manager - This is an oft overlooked, and seldom if ever discussed layer hiding in the /TSRM directory of your friendly neighborhood PHP source code bundle. By default, the TSRM layer is only enabled when compiling a SAPI which requires it (e.g. apache2-worker). All Win32 builds have this layer enabled enabled regardless of SAPI choice.

ZTS

Zend Thread Ssafety - Often used synonymously with the term TSRM. Specifically, ZTS is the term used by ./configure ( --enable-experimental-zts for PHP4, --enable-maintainer-zts for PHP5), and the name of the #define'd preprocessor token used inside the engine to determine if the TSRM layer is being used.

tsrm_ls

TSRM local storage - This is the actual variable name being passed around inside the TSRMLS_* macros when ZTS is enabled. It acts as a pointer to the start of that thread's independent data storage block which I'll cover in just a minute

TSRMLS_??

A quartet of macros designed to make the differences between ZTS and non-ZTS mode as painless as possible. When ZTS is not enabled, all four of these macros evaluate to nothing. When ZTS is enabled however, they expand out to the following definitions:

• TSRMLS_C tsrm_ls
• TSRMLS_D void ***tsrm_ls
• TSRMLS_CC , tsrm_ls
• TSRMLS_DC , void ***tsrm_ls

Globals

In any normal C program (just like in PHP) you have two methods of getting data access to the same block of data in two different functions. One method is to pass the value on the parameter stuck like so:

#include 

void output_func(char *message)
{
   printf("%s\n", message);
}

int main(int argc, char *argv[])
{
   output_func(argv[0]);

   return 0;
}

Alternately, you could store the value in a variable up in the global scope and let the function access it there:

#include 

char *message;

void output_func(void)
{
   printf("%s\n", message);
}

int main(int argv, char *argv[])
{
   message = argv[0];
   output_func();

   return 0;
}

Both approaches have their merits and drawbacks and typically you'll see some combination of the two used in a real application. Indeed, PHP is covered in global variables from resource type identifiers, to function callback pointers, to request specific information such as the symbol tables used to store userspace variables. Attempting to pass these values around in the parameter stack would be more than unruly, it'd be impossible for an application like PHP where it's often necessary to register callbacks with external libraries which don't support context data.

So common information, like the execution stack, the function and class tables, and extension registries all sit up in the global scope where they can be picked up and used at any point in the application. For single-threaded SAPIs like CLI, Apache1, or even Apache2-prefork, this is perfectly fine. Request specific structures are initialized during the RINIT/Activation phase, and reset back to their original values during the RSHUTDOWN/Deactivation phase in preparation for the next request. A given webserver like Apache1 can serve up multiple pages at once because it spawns multiple processes each in their own process space with their own independant copies of global data.

Now let's introduce threaded webservers like Apache2-worker, or IIS. Under these conditions, only one process space is active at a given time with multiple threads spun off. Each of these threads then act in the same manner as a single-threaded process might; Servicing requests one-at-a-time as dispatched by inbound requests. The trouble starts to brew as two or more threads try to service the a request at the same time. Each thread wants to use the global scope to store its request-specific information, and tries to do so by writing to the same storage space. At the least, this would result in userspace variables declared in one script showing up in another. In practice, it leads to quick and disasterous segfaults and completely unpredictable behavior as memory is double freed or written with conflicting information by separate threads.

Non-Global Globals

The solution is to require the engine, the core, and any extension using global storage to determine how much memory will be used by request-specific data. Then, at the spin-up of each new thread, allocate a chunk of memory for each of these players to store their data into thus giving each thread its own local storage. In order to group all the individual chuncks used by a given thread together, one last vector of pointers is allocated to store the individual sub-structure pointers into. It's the pointer to this vector which is passed around as the tsrm_ls variable by the TSRMLS_* family of macros. To see how this works, let's look at a example extension:

typedef struct _zend_myextension_globals {
   int foo;
   char *bar;
} zend_myextension_globals;

#ifdef ZTS
int myextension_globals_id;
#else
zend_myextension_globals myextension_globals;
#endif

/* Triggered at the beginning of a thread */
static void php_myextension_globals_ctor(zend_myextension_globals *myext_globals TSRMLS_DC)
{
   myext_globals->foo = 0;
   myext_globals->bar = NULL;
}

/* Triggered at the end of a thread */
static void php_myextension_globals_dtor(zend_myextension_globals *myext_globals TSRMLS_DC)
{
   if (myext_globals->bar) {
       efree(myext_globals->bar);
   }
}

PHP_MINIT_FUNCTION(myextension)
{
#ifdef ZTS
   ts_allocate_id(&myextension_globals_id, sizeof(zend_myextension_globals),
                  php_myextension_globals_ctor, php_myextension_globals_dtor);
#else
   php_myextension_globals_ctor(&myextension_globals TSRMLS_CC);
#endif

   return SUCCESS;
}

PHP_MSHUTDOWN_FUNCTION(myextension)
{
#ifndef ZTS
   php_myextension_globals_dtor(&myextension_globals TSRMLS_CC);
#endif

   return SUCCESS;
}

Here you can see the extension declaring its global requirements to the TSRM layer by stating that it needs sizeof(zend_myextension_globals) bytes of storage, and providing callbacks to use when initializing (or destroying) a given thread's local storage. The value populated into myextension_globals_id represents the offset (common to all threads) into the tsrm_ls vector where the pointer to that thread's local storage can be found. In the event that ZTS is not enabled, the data storage is simply placed into the true global scope and the thread initialization and shutdown routines are called manually during the Module's Startup and Shutdown phases. If you're wondering why TSRMLS_CC was included in the non-ZTS blocks, then I clearly havn't made you fall asleep yet. Those aren't needed there since we know they evaluate to nothing, but it helps encourage good habits to include them anywhere the function's prototype calls for them.

Putting it all together

The final piece of this thread-safe puzzle comes from the question: "How do I access data in these structures?" And the answer to that question comes in the form of another familiar looking macro. Each extension or core component defines, in one of its header files, a macro which looks something like the following:

#ifdef ZTS
# define   MYEXTENSION_G(v)     \
             (((zend_myextension_globals*)(*((void ***)tsrm_ls))[(myextension_globals_id)-1])->v)
#else
# define   MYEXTENSION_G(v)     (myextension_globals.v)
#endif

Thus, when ZTS is not enabled, this macro simply plucks the right value out of the imediate value in the global scope, otherwise it uses the ID to locate the thread's local storage copy of the structure and derefence the value from there.

Wanna know more, like how to deal with foreign callbacks where tsrm_ls isn't available? Buy my book!

Compiled Variables

Last month at php|tek I gave a presentation on "How PHP Ticks" where I covered, among other things the process of compiling source code into opcodes (an intermediate psuedo-language similar to what java calls "bytecode" or what .NET calls "MSIL"). As part of this section of the presentation, I showed one of the more interresting changes between ZE 2.0 (PHP 5.0) and ZE 2.1 (PHP 5.1), namely: How variables are retreived and used in an operation. More specifically, how they provide a small, yet cumulative, speedup to applications in a way that's transparent to the end-user -- One more reason to like PHP 5.1 right?

After listening to Marcus Whitney's interview with Brion Vibber of WikiMedia in which he mentions my presentation and makes reference to this engine change, I realized that I should clarify what this feature is (and more importantly, what it isn't) before any FUD spreads.

What Compiled Variables (CVs) are

First, let's look at the anantomy of an OpArray. Say you have the following simple script:

<?php
$a = 123;
$b = 456;
$c = $a + $b;
echo $c;

Now let's see how ZE 2.0 (PHP5.0) compiles this (ZE1.x/PHP4.x comes out to nearly identical opcodes). The $0 and ~0 references you see are (for lack of a better one sentence explanation) types of temporary variables (the latter moreso than the former, but don't worry about the distinction right now). What's important to know about this block and its statements are in the accompaning comments:

FETCH_W                  $0, 'a'          /* Retreive the $a variable for writing */
ASSIGN                   $1, $0, 123      /* Assign the numeric value 123 to retreived variable 0 */
FETCH_W                  $2, 'b'          /* Retreive the $b variable for writing */
ASSIGN                   $3, $2, 456      /* Assign the numeric value 456 to retreived variable 2 */
FETCH_R                  $5, 'a'          /* Retreive the $a variable for reading */
FETCH_R                  $6, 'b'          /* Retreive the $b variable for reading */
ADD                      ~7, $5, $6       /* Add the retreived variables (5 & 6) to gether and store the result in 7 */
FETCH_W                  $4, 'c'          /* Retreive the $c variable for writing */
ASSIGN                   $8, $4, ~7       /* Assign the value in temporary variable 7 into retreived variable 4 */
FETCH_R                  $9, 'c'          /* Retreive the $c variable for reading */
ECHO                         $9           /* Echo the retreived variable 9 */

Seem like a lot of work for one plus one? It is, here's the same code snippet compiled by ZE 2.1/PHP 5.1 (or later).

ASSIGN                   $0, !0, 123      /* Assign the numeric value 123 to compiled variable 0 */
ASSIGN                   $1, !1, 456      /* Assign the numeric value 456 to compiled variable 1 */
ADD                      ~2, !0, !1       /* Add compiled variable 0 to compiled variable 1 */
ASSIGN                   $3, !2, ~2       /* Assign the value of temporary variable 2 to compiled variable 2 */
ECHO                         !2           /* Echo the value of compiled variable 2 */

These !0 variables refer to a new structure in the execution stack which stores and references to the "real" variables out in userspace. The hash value for each variable is computed at compile time (meaning that it's only done once per variable no matter how often it's used and that opcode caches save this work from being done during subsequent page views at all). The first time one of these CVs is used, the engine looks it up in the active symbol table and updates the CV cache to know where it is. All subsequent uses of that compiled variable use that pre-fetched address and don't have to look it up again. On an individual lookup, this isn't a major leap forward in speed, however consider a for loop where the test value is checked on every iteration; To put it in PHP terms, which would you rather do?

for($i = 0; $i < foo =" lookup_variable('foo');">increment();
$foo = lookup_variable('foo');
$foo->check_value();
}

or

$foo = lookup_variable('foo');
for($i = 0; $i <>increment();
$foo->check_value();
}

What compiled variables are not

Don't assume you're going to get a speedup on all of your code, especially if you use arrays or objects (which most code taking advantage of PHP5's new features does). The CV speedup has one minor achilles heel: It only works on simple variables. Putting it in terms of opcodes, let's consider this PHP source:

<?php
$f->a = 123;
$f->b = 456;
$f->c = $f->a + $f->b;
echo $f->c;
?>

Basicly the same code right? Just a little oopified... Let's look at the ZE2.1/PHP5.1 compilation of that:

ASSIGN_OBJ                $0, !0, 'a'     /* Assign the numeric value 123 to property 'a' of compiled variable 0 object */
OP_DATA                       123         /* Additional data for ASSIGN_OBJ opcode */
ASSIGN_OBJ                $1, !0, 'b'     /* Assign the numeric value 456 to property 'b' of compiled variable 0 object */
OP_DATA                       456         /* Additional data for ASSIGN_OBJ opcode */
FETCH_OBJ_R               $3, !0, 'a'     /* Retreive property 'a' from compiled variable 0 object */
FETCH_OBJ_R               $4, !0, 'b'     /* Retreive property 'b' from compiled variable 0 object */
ADD                       ~5, $3, $4      /* Add those values and store the result in temp var 5 */
ASSIGN_OBJ                $2, !0, 'c'     /* Assign the ADD result to property 'c' of compiled variable 0 object */
OP_DATA                       ~5          /* Additional data for ASSIGN_OBJ opcode */
FETCH_OBJ_R               $6, !0, 'c'     /* Retreive property 'c' from compiled variable 0 object */
ECHO                          $6          /* Echo the value */

What's important to see here is that the properties are refetched each time a read or write is performed on them, which at first glance looks as bad as the pre PHP5.1 way of dealing with variables. Don't let your enthusiam for compiled variables blind you though. Remember the magic __get(), __set(), __offsetget(), and __offsetset() methods which objects allow for. These overloading tricks are great, but they mean that the variable returned by one fetch may not be the variable returned by a subsequent fetch. It's unfortunate that this can't be guaranteed, but it's the reality of a dynamic language like PHP. Know your particular class isn't overloaded? You can get that speedup (at least some of it) back by using good 'ol references to turn your object variables into simple variables:

<?php
$a = &$f->a;
$b = &$f->b;
$c = &$f->c;
$a = 123;
$b = 456;
$c = $a + $b;
echo $c;
?>

Becomes:

FETCH_OBJ_W               $0, !1, 'a'     /* Retreive property 'a' from compiled variable 1 object */
ASSIGN_REF                $1, !0, $0      /* Make compiled variable 0 a reference to the retreived variable */
FETCH_OBJ_W               $2, !1, 'b'     /* Retreive property 'b' from compiled variable 1 object */
ASSIGN_REF                $3, !2, $2      /* Make compiled variable 2 a reference to the retreived variable */
FETCH_OBJ_W               $4, !1, 'c'     /* Retreive property 'c' from compiled variable 1 object */
ASSIGN_REF                $5, !3, $4      /* Make compiled variable 3 a reference to the retreived variable */
ASSIGN                    $6, !0, 123     /* Assign the numeric value 123 to compiled variable 0 */
ASSIGN                    $7, !2, 456     /* Assign the numeric value 456 to compiled variable 2 */
ADD                       ~8, !0, !2      /* Add compiled variable 0 to compiled variable 2 */
ASSIGN                    $9, !3, ~8      /* Assign the value of temporary variable 8 to compiled variable 3 */
ECHO                          !3          /* Echo the value of compiled variable 3 */

Now, this particular example is actually a few more opcodes, and a little bit more work for the engine too, but the more your code following this point uses the local/simple-variable/reference copies rather than the object copies, the balance will quickly tip towards your favor because the variables are already fetched, and they don't need to go through the expensive re-fetch process (which is worse for objects than it ever was for regular variables).

Another caveat to CVs, is that they are entirely scope-local. This should make since as $a in the globals scope is not the same as $a in a given function. What this means for your script, is that when execution enters a new function (execution scope) the CVs for that function are a blank slate and everything has to be fetched anew, even if that function was called before.

Globals and Statics

Statics and Globals are treated to CV status, but only by way of the reference trick I just mentioned:

<?php
static $bar;
echo $bar;

Turns into:

FETCH_W      static      $0, 'bar'
ASSIGN_REF                   !0, $0
ECHO                         !0

What does that mean for your use of the $GLOBALS array? That's right, the global keyword is technically faster. Now, I want to be really clear about one thing here. The minor speed affordance given by using your globals as localized CVs needs to be seriously weighed against the maintainability of looking at your code in five years and knowing that $foo came from the global scope. something_using($GLOBALS['foo']); will ALWAYS be clearer to you down the line than global $foo; /* buncha code */ something_using($foo); Don't be penny-wise and pound foolish..

No, seriously. This post shouldn't be taken as a guide-book to speeding up your apps, they're slow for other reasons.

Doctor! Doctor! It hurts when I do this!

I came across a bug while working on runkit today. On testing a batch of changes I'd made I was suddenly* running into a buffer overrun error on any request which instantiates the Runkit_Sandbox class. A few valgrind and gdb rounds later and I'd traced the corruption to apc.enabled's ini_entry->value which was actually encountering a form of double-free.

Now, I have no illusions regarding the fact that runkit is a black sheep. The things it does are contrary to PHP's design and probably shouldn't be done, so my first assumption is that APC is not the one at fault and that runkit is just coincidentally stepping on APC's toes. Time to break out the caffeine and get dirty...

After a little more digging I notice this block of code in APC:

PHP_MINIT_FUNCTION(apc)
{
   ZEND_INIT_MODULE_GLOBALS(apc, php_apc_init_globals, php_apc_shutdown_globals);

   REGISTER_INI_ENTRIES();

   /* Disable APC in cli mode unless overridden by apc.enable_cli */
   if(!APCG(enable_cli) && !strcmp(sapi_module.name, "cli")) {
       zend_alter_ini_entry("apc.enabled", sizeof("apc.enabled"), "0", 1,
                       PHP_INI_SYSTEM, PHP_INI_STAGE_ACTIVATE);
   }

   if (APCG(enabled)) {
       apc_module_init(module_number TSRMLS_CC);
   }

   return SUCCESS;
}

Ignore for the moment the fact that it's claiming we're in the ACTIVATE stage (we're really in the STARTUP phase -- but that's not where the problem lies). The problem with this statement is that zend_alter_ini_entry() uses estrndup(), a call which assumes that thread storage has been spun up and that the per-request pointer list is ready for the new entry to be indexed. During MINIT, that's just not the case.

The "right" solution for this is to move the CLI mode override to RINIT and sure enough, this fixes the overrun/double-free issues just fine. The trouble is, it's also a wasteful proposition. Compound this by the fact that the error only presents itself when all of the following conditions are met:

• APC being used on command line
• apc.enable_cli switch not set
• Thread Safety turned on (requires --enable-maintainer-zts during bulid)

Should APC take a performance hit in order to satisfy such an edge case scenario? &#@% NO! This is one bug that can be casually swept under the rug. Move along now, nothing to see here.

P.S. - In case you were wondering, the only reason I didn't notice this before is that I'd only just recently added APC to my development build. The coincidence of this bug showing up with this last batch of changes was precisely that, a coincidence.

re2c is no lemon

For the longest time I've been wanting to learn how to actually write a compiler. Sure, I've been twiddling the Zend Engine here and there for a few years, but there's this gulf between tweaking and actually building up from scratch which, until today, I had just never found my way across. Enter PDO_User, my most recent extension for implementing PDO drivers in userspace. Although some uses of PDO_User might be as simple as wrapping real database drivers that don't have a PDO version (e.g. informix), there's also a class of userspace implementation that might want to perform decidedly non-SQL actions via a SQL front end.

Since writing a compiler in userspace is... difficult to say the least, I decided this was the perfect opportunity to sharpen my teeth on the task and provide a generic SQL compiler via the PDO_User class. For the lexer I chose re2c as Marcus has been pushing that and what I'd seen of the syntax made it look pretty straight forward, for the parser I went with lemon which Wez is generally ga-ga over. I considerd flex and yacc, but tossed them out since I've already got a basic understanding of their syntaxes. I wanted a start fresh.

After a couple-three days and some aborted attempts, I've managed to implement a relatively feature-rich SQL tokenizer and compiler accessible to userspace via array PDO_User::tokenizeSQL($sql[, $ignore_whitespce=true]) and array PDO_User::parseSQL($sql). Those of you intimately familiar with SQL specs will certainly find faults in my grammar, and just about everyone should notice that there's no support for floating point numbers yet (a minor oversight). What's important is: (A) I've got a working implementation to build from, and (B) It was damned easy to learn both these tools.

Some thoughts on re2c and lemon:

• Not enough samples provided on the manual pages. As an OSS documentation maintainer myself I know that this is easier to complain of than actually do something about.
• I got stuck early on with re2c due to having an old version (the one packaged with debian) which didn't support case-insensitive matching. If you run into this problem, check your re2c version. If `re2c -v` isn't even able to output a version, then it's certainly too old.
• What the @!#& is the "error" non-terminal for in lemon? The documentation talks about using it to gracefully deal with parse errors, but doesn't talk about how to use it.

If all you want is some sample output, have a look at this:

<?php
  print_r(PDO_User::parseSQL("SELECT foo,bar as b,baz
                            FROM boop LEFT JOIN doop as d
                            ON (bid=did)"));


Array
(
  [type] => statement
  [statement] => select
  [fields] => Array
    (
      [0] => foo
      [1] => Array
        (
          [type] => alias
          [field] => bar
          [as] => b
        )
      [2] => baz
    )
  [from] => Array
    (
      [type] => join
      [table1] => boop
      [join-type] => left
      [table2] => Array
        (
          [type] => alias
          [table] => doop
          [as] => d
        )
      [on] => Array
        (
          [type] => condition
          [op1] => bid
          [condition] => =
          [op2] => did
        )
    )

  [modifiers] =>
  [terminating-semicolon] =>
)

PDO_User and the effects of Slide Driven Development

Last Fall, Wez and I (and others) met up at the Zend conferrence in Burlingame, CA (US). This was primarily a pre-PDM checkin for those of us who wouldn't be able to make it to Paris to get our thoughts in on the direction of PHP6. Apart from these topics, Wez mentioned wanting to have a userspace bridge for implementing PDO drivers, but that there was entirely too much going on that was (in all honesty) more important.

I agreed on both counts. At the time I was doing my best to ignore the temptation to work on operator... Click through and you'll see just how weak willed I can be :)

Fast forward to last week in Orlando, FL (US) and the php|tek conferrence. As mentioned in earlier blog posts, I think my presentations went over okay for a first couple of talks, but if I have to make one complaint, it's that the subject matter was ill-aimed. I spoke to Sean (one of the conferrence organizers) after the closing keynote and asked him about the upcoming php|works conf in toronto. When he said that one of the focuses would be on database interaction (since it would be taking place along side db|works), my thoughts immediately jumped to the PDO userspace driver which had been running around the back of my mind for months.

By the end of my flight back I had half the implementation written and had generally come to the conclusion that the PDO API was pretty damned nice. Long story short, I've just uploaded release 0.1 of pdo_user which is ready for testing. The file you'll want to pay attention to is: README.OBJECTS as that describes the mechanics of actually implementing a driver in userspace. Another interresting spot to look is tests/globals.phpt which includes a simple, yet functional userspace driver for accessing variable contents as database resources.

I don't expect it to work perfect right out of the box, but it should get the job done. Please Report Bugs when you find them so I can get 'em fixed.