Still trying to get it all out

Slashdotted; Post-mortem

2009-06-11T14:34:00.000-07:00

About a day and a half ago, Reddit user stderr posted a link to the PHP documentation showing that GOTO will be a part of the language as of version 5.3. Somewhere in the reddit comments for this post, a link was pasted to an entry on this blog where I announced the feature being added years earlier. This naturally brought out the usual flame wars that circle around something like GOTO and drove some new traffic to my blog. No big deal, my server is pretty low-traffic, it can handle a few extra hits.

Within a few hours, burghler had browsed through other entries on my blog, finding what was at the time, the most recent entry about my friend's experience burying her mother . Just like the first reddit post, which had made the front page, this one also had a somewhat incendiary title.

Okay, more than a little incendiary, but I'll get to that in a moment...

The lesson

You would think, given that I'm the Architect for Yahoo WebSearch Front-end Engineering, that I would know something about configuring a server to not fall over under load. And in fact, I spend a good portion of my time on making sure that unexpected traffic spikes aren't enough to make a server get overloaded and trigger a chain-reaction of front-ends falling over. I really have no excuse for not preparing my server for what happened.

When I woke up Wed morning, I found that my server just didn't seem to respond to SSH or HTTP attempts. A reboot request didn't help, and the colo folks insisted that the server was simply running slow, but it was running. So I left my ssh connection attempt running and eventually I did get a login prompt. Several pained minutes later I managed to get an iptables rule in place to block off the flood of traffic (quicker than trying to stop the webserver). Suddenly, the CPU load was gone! Turns out I had my MaxClients setting much too high, and after a certain degree of concurrency, enough web-server children had spawned off to use up the available memory, which triggered disk swap, and made the CPU load 10x worse. Again, I know this effect exists, I really should have set up this server better.

A few tweaks to the config later and I got my server running smoothly. I also took the time to re-run my access log statistics. Of the aproximately 3 years of stats I've got, a full 2% of my hits were logged yesterday. Impressive reddit.... you win this round...

Bright squares indicate heavy traffic, dark squares indicate low traffic

Prior to yesterday's traffic

Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
	01	02	03	04	05	06	07	08	09	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31

Current stats

Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
	01	02	03	04	05	06	07	08	09	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31

My flickr views...

Clarifications

Despite the title of the reddit entry, I have nothing against Christians. Further, I count myself as one. So if you really want to take a piss at people who are calling Christians evil, don't aim at me. Thanks.

Moreover, I don't think those people were evil, or even horrible (though I did use that word in the heat of the moment). A loved one had just died, and everyone was in pain. Death SUCKS, and it was a bad situation no matter how you slice it. I don't hate those people for trying to erase my friend though, I pity them. I pity the fact that they turned down the chance to mourn the passing of their loved one by crying on another loved one's shoulder. I pity the fact that they didn't get nearly the closure that my friend did. I pity them for willingly becoming victims of their own grief.

The rest is between them and God.

Wait, did you seriously just say that?

2009-06-05T12:41:00.000-07:00

I grew up in California, and my state is pretty well true to it's reputation of being....less conservative than average. Granted, I've been harassed, shunned, even outright beaten over being queer, but I always thought that when it comes to certain things, especially family, there's a line that just isn't crossed. This is why the events of the past weekend so thoroughly amazed me.

Several months ago, when she told me that her mother had Cancer, I promised my kinda-ex-nevermind-it's-complicated-girlfriend that when the time came, I'd jump on a plane and fly out to the Midwest to help her bury her mom. So it was, that when she called me from Toronto (she was in the middle of a business trip), I booked a one-way flight (we weren't sure how long things would take), and flew off to Cleveland to meet her and offer my sympathies.

Death is hard, no matter how prepared for the end you are, but for my friend it was doubly difficult. For the past four years, she'd only rarely been able to see her mom, due to both distance and conflict between her and other people in her mother's family. We'd hoped that in the darkness of losing a loved one, those hard feelings and prejudices could be set aside, that family could come together for just one day to share grief and say good-bye. We were unbelievably wrong...

We landed together at CLE on friday night and headed over to her dad's house (her parents had been long divorced). He greeted us, filled her in on the past few months and her mother's last days, and we left feeling the weight of the occasion, but ready to grieve. On the way out, her sister called saying, "Some of the family have asked that you don't show up to the viewing. They say you can come into the funeral home for a few minutes afterward, but not during the general viewing."

I know, take a minute to absorb that statement. Who are these family members? Vague mumbles always answered this question, but they were certainly not going to tolerate my friend's presence. We resolved to call around the next day...

It was Saturday and the obituary had come out. Anachronistic newsprint mourned the loss of a sainted mother to my friend's sister, "special mother" to my friend's ex-spouse, but no mention of my friend. They had simply erased her, with a nod to her ex as a special kind of insult.

My friend called her sister back, wanting to understand how so much animosity could survive and flourish like that. She wasn't going to be erased, so she told her sister that the family could just deal with the fact that she would be present at the viewing. It was her mother that'd died, and she intended to say good-bye. A few hours later, news had filtered through to her step-father who called and delivered a matter-of-fact edict. "Here's your options, and I'm only going to say this once. You can come at 7, after everyone else is done, and take your 15 minutes to say good-bye, or you can not come at all. I'll have a cop at the door to make sure of that. *click*"

Yes, you heard that right, he was hiring an off-duty police officer to keep her out of her own mother's services.

And before you ask, yes, her step-father had that legal right, as the funeral home was private property contracted by him. So it was, that the next day, we (My friend, another friend of her's, and myself), drove to a shopping center near the funeral home and parked where our cars couldn't be identified and vandalized, and we walked to the funeral home at the official start time. Sure enough, a uniformed police officer stood guard at the entrace.

Arriving

We sent in the third member of our party, as she comes off much less threatening than I, and hadn't been explicitly told not to show up. She went inside, and formally asked the funeral home's director (with the step-father standing nearby), if we could be allowed inside. "Oh my God! They're here! They're here!", I wasn't present, of course, but I'm told the step-father reacted like we'd just stormed the beach at Normandy. "Those people are not to be allowed inside under any circumstances!", the funeral director could only agree with his client's request, so our other friend walked back to the sidewalk, where we sat down and initiated project Ghandi.

Okay, we didn't have any clever name for it at the time, but the premise was simple enough. We were in a labor state where the right to strike is sacrosanct. Being a public easement, we knew we were in our legal rights to remain there. This was the crux moment. Although we had legal rights, the police officer could have trumped up a reason to remove us for the duration of the ceremony, and we'd have lost our gamble completely, but he just stood at the door, showing no hint of emotion of sympathies for either side. A credit to his uniform.

I'm told, from those who were inside, that the step-father was yelling at the cop to have us removed and/or arrested. The cop answered that we were on public grounds and not causing a disturbance, so there was nothing he could do. I'm told he would get up every 15 minutes to look out the window and see if we were still there. We were. I'm told he called the police department to have additional officers sent around to "round us up", each new arrival told him the same story, no laws were being broken.

The challenge

About half an hour into the services, we were approached by the ex, who calmly flipped through the binder full of memories my friend had brought with her. They hugged, but didn't say much... What can be said at that point? Then my friend's ex-mother-in-law came over and just as I thought we were going to see another calm, but sad episode of sharing... "You know, you brought this on yourself."

I felt numb for a moment. I couldn't possibly have heard that come out of her mouth, not with my friend's mother lying in a casket 20 feet away... My friend managed to utter, "No", then turned away and began crying. "Yes, you did. You brought this on yourself." Our other friend interjected, "That's really not necessary..." "Yes it is", the mother in law replied, "this situation is because of her actions..."

She didn't get to finish that sentence. Something inside me snapped and a wellspring of rage came over me. I stood directly in front of the MiL, and found the most even temped voice I could manage, "Ma'am, I understand that you've just lost someone dear to you. I understand that you're in pain, and I understand that you think you're doing good, so I'm going to try to say this in the nicest way possible.", from here, my decorum ran out of rehearsed speeches and my rage took over, "You are an unchristian, bad...bad... Horrible person."

"No she isn't.", I heard from my firend's ex, but I ignored that. I stared at, and through her MiL, almost wanting a physical confrontation. "Sara..." The plaintive caution of our other friend snapped me out of my anger and I stepped away, andrenaline thundering through my blood. The MiL gave up trying to berate my friend and began to walk away. "I'm sorry", I offered, "That was unnecessary and uncalled for, I apologize." I don't remember much of the next several minutes, as my stress level slowly calmed down.

"Thank you", my friend told me. "I needed some strength right there and that's exactly what you gave me." After a moment she smiled, "You did unload on her a bit though...". "Yeah", I smiled back, "I was killing a few of my own demons there at the same time, might've over-estimated the firepower...

Soon afterward, her step-grandmother came around. "Why don't you just leave? You've making a spectacle of your mother's viewing!" She continued and I held myself back as long as I could. My friend held her own against this onslaught much better, continually drawing the conversation back to her mother, and trying to remember the best of her mother and focus on the mourning of her passing. The step-grandmother continued to catalouge all the (made-up) offenses my friend had allegedly committed against the family (all of which were thinly veiled reproachments for her queerness), until I spoke up, asking her if Jesus taught forgiveness, "Yes, BUT... we someone, when you have, when the person that has brought you into this world..." Honestly, even though I captured video of the entire event, I couldn't tell you for certain what she said as her argument descended and crashed into incoherency. Eventually she threw her hands up and walked away. Our other friend looked at me a little dumb-founded... "That... Was almost a thing of beauty the way she fell apart..." Yeah, I felt a little guilty at the efficacy of it too....

Redemption

Next up, was my friend's cousin. "What are y'all doing out here? Aren't you coming inside?" We told him what the step-father had said. His curious smile fell into disbelief, then anger rose in his face until his aura throbbed a dark crimson red. "That's ridiculous, I'll be right back." He stormed inside, and I'm told he raised holy hell on our behalf. Shortly thereafter, several more family members came out until there was all but a line queuing up around my friend to console her and share grief over her mother's death. For myself, I began to worry that we'd start to qualify as an unruly mob and give the police something to raise a fuss about. Nothing happened though, I think I even saw the cop smile.

Vindication

Eventually her mother's nurse, who was no part of the family, pulled my friend aside. "I only knew your mother for a few months, but she told me things about you in that time. Good things.", and she proceeded to list off things that she couldn't have made up. Things that proved to my friend that her mother really had loved her at the end, but was too afraid to show it.

My friend visibly brightened after that. As though she'd let go of something sour and offal. A few more hours passed in the pleasant late-spring weather. People came out and went back in, arrived and departed, noone else said an unkind word after her step-grandmother. Finally, when the funeral home was nearly empty, the cop approached us, "The funeral director's told me it's okay for you to come in now. I'm sorry for your loss." We thanked him and shook his hand and made our way inside. The step-father and other naysayers were nowhere to be seen, but there at the end of an array of fold-out chairs, was my friend's mother.

A poetic ending

I won't go into the details of her weeping, but she cried out the last of her grief. After hours of sharing, but not having access to the body, my friend had excised the worst of her grief already. She had prepared herself and found the peace she needed to see her mother's still form. For all her step-father's machinations and attempts to rob her of her mother and her place in her family, my friend was able to grieve her mother's passing in a way that couldn't possibly have been any better or more complete. If we had been inside the whole time, the shock of seeing her with so many unresolved things would have been devastating. If we had simply taken orders and not shown up until the last few minutes, there would not have been nearly enough time, and there would have been no family to lean on. By standing on the sidewalk, we took away that power, we reclaimed it as our own, and we amplified it even further. We said good-bye.

And the naysayers? They just looked like monsters. Love: 1, Hate: 0; Schadenfreude, I haz it.

Yahoo!, doing something right!

2009-01-08T23:16:00.000-08:00

I love it when Yahoo! manages to just get something right. Checkout the awesome sauce of a media player widget. By doing nothing more than linking to an external script. All three of these P3 podcasts become magically playable directly from my blog.

ZendCon 2008, PHP Extension Writing (Apologies for the gay-disco lead-in)
Interview following php|tek 2006
Guest hosting for the newscast
Ramblecast 2.0

Click the play buttons, but don't forget to notice the little slide-out player in the lower-left corner. Swanky, n'est-ce pas?

2008 Lookback

2008-12-30T20:49:00.000-08:00

It's been a helluva year... No, seriously, it's been... unique.

Last January, I started off the year with a presentation at geekSessions-1.3, along with some associated "hanging-out" in San Francisco (a city I normally avoid, because I dislike large crowds of people). I also got a brand new insulin pump to replace the seven year old beast who's LCD display had finally died. The pump itself didn't die (kudos to Medtronic for that), just the display. I've enjoyed my new "pocket pancreas" about as much as the old one, though of course the incremental improvements to its design haven't gone unappreciated.

February was a fairly quiet month for me professionally. A mixup in some of my medications (not related to the new pump), left me really distracted and irritable. The medications have been sorted, but I got to like the fear I induced in my coworker's eyes to the point that I keep up the show for appearances sake. :)

March was a flurry of activity getting things sorted at home and at work so that I could take a much needed month off in April. This month would ultimately turn into a mere three weeks since I refuse to relax, but it was time well spent anyway. During March I met two new friends who I've since come to appreciate deeply, and in April I got to spend a lot of time with my grandma; Something I do far too little of.

May saw the launch of Yahoo! Search Monkey move into full swing, bringing a lot of time and effort by the entire Search team along with it. As part of the launch, we held a developer event in Sunnyvale, and I learned that if you blog ~~long and hard~~ enough, random strangers will eventually show you naked pictures of themselves.

June was a hard month to handle. My wife of more than 10 years got herself a 1 bedroom apartment and moved out. We're both feeling the repercussions of that, and while reconciliation is a possibility, we really haven't fixed any of the things were wrong with our relationship. See, we're just as F-ed up as a straight couple, but with twice the estrogen! It was also at this time that my grandfather's cancer took a turn for the worse and he found himself spending a great deal of time in ICU. His children and grandchildren did their best to avoid feeling useless as we waited outside him room, not sure if he was going to pull out of it...

In July, my grandfather was feeling good enough that we threw a small party for my cousin's graduation, and brought him along for what we all knew was going to be his last summer. There was a lot of smiling outdoors and crying in the bathroom. During this party my grandfather pulled me aside and gave me a gift that... I'm not even sure he understood how much it meant to me... My sister took photos, and we all doted a bit more than we should have, but we built a new memory around him that we can carry with us.

Having managed some pretty downer months so far this summer, I took a week at the end of August to take a real vacation to south Utah where I went hiking with four complete strangers I'd met over the internet. Hrmmm... desert... strangers... dangerous climbs? Not the most sane thing I've ever done, but I made some really good friendships that'll last for years to come, and if all goes well, I'll be taking another trip with them this coming June, back to Zion National Park.

September saw ZendCon08 and PHP Appalachia, the latter of which spawned far too many anecdotes about the rowdiness of the PHP community. Where should I start? The semi-naked toxic hot tub? The jagged piece of glass that Chris tried to kill Ben with but got stuck in Cal's foot instead? The fire department we bribed with a cute blonde? Nonono, what happens in Pigeon Forge, stays in Pigeon Forge...

In October, my grandfather fell ill again, holding on just long enough to say good-bye to his children and grandchildren. Sadly his great-grandchildren aren't likely to remember him very well, but thankfully I know that my last words to him were, "I love you", and at least he went quickly. We can all hope for so much when out time eventually comes.

November led to another rowdy conference get together. This time it was php|works in Atlanta, GA, and I dunno about you, but Marco really needs to stop letting me near the open bar. I'm not a frequent drinker, so my tolerance is really very low, and I do... unconventional things when I've had my ninth vodka/cranberry... Doesn't hurt that the bartender was very generous... Oh well, at least I didn't throw up in the taxi on the way to Cal's going-away party. ((Note: I didn't say who did... assuming anyone did... which I didn't say either...))

And oh yeah... the election... biggest sack full of mixed emotions in a long time... Okay, sure, let me get it out, YAY! Obama won. I have HUGE hopes for this guy, and given what I've seen of his transition process, I think he might just stand a chance of living up to his hype. YES. WE. CAN. There, now that that's out of my system, I can go back to being really dissapointed in the One Million californians who voted in favor of Proposition 8. I can go back to slamming my face into the desk whenever I hear the FALSE argument that equal marriage rights will lead to sodomy in the streets and pedophilia in the classrooms. I can go back to being disappointed by my fellow Christians who call homosexuality as significant a threat to humanity as global warming and a threat to world peace. That's not just any Christian saying that by the way, that's the ever-lovin' POPE.

And then here it is: December. My wife and I are talking about getting back together, though I'm not sure we're making progress... I'm doing my best to focus back on my work. I've gotten back into rock climbing, and am eagerly awaiting the snow season to start in earnest. I got up to my dad's for another Christmas with the family, strained by not having gramps around, but I can tell my dad's trying to step up and fill the patriarchal void that's been left. I've spent the past week fighting off a cold and baking cookies. Tomorrow I'll head back into the office and get some work done before having friends over for a new year's eve bash. On thursday, it'll all start over again...

Happy new year.

I have officially arrived

2008-05-15T23:26:00.000-07:00

I was at the Search Monkey Developer Launch tonight to answer questions and give demos to the attendees. The event itself was a nice success, but that's not what I'm writing about. What I'm writing about is one particular attendee. Apologies to this guy if I embarrass him, but he had to know I'd say something.

After getting through the initial queue of developers asking about Search Monkey, I was approach by a tall young man:

<guy> Hey, you're Sara Golemon, right?
<me>  Yeah...
<guy> Nice to meet you, I really like your blog entries about how PHP works,
     and I bought your book!
<me>  Oh fantastic! Are you enjoying it?
<guy> Yeah, it's really heavy, and I haven't had a chance to use a lot,
     but I was experimenting with writing my own libssh2 wrapper...

By this point, I'm suitably blushing. I'm easily swayed by compliments thanks to my lack of self-esteem. Then he takes a turn...

<guy> I almost emailed this to you, it's a photo my girlfriend took of me without warning...
     Let me just find it in my camera
* guy shows photo of himself lying in a water-filled bathtub, naked, covering his... bits
     with a copy of my book

I... um... well... okay... I can see why you wouldn't email that out of the blue.... Now, I try to think of myself as a fairly progressive, hard-to-phase sort. I can tell a dirty joke with the best of 'em, but this just... left me giggly for the next hour. Like, um...wow... okay... Thanks for sharing...

Is this a sign? Is it a measure of notoriety when strangers show you naked photos of themselves at random tech gatherings?

Understanding Opcodes

2008-01-19T01:59:00.000-08:00

A blog reader (I have readers???) recently shared his wishlist, "I'm trying to figure out how to show the opcodes like you have in your post...". I promised that I'd throw something together, so here it is:

Slow down, wtf is an "Opcode"?

Short answer: It's the compiled form of a PHP script, similar in principle to Java bytecode or .NET's MSIL. For example, say you've got the following bit of PHP script:

<?php
 echo "Hello World";
 $a = 1 + 1;
 echo $a;

PHP (and it's actual compiler/executor component, the Zend Engine) are going to go through a multi-stage process:

Scanning (a.k.a. Lexing) - The human readable source code is turned into tokens.
Parsing - Groups of tokens are collected into simple, meaningful expressions.
Compilation - Expressions are translated into instruction (opcodes)
Execution - Opcode stacks are processed (one opcode at a time) to perform the scripted tasks.

Side note: Opcode caches (like APC), let the engine perform the first three of these steps, then store that compiled form so that the next time a given script is used, it can use the stored version without having to redo those steps only to come to the same result.

Er... okay... can you elaborate a little? What's lexing? I thought superman put him in jail...

That's Lex Luthor you nit-wit! The most expedient way to explain lexing is by example. Take a look at the manual page for token_get_all(), this gem is actually a wrapper around the Zend Engine's own language scanner. Play around with it a bit, and you'll notice that plugging the short script above into it will produce:

Array
(
   [0] => Array
       (
           [0] => 367
           [1] => <?php
       )
   [1] => Array
       (
           [0] => 316
           [1] => echo
       )
   [2] => Array
       (
           [0] => 370
           [1] =>
       )
   [3] => Array
       (
           [0] => 315
           [1] => "Hello World"
       )
   [4] => ;
   [5] => Array
       (
           [0] => 370
           [1] =>
       )
   [6] => =
   [7] => Array
       (
           [0] => 370
           [1] =>
       )
   [8] => Array
       (
           [0] => 305
           [1] => 1
       )
   [9] => Array
       (
           [0] => 370
           [1] =>
       )
   [10] => +
   [11] => Array
       (
           [0] => 370
           [1] =>
       )
   [12] => Array
       (
           [0] => 305
           [1] => 1
       )
   [13] => ;
   [14] => Array
       (
           [0] => 370
           [1] =>
       )
   [15] => Array
       (
           [0] => 316
           [1] => echo
       )
   [16] => Array
       (
           [0] => 370
           [1] =>
       )
   [17] => ;
)

In the array returned by token_get_all(), you have two types of tokens: Single character non-label characters are returned as just that. The character that was found in the source file at that point. Everything else, from labels, to language constructs, to multi-character operators (like >>, +=, etc...) are returned as an array containing two elements: The token ID (which corresponds to T_* constants -- e.g. T_ECHO, T_STRING, T_VARIABLE, etc...), and the actual text which that token came from. What the engine actually gets is slightly more detailed than what you see in the output from token_get_all(), but not by much...

Okay, tokenization just breaks the script into bite-size pieces, how does parsing work then?

The first thing the parser does is throw away all whitespace (Unlike some other P* language...). From the reduced set of tokens, the engine looks for irreducible expressions. How many expressions do you see in the example above? Did you say three? WRONG There are three statements, but one of those statements is made of two distinct expressions. In the case of $a = 1 + 1; the first expression is the addition, followed by the assignment to the variable as a second, distinct expression. All together our expression list is:

echo a constant string
add two numbers together
store the result of the prior expression to a variable
echo a variable

Hey! That's starting to sound familiar! Did I see that kind of description before?

Oh, you must mean my post about strings (plug). That's correct, because these expressions are exactly the pieces which go into making up oplines! Given the expression list we've just reached, the resulting opcodes look something like:

ZEND_ECHO 'Hello World'
ZEND_ADD ~0 1 1
ZEND_ASSIGN !0 ~0
ZEND_ECHO !0

What happened to $a? What's the difference between ~0 and !0?

Short answer: !0 is $a

So here's the deal.... oplines have five principle parts:

Opcode - Numeric identifier which distinguishes what the opline will do. This is what coresponds to ZEND_ECHO, ZEND_ADD, etc...
Result Node - Most opcodes perform "non-terminal" actions. That is; after executing there's some result which can be consumed as an input to a later opline. The result node identifies what temporary location to place the result of the operation in.
Op1 Node - One of two inputs to the given opcode. An input may be a constant zval, a reference to a previous result node, a simple variable (CV), or in some cases a "special" data element, such as a class definition. Note that an opcode may use both, one, or neither input node. (Some even use more, see ZEND_OPDATA)
Op2 Node - Ditto
Extended Value - Simple integer value used to differentiate specific behaviors of an overloaded opcode.

So obviously the nodes are the most complicated parts of an opline, here's the important parts of what they look like:

op_type - One of IS_CONST, IS_TMP_VAR, IS_VAR, IS_UNUSED, or IS_CV
u - A union of the following elements (the one which is used depends on the value of op_type):
- constant (IS_CONST) - zval value. This node results which you include a literal value in your script, such as the 'Hello World' or 1 values in the example above.
- var (IS_VAR or IS_TMP_VAR or IS_CV) - Integer value corresponding to a temporary slot in a lookup table used by the engine.

Now let's look at the difference between those optypes, particularly with respect to u.var:

IS_TMP_VAR - These ephemeral values are strictly for use by non-assignment non-terminal expressions. They don't support any refcounting because they're guaranteed not to be shared by any other variable. These are denoted in the examples I use on this site (and in VLD output) as tilde characters (~)
IS_VAR - Usually the result of a ZEND_FETCH(_DIM|_OBJ)?_(R|W|RW), or one of the assignment opcodes (which are technically non-terminal expressions since they can be used as inputs to other expressions. Since these are tied to real variables, they have to respect reference counting and are passed about at an extra degree of indirection. They're stored in the same table though. These are denoted by the string symbol ($)
IS_CV - "CV" stands for "Compiled Variables". These are basicly cached hash lookups for fetching simple variables from the local symbol table. Once a variable is actually looked up at runtime, it's stored at an extra level of indirection in an even faster lookup table using an index into a vector. That's what the number in this node denotes. These types of nodes are distinguished by a bang (!)

Boggle... You...so lost me there...

Yeah, that explanation sort of got away from me didn't it? What can I clear up?

All I really want to know is how to translate some source code into an opcode..list...thingy...

Heh, okay... first off, that "opcode list thingy" is called an op_array, and you can generate those really easily using one of two PECL packages. You can use my parsekit package, which is useful for programmatic analysis of script compilation, but frankly... it's not what you're looking for and there's not much call for scripts analyzing other scripts anyway. I recommend Derick's VLD (Vulcan Logic Disasembler) which is what'll actually generate the kinds of opcode lists you'll see me use in blog posts.

Once you've got it installed (it installs like any other PECL extension), you can run it with a command like the following:

php -d vld.active=1 -d vld.execute=0 -f yourscript.php

Then sit back and watch the opcodes fly! Important note: Using -r with command line code may not work due to a quirk of the way the engire parses files in older versions of PHP (and with older versions of VLD). Be sure to put your script on disk and reference it using -f if -r doesn't work for you.

Holy schnikies! That's a lot of opcodes! How can I tell what they all do?

Take a look at Zend/zend_vm_def.h in your PHP source tree. In here you'll find a meta-definition of every single opcode used by the engine. Side note: It's used as a source for zend_vm_gen.php which generates the actual code file zend_vm_execute.h. How's that for chicken and egg? Every version of PHP since 5.1.0 has required PHP be already built in order to build it!

I'm syndicated, and it has nothing to do with PHP!

2008-01-17T22:40:00.000-08:00

I was running some test queries using my usual spread of values and came across a result of "Lieutenant Fluffy?" which was far too whimsical a topic to ignore. Turns out someone thought my holiday photos made for a good creative commons pick. Kudos to the news site for respecting licensing terms properly!

Houston, we have a bolus

2008-01-07T12:14:00.000-08:00

It took three weeks to filter through Kaiser, but my new insulin pump finally arrived (the purple one on top in the photo below is the new one). My old pump was about seven years old when it broke down for the second time (the first was under warranty, about four years ago). By the way, the old one *IS* turned on, and you *SHOULD* see something on the display.... Hence my problem... Anyway, I wasn't expecting it till tomorrow, but christmas came early (well, I guess technically late) 'cause the FedEx guy rang the bell just as I was headed out the door for work.

SHINEY!!!!!! I haven't had a decent basal profile in nearly a month, so I ripped open the packaging, read the important parts of the instructions and plugged in.... Ah, that's the stuff... I'm now reading through the rest of the manual, here's my thoughts so far:

Batteries: A+ The MiniMed508 takes three Energizer 357s which... while not terribly hard to find, can be pretty annoying when you forget to replace them and the unit shuts down at 1AM... The new 722 fixes that by taking the larger, but more readily available AAA size.
Menus: B There's more complexity to the menus (driven by a wider feature set), but the most common task (bolusing) is a quick single-button action. ~~I'd have marked this as an A, but for the fact the bolus amounts no longer wrap-around. Pressing down from 0.0 stays at 0.0, and pressing up from 10.0 stays at 10.0.~~Update: Turns out the wrap-around does work. You just have to enable it.... Not sure how I enabled it though... Anyway, I'm leaving this at B since the the nag-messages are a bit too insistent... I know what I'm doing damnit... you don't have to be so annoying!
Backlight: C Same backlight, still not quite enough contrast...
Delivery mechanics: A Blatently stolen from Diesetronic (a competitor), but a good steal. Smoother, quicker delivery and a nice resevoir window to boot.
Loading mechanics: A- The resevoir-vial interface is a tiny bit temperamental, but once I get the knack of it, I expect cartridge loading will be twice as fast and accurate as it used to be. The priming mechanics also manage to overcome one of my long standing gripes about volume detection.
Interface/API: F Same gripe as I had with the MM508. The system is capable of interfacing with external equipment, but the company won't share anything useful about their specs (I knew this from research before hand, but I can still complain about it).
Durability: TBD We'll see how this one puts up with my.... abuse is too harsh a word.... "active lifestyle".... Let's go with that... At least it's (supposedly) more water-proof than the old one...

Hey Facebook! What gives?

2007-09-27T23:31:00.000-07:00

One of the ways in which I prepare for an interview is to research the person I'm going to be interviewing. The first step is to toss their name into one (or more) search engines and see what falls out. Not getting any useful results isn't a negative (many people work on internal stuff with no concern for fame fortune and glory, and they're very good at it), but if I happen to learn a little about them from a blog entry or a profile on a social networking site, then I can direct my questions in a way that more precisely fits the candidate.

So today, I sat down to read through the resumé of an upcoming candidate and punched his name into the best damned search engine ever. What's relatively unique about this candidate is that his name contains an accented character. The search results handled this accented character cleanly of course, and when I followed the first result for "José Smith" (Not his real name -- I'm respecting privacy here), I found myself looking at his Facebook profile. All well and great of course, but then I noticed the URL in the address bar: http://www.facebook.com/Jos_Smith/123456789.

Jos??? The only Jos I know is a director with a penchant for vampires.

Now, I get not putting the UTF8 or latin1 literal in the URL, browsers don't agree nearly enough on URL encoding to be able to expect it to work reliably. And putting the urlencoded version would just be silly since /Jos%C3%A9_Smith/ looks even uglier. I get that. But what if his name were entirely non-ascii? Perhaps a chinesse name? Would it be so hard to do a transliteration? I seem to recall Derick sharing a few ideas on the subject a couple years ago...

Dress Code

2007-09-15T10:41:00.000-07:00

The background

As most know, I work for Yahoo!, the US Web Search front-end team specifically. The company as a whole has (to my knowledge) no more of a dress code than "Don't show up naked". Consequently, most employees, especially those in engineering/operations roles, tend to dress pretty casually. During the summer, this progresses clear into the realm of shorts and sandals.

There's one coworker in our team, Venu, who insists on "dressing for success". Not ultra formal mind you, but we've never seen him wear anything short of a dress shirt, slacks, and nice shoes. At the other end of the spectrum is our boss, Lalit, who has only rarely been seen wearing anything other than shorts (and then, only in the coldest of winters). It was from this, that another coworker, Venkat, hatched his plan.

The setup

To everyone but Lalit:
 Hey, we should play a prank on Lalit, next monday, everyone should show up in shorts and flip-flops
        to make fun of him for his slacker dressing...


Meanwhile, to everyone but Venu:
 Okay, Venu's falling for it, now the REAL prank will be on him.  He'll show up in shorts,
        and the rest of us will dress formally...

Sure enough, today Venu "the slob" showed up in shorts and some (but not all) the rest of us showed him up in our "business professional" attire. Looking at my towering stature in this photo, I'm doubting the wisdom of wearing heels, but somehow flats just didn't create the right effect ;)

Lalit, Me, Venu, Ryan, Venkat, Nam

Fun with Unicode

2007-07-26T20:06:00.000-07:00

I caught a blog entry by Alexey Zakhlestin via Planet-PHP today which asked the question "Why aren't unicode math symbols supported by programming languages?". The obvious answer, of course, is that they work just fine with the symbols they have and there's no need to mess with a good thing (it doesn't help that typing these symbols is a pita on your average terminal).

Being a whimsical sort, I decided that actually implementing his request would be more fun than simply pish-poshing it. I'm not suggesting this be part of PHP6 (I still don't personally think it's a good idea), but it's a fun exercise and good for a conversation starter... Here's just a few of the things I can do now:

<?php
var_dump(¼, ½ ¾);
// float(0.25)
// float(0.5)
// float(0.75)

var_dump(1 ≤ 2, 2 ≯ 3, 5 ≠ 6);
// bool(true)
// bool(true)
// bool(true)

var_dump(3 × 4, 15 ÷ 5);
// int(12)
// int(3)

var_dump(1 « 3);
// int(8)

/* Your font may be too small,
 * but that's a skull and crossbones
 */
☠('aka die/exit');

P.S. - Just FYI, I am aware that the patch referenced above has flaws... The problems are fixable, I just didn't bother fixing 'em because this isn't a serious patch anyway.... Best spend that time on something worthwhile...

P.P.S. - Since the original posting of this article, PHP6-Unicode has been... well, essentially scrapped. So the patch itself is no longer relevant.

Index: Zend/zend_language_scanner.l
===================================================================
RCS file: /repository/ZendEngine2/zend_language_scanner.l,v
retrieving revision 1.167
diff -u -p -r1.167 zend_language_scanner.l
--- Zend/zend_language_scanner.l 12 Jul 2007 09:23:48 -0000 1.167
+++ Zend/zend_language_scanner.l 27 Jul 2007 05:05:47 -0000
@@ -402,6 +402,109 @@ ZEND_API int zend_copy_scanner_string(zv
  return 1;
 }
 
+/* Used by {LABEL} for converting unicode operator symbols */
+static inline int zend_scan_unicode_operator(zval *zendlval, char *str, zend_uint str_len, UConverter *conv, int *oplen, int have_equal TSRMLS_DC)
+{
+ int ret = 0;
+
+ switch (Z_USTRVAL_P(zendlval)[0]) {
+  case 0x2260: /* NOT EQUAL TO */
+   ret = T_IS_NOT_EQUAL;
+   break;
+
+  case 0x2264: /* LESS-THAN OR EQUAL TO */
+  case 0x226F: /* NOT GREATER-THAN */
+   ret = T_IS_SMALLER_OR_EQUAL;
+   break;
+
+  case 0x2265: /* GREATER-THAN OR EQUAL TO */
+  case 0x226E: /* NOT LESS-THAN */
+   ret = T_IS_GREATER_OR_EQUAL;
+   break;
+
+  case 0x00AB: /* LEFT-POINTING DOUBLE ANGLE QUOTATION MARK */
+  case 0x226A: /* MUCH LESS THAN */
+   ret = have_equal ? T_SL_EQUAL : T_SL;
+   break;
+
+  case 0x00BB: /* RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK */
+  case 0x226B: /* MUCH MORE THAN */
+   ret = have_equal ? T_SR_EQUAL : T_SR;
+   break;
+
+  case 0x2270: /* NEITHER LESS-THAN NOR EQUAL TO */
+   ret = '>';
+   break;
+
+  case 0x2271: /* NEITHER GREATER-THAN NOR EQUAL TO */
+   ret = '<';
+   break;
+
+  case 0x2276: /* LESS-THAN OR GREATER-THAN */
+  case 0x2277: /* GREATER-THAN OR LESS-THAN */
+   ret = T_IS_NOT_EQUAL;
+   break;
+
+  case 0x2278: /* NEITHER LESS-THAN NOR GREATER-THAN */
+  case 0x2279: /* NEITHER GREATER-THAN NOR LESS-THAN */
+   ret = T_IS_EQUAL;
+   break;
+
+  case 0x00D7: /* MULTIPLICATION SIGN */
+   ret = have_equal ? T_MUL_EQUAL : '*';
+   break;
+
+  case 0x00F7: /* DIVISION SIGN */
+   ret = have_equal ? T_DIV_EQUAL : '/';
+   break;
+
+  case 0x00BC: /* VULGAR FRACTION ONE QUARTER */
+   zval_dtor(zendlval);
+   ZVAL_DOUBLE(zendlval, 0.25);
+   ret = T_DNUMBER;
+   break;
+
+  case 0x00BD: /* VULGAR FRACTION ONE HALF */
+   zval_dtor(zendlval);
+   ZVAL_DOUBLE(zendlval, 0.5);
+   ret = T_DNUMBER;
+   break;
+
+  case 0x00BE: /* VULGAR FRACTION THREE QUARTERS */
+   zval_dtor(zendlval);
+   ZVAL_DOUBLE(zendlval, 0.75);
+   ret = T_DNUMBER;
+   break;
+
+  case 0x2620: /* SKULL AND CROSSBONES */
+   ret = T_EXIT;
+   break;
+ }
+
+ if (ret) {
+  /* How much of the input do we need to consume for this codepoint in this character set? */
+  UChar buffer[2], *buffer_ptr = buffer;
+  UErrorCode status = U_ZERO_ERROR;
+  const char *str_ptr = str;
+
+  ucnv_toUnicode(conv, &buffer_ptr, buffer + 1, &str_ptr, str + str_len, NULL, TRUE, &status);
+  if (U_FAILURE(status) && status != U_BUFFER_OVERFLOW_ERROR) {
+   /* Shouldn't happen... */
+   ret = 0;
+  } else {
+   *oplen = (str_ptr - str);
+  }
+ }
+
+ if (ret && ret != T_DNUMBER) {
+  zval_dtor(zendlval);
+  ZVAL_NULL(zendlval);
+ }
+
+
+ return ret;
+}
+
 static inline int zend_check_and_normalize_identifier(zval *zendlval)
 {
  UChar *norm;
@@ -2217,13 +2320,47 @@ HEREDOC_CHARS       ("{"*([^$\n\r\\{]|("
  return T_ENCAPSED_AND_WHITESPACE;
 }
 
-{LABEL} {
- if (!zend_copy_scanner_string(zendlval, yytext, yyleng, UG(unicode)?IS_UNICODE:IS_STRING, SCNG(output_conv) TSRMLS_CC)) {
-  return 0;
+{LABEL}"="? {
+ zend_bool have_equal = 0;
+
+ if (yytext[yyleng-1] == '=') {
+  have_equal = 1;
  }
- if (UG(unicode) && !zend_check_and_normalize_identifier(zendlval)) {
+
+ if (!zend_copy_scanner_string(zendlval, yytext, yyleng - have_equal, UG(unicode)?IS_UNICODE:IS_STRING, SCNG(output_conv) TSRMLS_CC)) {
   return 0;
  }
+
+ if (UG(unicode)) {
+  UChar *norm;
+  int norm_len;
+
+  if (!zend_is_valid_identifier(Z_USTRVAL_P(zendlval), Z_USTRLEN_P(zendlval))) {
+   int oplen = 0, ret = zend_scan_unicode_operator(zendlval, yytext, yyleng, SCNG(output_conv), &oplen, have_equal TSRMLS_CC);
+
+   if (ret) {
+    yyless(oplen + have_equal);
+    return ret;
+   }
+
+   zval_dtor(zendlval);
+   zend_error(E_COMPILE_WARNING, "Invalid identifier syntax: %r", Z_USTRVAL_P(zendlval));
+   return 0;
+  }
+
+  if (zend_normalize_identifier(&norm, &norm_len, Z_USTRVAL_P(zendlval), Z_USTRLEN_P(zendlval), 0) == FAILURE) {
+   zend_error(E_COMPILE_WARNING, "Could not normalize identifier: %r", Z_USTRVAL_P(zendlval));
+   efree(Z_USTRVAL_P(zendlval));
+   return 0;
+  }
+
+  if (norm != Z_USTRVAL_P(zendlval)) {
+   efree(Z_USTRVAL_P(zendlval));
+   ZVAL_UNICODEL(zendlval, norm, norm_len, 0);
+  }
+ }
+ yyless(yyleng - have_equal);
+
  return T_STRING;
 }

create_function() is not your friend, buddy

2007-05-19T07:13:00.000-07:00

While browsing through PHP Developer today, I came across this blog entry: "My new best friend" extolling the virtues of create_function(). Let me tell you why create_function() is not my best friend...

First, despite the disclaimer in the mentioned blog entry, it is as bad as its kissing-cousin eval(). Let's take a look at what create_function() actually does by translating it into userland code:

function create_function($args, $code)
{
   static $id = 0;
   eval("function __lambda_func($args) { $code }");
   while (!runkit_function_rename('__lambda_func', "\0lambda_" . (++$id)));
   return "\0lambda_$id";
}

I'll let you contemplate on that awhile....You should be noticing the following sets of problems:

Prone to critical abuse by user-supplied code
Skips opcode cache optimizations

You should also be thinking about the practical issues with it:

Code lives inside quoted strings which means awkward escaping of embedded quotes
Encourages not using comments (evil)
100% blind to reflection or PHPDoc style documentation generation
I'm sure you can come up with a couple more...

"If eval() is the answer, then you're asking the wrong question"

You're being lied to

2007-01-26T12:21:00.000-08:00

If you're among the crowd who have migrated an OOP based application from PHP4 to PHP5, then I'm sure you've heard the expression "Objects are copied by reference by default in PHP5". Whoever told you that, was lying.

Now, to be fair, it's an innocent lie, since objects do behave in a reference-like manner, but references are NOT what they are. Let's start with a simple illustration proving that they aren't references:

<?php
  $a = new stdClass;
  $b = $a;
  $a->foo = 'bar';
  var_dump($b);
  /* Notice at this point, that $a and $b are,
   * indeed sharing the same object instance.
   * This is their reference-like behavior at work.
   */

  $a = 'baz';
  var_dump($b);

  /* Notice now, that $b is still that original object.
   * Had it been an actual reference with $a,
   * it would have changed to a simple string as well.
   */
?>

What's going on here? Well, the answer is easiest to explain by explaining what the underlying structure of objects are. In PHP5, a variable containing an object identifies the instance by storing a simple numeric value. When an action is going to be performed on an object, that numeric value is used with a lookup table to retreive the actual instance. In PHP4, by contrast, a variable containing an array identifies that object by carrying around the actual properties table itself. What this means in practice is that when you assign (not by reference) a PHP5 object to a new variable, that integer handle is copied into the new variable, but it still points at the same instance, because it's still the same number. Assigning a PHP4 object however, means copying all the properties, effectively generating a new instance, since changes to one will not effect the other.

To put this another way, PHP4 objects are basically Arrays with functions associated with them, PHP5 objects are basicly Resources (a la MySQL result handles, or file pointers) again with functions loosely associated to them. Consider the following code in PHP4 (or any version):

<?php
  $fp = fopen('foo.txt', 'r');
  $otherVar = $fp;
  fwrite($fp, "One\n");
  fwrite($otherVar, "Two\n");
  fclose($fp);

  /* This fails, because the file is closed */
  fwrite($otherVar, "Three\n");
?>

You'd fully expect data to be written to the same file, as though you'd used $fp everywhere, rather than interchanging the variables right? Well, PHP5 objects are the same. The instance itself isn't duplicated when you assign to a new variable, just the unique identifier.

I'm lying to you also

"Copying" a variable doesn't exactly mean copying. Take the following code block:

<?php
  $a = 'foo';
  $b = $a;
  $a = 'bar';
?>

Now, you know PHP well enough to know that by the end of this code block, the value of $b will still be 'foo'. What you may not know, is that the original copy of 'foo' that was in $a, was never actually duplicated.

To understand what PHP is doing, you need to understand the internal structure of the variable and how it relates to userspace visible variable names ('a' and 'b' in this case). First off, the actual contents of a variable (known as a zval) consists of four parts: type (e.g. NULL, Boolean, Integer, Float, String, Array, Resource, Object), a specific value (e.g. 123, 3.1415926535, etc...), is_ref - a flag indicating if the value is a reference or not, and refcount which tells how many times this value is being shared.

What you think of as a variable (e.g. $x) is actually just a label, that label ('x' in this case) is used as a lookup to find the zval which conatins the actual value. These are just like keys in an associative array, in fact, the mechanisms are identical.

With me so far? Good. Now, when you first create a variable (e.g. $x = 123;, PHP allocates a new zval for it, stores the specific value, and associates the label with the value:

  'x' => zval ( type       => IS_LONG,
                value.lval => 123,
                is_ref     => 0,
                refcount   => 1 )

So far, refcount is 1 since the zval value is only being referenced by one label. If we now put this value into a full-reference set using $y =& $x;, the same zval is reused. It's simply associated with a new label and it's reference counters are adjusted properly.

  'x' => zval ( type       => IS_LONG,
      |         value.lval => 123,
      |         is_ref     => 1,
      |         refcount   => 2 )
  'y' /

This way, when you later change the value of $x, $y appears to change as well because it's looking at the same internal value. But what if we hadn't done a reference assignment, what if we'd done a normal assignment: $y = $x;, surprisingly, the result would be almost the same.

  'x' => zval ( type       => IS_LONG,
      |         value.lval => 123,
      |         is_ref     => 0,
      |         refcount   => 2 )
  'y' /

Again, the original zval associated with $x is reused, the only difference this time is that is_ref is not set to 1. This is known as a copy-on-write reference set (as opposed to the full-reference set described above). This 0 flag tells the engine that if anyone tries to change this value (regardless of which label they use to reach it), any other references to it should be left alone. Here's what happens if we take that current state and do $x = 456;

  'y' => zval ( type       => IS_LONG,
                value.lval => 123,
                is_ref     => 0,
                refcount   => 1 )
  'x' => zval ( type       => IS_LONG,
                value.lval => 456,
                is_ref     => 0,
                refcount   => 1 )

$x has been disassociated from the original zval (thus dropping its refcount back to 1), and new zval has been created for it.

Why referencing when you don't have to is a bad idea.

Let's consider one more situation, take a look at this code block:

<?php
  $a = 'foo';
  $b = $a;
  $c = &$a;
?>

At the first instruction, a single zval is created, associated to a single label:

  'a' => zval ( type => IS_STRING, value.str.val = 'foo', is_ref = 0, refcount = 1 )

At the second intstruction, that zval is associated to a second label, so far so good:

  'a' => zval ( type          => IS_STRING,
      |         value.str.val => 'foo',
      |         is_ref        => 0,
      |         refcount      => 2 )
  'b' /

At the third intstruction, however, we run into problems. Since this zval is already tied up in a copy-on-write reference set which include $b, that zval can't be simply promoted to is_ref==1. Doing so would drag $b into $a and $c's full-reference set, and that would be wrong. In order to resolve this, the engine is forced to duplicate that zval into two identical copies, from which it can begin to shuffle around reference flags and counts:

  'b' => zval ( type          => IS_STRING,
                value.str.val =>'foo',
                is_ref        => 0,
                refcount      => 1 )
  'a' => zval ( type          => IS_STRING,
      |         value.str.val => 'foo',
      |         is_ref        => 1,
      |         refcount      => 2 )
  'c' /

Now you've got two copies of the same literal value, so you're wasting memory for the storage, and processing time required to actually make the duplication. Since a LOT of events lead to copy-on-write uses (including simply passing an argument to a function), this sort of forced duplication actually happens very commonly when you start involving actual references.

The moral of the story

Assigning values by references when you don't need to (in order to later modify the original value through a different label) is NOT a case of you outsmarting the silly engine and gaining speed and performance. It's the opposite, it's you TRYING to outsmart the engine and failing, because the engine is already doing a better job than you think.

How does this reflect on objects? They're not special. They're not different from other variables. They are not pretty snowflakes. In this code block:

<?php
  $a = new stdClass;
  $b = $a;
?>

The labels are still placed into copy-on-write reference sets. What's important, is that even when a duplication does occur, (A) only that unique integer is copied (which is cheap), and (B) the duplicated integer still points to the same place. Hence you get reference-like behavior, but not an actual reference by default.

Hungry for more? Check out my coverage of the zval.

PHP-2006: A look back

2006-12-28T18:54:00.000-08:00

Well, Davey Shafik started us off with his year-end wrapup so I'll follow suit with mine. The thoughts below are mine and mildly influenced by alcohol. They represent a foggy review of how I experienced the year through the imperfect recollection of mailing list archives.

January began with releases of 4.4.2 and 5.1.2. Version 5.1.2 was especially close to my heart since it was the first version to ship an extension of mine not only bundled, but enabled by default. I've had my hands in most of the PHP runtime, but this was the first time I could point at a standard extension and say that it was basicly my work (Note: Mike Wallner did a fair bit of work adding to the number of hash algorithms supported, don't let me discount his efforts). Tim Starling wrote to ask why PHP4 refcounts are 16bit, and what he could do about getting that counter increased in future versions of PHP4. Since such a change would break binary compatability and since the PHP4 branch is already quite dead, the request was ultimately left alone with an admonishment to "not do that".

A few other requests were broached or continued, such as support for Friend Classes, Named Arguments, and Naming Arguments (The last of which would eventually be implemented). It was also this month in which Rasmus suggested adding JSON to the standard distribution, this quietly morphed into votes for including filter; Both were eventually linked in. James Crane had the idea that Array Literals in PHP could use sprucing up, meanwhile Sean Coates planted the seed for what would become the PHP6 Unicode Progress tracker.

February came in fairly quiet, mostly wrapping up the topics from January. Appearantly Steph didn't like the quiet and decided to stir up the hornet's nest with some four letter words cleverly disguised under the heading of True Labeled Breaks. For those who have blocked out 2005, this was easily one of the longest threads of that year and this resumption promised to be just as bad. By March, this thread would finnally be brought to a halt as the functionality was slipped quietly into the engine. I did not see that coming...

Amidst this unexpected addition, others were busily pulling things out, with Andi slashing away at safe_mode in HEAD, and Marcus integrating support for function deprecation into the 5.2 branch.

March came in like a lion with Marcus poking the list about Late Static Binding with less argument about Why, and a healthy focus on How. Johannes Schlueter made a very pragmatic and uncontroversial suggestion to change the internal symbol prefix applied to methods in order to distinguish them from functions. Pierre ran with some PDM recommendations shouting Adieu register_globals and Adieu a la magie. Sebastian Bergmann was frustrated with trying to apply streams filters to include and require, until I pointed him at the php://filter wrapper. Though this met his need, he made a strong case for being able to set an automatic filter to be applied to all streams. I promised to do this after streams in HEAD had been cleaned up, and while I've gotten streams where I want them, I still havn't fullfilled his feature request... Shame on me.

All this time, the GOTO debate was still raging and similar types of language altering features were being put on the table. Finally, Zeev decided he'd had enough calling for people to "Give the language a rest". This managed to have a decent effect which Rasmus soon channeled into his call for performance geeks to narrow the performance gap between 4.4 and 5.1. Towards the end of the month, I suggested adding an open_basedir_for_include directive. While there was some interrest for this, Ilia quickly convinced me that my approach to the problem was flawed and wouldn't give any real benefit.

April showered the internals list with Round 2 of the Late Static Binding discussion, and Thomas Boutell's anouncement that he'd be turning over primary development of the GD library to the PHP project, spearheaded by Pierre. I made some more noise complaining of RETURN_RT_STRING() (and family)'s leakage of memory under certain not-uncommon conditions, while Nuno brought the infamous Coverity Report to attention.

Richard Lynch shared his WTF concerning the oft misunderstood tsrm_ls parameter in PHP's sources. Sadly, I hadn't written my summary of the topic yet, but although noone gave him a detailed description (It's a bit long to go into in an email), several helpful links were supplied. Round about the middle of the month, Rasmus announced PHP's eminent participation in the Google Summer of Code project (wait, don't you work for Yahoo!?). After signing up as a mentor (though I never actually mented -- is that a word?), I gathered up my materials and flew off to php|tek 2006 in Orlando.

May opened to the initial planning rounds for PHP 5.2.0, and the resumption of the ifsetor() request, now dressed up as coalesce(). By the way, why havn't we embraced this feature request yet? About a week into the month, Ilia branched the PHP_5 tree leaving room for the earnest development of 5.2.

William Candillon wrote in to request a PHP version of C's #line macro, but was drowned out by the roar of Derick's plea to "Stop breaking our apps for the sake of OO". Once the din of that had quieted down a little, Jason Garber decided to ask about making it possible to mark properties as read only. In case anyone was curious, none of these proposals gained footing.

June now, and my book is finally released bringing with it an invitation to appear on php|architect's Pro::PHP podcast. Marcus Boerger suggested that array indices could benefit from implicit __toString() calls, but he was eventually shot down. Dimitry's suggestion however, which provided for automatic module global registration via the module entry, did receive a warm welcome and you'll see it if you look inside PHP 5.2.0.

Clearly this was the month of bright ideas, because Nuno Lopes tossed in gcc branch prediction. It got a warm initial reception from the engine folk including Zeev stating "I actually like how it makes the code more readable hinting which branches are rare.". Andi agreed that it was a nice idea, but brought in the sobering reality that it didn't really do much for performance and could potentially bring unexpected results if applied heavily.

July was a busy month for me as I changed jobs for the first time in over six years. I'm loving the new job by the way. Fantastic coworkers and....interresting challenges.... Laupretre François thought it'd be a good idea to extend include_path to support stream wrappers. Those of us who know what trouble include_path already causes for performance and security were quick to nix that particular idea; Short version: No.

Marcus popped in mid-month with an implementation of the #line directive suggested back in May. Still no love from the internals community at large though; Short version: no #line for you. Dmitry committed a large patch to the Zend Engine to change how non-persistent memory is allocated and freed within a request. The good news is that emallocs are now faster, the bad news is that a hack I'd made in PHP5.1 for manipulating the non-persistent memory pool no longer worked...Grrrr.....

On July 27th, Jani said Good-Bye.

August picked up Mike Wallner's July post decrying the state of OO strictness building up in PHP. This maelstrom took up most of the first week with no real conclusion (at least, not a satisfactory one). A different Mike made some more noise later in the month asking why accessing non-existant functions/methods has to be so darned fatal. The good news is that they aren't so much anymore. Yay for E_RECOVERABLE_ERROR.

September brought a landslide of movement in the PHP6 function migration/review process. There was some degree of question as to whether unicode.semantics should be SYSTEM, PERDIR, or USER. Noone really considered the latter as a possibility as it wrecks way too many assumptions, but the SYSTEM/PERDIR debate raged for awhile with the idealists wanting PERDIR support to aid migration, and the realists clinging to the maintainable simplicity of SYSTEM. In the end, SYSTEM won.

October was a relatively slow month, leading up to conference season such as it was. Midmonth sometime I tossed out the idea of allowing open_basedir to be tightened (but not loosened) during runtime and it was green-lighted surprisingly quietly. Ilia got a little frustrated with the mounting delays holding back the release of PHP 5.2.0 which was already way behind schedule. Tragicly, this plea made the delay last even longer. Finally at the end of the month, he rolled final.

November was a big conference month for me. First I crashed the nearby ZendCon, then turned right around and flew off to Germany for the International PHP Conference. Sean poked the namespaces topic with a sharp stick and managed to generate a week's worth of noise resulting in no actual commitments by any capable/interrested parties. No sooner had that comotion died down than did Antony Dovgal bemoan a regression in fgets()'s behavior which I introduced in HEAD. I still think it's a silly behavior for a userspace function, but BCs are BCs, so the regression's been reverted.

December's most exciting event was a debaucle over the backward/forward compatability of serialize given the additional escaping concerns that processing unicode strings imposes. Ilia brought forward a proposal to finally remove the unnecessary COM/Sockets/MHash extensions from the distribution bundle. After some light debate it was decided to keep COM around and only nix Sockets and MHash as of PHP6 (possibly 5.3 if such a version comes about). Wietse Venema brought back the concept of introducing a taint mode for PHP. The topic is still being discussed, but things don't look good for Wietse...

And now here we are, at year's end. PHP6's unicode function migration process has passed the 50% mark and a preview release is sure to come once we handle a couple more extensions. PHP 5.2 is grabbing hold amongst the serious PHP shops, and even web hosters are trickling away from PHP4. There's still a huge, bright future ahead for this language, together we can make it happen.

¡Feliz Año Nuevo!

When good encodings go bad

2006-11-29T13:26:00.000-08:00

In the past year, I've been doing some work with Unicode as part of the PHP6 upgrade. I've learned more than I wanted to know about all sorts of encodings from UTF-7 to koi8-r to good old iso-8859-1. I've picked apart the picayune differences between UCS-2 and UTF-16, and played the game of surrogate pairing and orphaning. Despite all that exposure however, I wasn't prepared when a question crossed my inbox about a lesser known encoding called AL32UTF8.

I'd never heard of this one before, so I went to my favorite search engine for some answers. Turns out it's something Oracle came up with and later got adopted as a proper standard with the name CESU-8. At first glance, CESU-8 looks identical to UTF-8 in the same way that UCS-2 looks a lot like UTF-16. In fact every codepoint from U+0000 to U+FFFF is encoded identically under both sets of rules: 16 bits, split up over one, two, or three bytes, with leftover bits framing the encoding protocol.

When you jump up above U+FFFF however, into the realm of CJK codepoints and the like (such as my personal nom du pointe: 𣚺) something funny starts to happen. In the UTF8 world, these codepoints are accomodated by adding one extra byte to the mix which allows for up to 22bits of data (All of unicode only requires 21). In the CESU-8 world however, the code point is split according to UTF-16 surrogacy rules making two separate unicode points (each in the range U+D800 - U+DFFF). These two unicode points are then encoded individually into UTF-8 sequences. This means that we've now promoted our variable length (4 max) multibyte encoding to a variable length (6 max) multibyte-multibyte encoding. Thank you Oracle. Thank you for adding complexity to encoding rules while increasing data storage requirements. What would the world do without you?

P.S. - Java is at fault too... its 'Modified UTF-8' uses nearly identical rules.

Don't worry, I slept last Thursday

2006-11-05T23:08:00.000-08:00

On this, my first foray to Europe, indeed my first real trip outside the US (Those couple hours in Tijuana don't count), I'm faced with one undeniable, inexcapable fact. Jet lag sucks.

It doesn't help that all last week I was staying up past my normal bedtime partying with the attendees of ZendCon06, but I think my real mistake was trying to outsmart my own circadian rhythm. See, I figured "I've got this long flight across the atlantic, it'll go faster if I can fall asleep at some point." Seems reasonable so far. How to ensure sleep? Why, stay up all night before the flight. Brilliant! But wait, what if I can't fall asleep during the flight?

Sometime sunday morning, my plane lands in Frankfurt and I wander, zombie-like, through passport control, baggage claim, and customs somehow managing to board the right shuttle to reach the conference hotel. Based on advices from battle-hardened globetrotters, I was planning to put in a one hour power nap, then go for a walk to "reset" my internal clock. Unfortunately the sixty-plus hour run of consciousness had other plans and by the time I awoke, the sun had set.

Finally this morning (Monday), I managed to put in that walk, touring a nearby suburb which reminded me somewhat spookily of the setting from "Shaun of the Dead" (more zombie tie-ins). Five euros and three liter bottles of diet coke later and I'm almost coherent enough to....what the hell was I gonna say? Sorry, my brain has been cutting out a lot the past few days...

Guten Morgen!

Moving right along

2006-07-05T13:34:00.000-07:00

After well over half a decade at the University of California at Berkeley, I'm moving on to greener pastures. Well, maybe not greener (Berkeley is full of evergreens and...other verdant plant substances), but pastures at the very least. Next monday, my life's journy will bring me to that farmland come technopolis known the world over as Silicon Valley. I'll be cubefarming private sector style for one of the few dotcom survivors, a little mom & pop outfit called Yahoo!. I've worked in education and the public sector since 1997 and while I'm optimistic everything will go smoothly, well....I think Dan Aykroyd put it best as Dr. Raymond Stantz in Ghostbusters: "You don't know what it's like out there, you've never worked in the private sector, They expect results! (shudder)". If nothing else it'll be nice to get my commute back down to the 20minute zone. At any rate, I've got 2 days and as many going-away parties to get through, then it's on to the enemy camp (Yahoo's birth {not as a company, but as the index site it started out as} was at UCBerkeley's long-time rival, Stanford University).

How long is a piece of string

2006-06-18T19:05:00.000-07:00

Sunday morning I was asked by an IRC regular: "Where does the engine parse quoted strings?". Being a sunday morning, I began to launch into a sermon on the distinction between CONSTANT_ENCAPSED_STRING and the problems which befall a single-pass compiler when you start to introduce interpolation. Not what he asked precisely, but an important component in answering his question. Unfortunately, at the time I was busy watching the Brasil-Australia game so I didn't go into the kind of detail I would have. Now, some 12 hours later, since Angela is off buying toe-socks in Santa Cruz, I'll bore anyone with little enough life to read my blog by explaining the pitfalls of using PHP's string interpolation without using an optimizer.

To start things off, let's take a page from my earlier discourse on Compiled Variables and look at the opcodes generated by a few simple PHP scripts:

<?php
echo "This is a constant string";
?>

Yields the nice, simple opcode:

ECHO            'This is a constant string'

No problem... Exactly what you'd expect... Now let's complicate the expressions a little:

<?php
echo "This is an interpolated $string";
?>

Yields the surprisingly messy instruction set:

INIT STRING  ~0
ADD_STRING   ~0 ~0 'This'
ADD_STRING   ~0 ~0 ' '
ADD_STRING   ~0 ~0 'is'
ADD_STRING   ~0 ~0 ' '
ADD_STRING   ~0 ~0 'an'
ADD_STRING   ~0 ~0 ' '
ADD_STRING   ~0 ~0 'interpolated'
ADD_STRING   ~0 ~0 ' '
ADD_VAR      ~0 ~0 !0
ECHO            ~0

Where !0 represents the compiled variable named $string. Looking at these opcodes: INIT_STRING allocates an IS_STRING variable of one byte (to hold the terminating NULL). Then it's realloc'd to five bytes by the first ADD_STRING ('This' plus the terminating NULL). Next it's realloc'd to six bytes in order to add a space, then again to eight bytes for 'is', then nine to add a space, and so on until the temporary string has the contents of the interpolated variable copied into its contents before being used by the echo statement and finally discarded. Now let's rewrite that line to avoid interpolation and use concatenation instead:

<?php
echo "This is a concatenated " . $string;
?>

Which yields the significantly shorter and simpler set of ops:

CONCAT       ~0 'This is a concatenated ' !0
ECHO            ~0

A vast improvement already, but this version still creates a temporary IS_STRING variable to hold the combined string contents meaning that data is duplicated when it's being used in a const context anyway. Now let's try out this oft-overlooked use of the echo statement:

<?php
echo "This is a stacked echo " , $string;
?>

Look close, there is a meaningful difference from the last one. This time we're using a comma rather than a dot between the operands. If you don't know what the comma is doing there, ask the manual then check back here. Here's the resulting opcodes:

ECHO            'This is a stacked echo '
ECHO            !0

Same number of opcodes, but this time no temporary variables are being created so there's no duplication and no pointless copying (unless of course $string wasn't of type IS_STRING, in which case it does have to be converted for output, but don't get picky now). Think this is bad? Consider the average heredoc string which spans several lines of prepared output embedding perhaps a handful of variables along the way. Here's one of several such blocks found in run-tests.php within the PHP distribution source tree:

<?php
echo <<NO_PCRE_ERROR

+-----------------------------------------------------------+
|                       ! ERROR !                           |
| The test-suite requires that you have pcre extension      |
| enabled. To enable this extension either compile your PHP |
| with --with-pcre-regex or if you've compiled pcre as a    |
| shared module load it via php.ini.                        |
+-----------------------------------------------------------+

NO_PCRE_ERROR;
?>

Notice that we're not even embedding variables to be interpolated here, yet does this come out to a simple, single opcode? Nope, because the rules necessary to catch a heredoc's end token demand the same careful examination as double-quoted variable substitution and you wind up (in this case) with SEVENTY-EIGHT opcodes! One INIT_STRING, 76 ADD_STRINGs. and a final ECHO. That means a malloc, 76 reallocs, and a free which will be executed every time that code snippet comes along. Even the original contents take up more memory because they're stored in 76 distinct zval/IS_STRING structures.

Why does this happen? Because there are about a dozen ways that a variable can be hidden inside an interpolated string. Similarly, when looking for a heredoc end-token, the token can be an arbitrary length, containing any of the label characters, and may or may not sit on a line by itself. Put simply, it's too difficult to encompass in one regular expression.

The engine could perform a second-pass during compilation, however the time saved reassembling these strings will typically be about the same amount of time spent actually processing them during runtime (if one assumes that each instance will execute exactly once). Rather than complicate the build process (potentially slowing down overall run-times in the process), the compiler leaves this optimization step to opcode caches which can achieve exponentially greater advantage cleaning up this mess then caching the results and reusing the faster, leaner versions on all subsequent runs.

If you're using APC, you'll find just such an optimizer built in, but not enabled by default. To turn it on, you'll need to set apc.optimization=on in your php.ini. In addition to stitching these run-on opcodes back together, it'll also add run-time speed-ups like pre-resolving persistent constants to their actual values, folding static scalar expressions (like 1 + 1) to their fixed results (e.g. 2), and simpler stuff like avoiding the use of JMP when the target is the next opcode, or boolean casts when the original expression is known to be a boolean value. (It should be noted that these speed-ups also break some of the runtime-manipulation features of runkit, but that was stuff you....probably should have been doing anyway)

Can't use an optimizer because your webhost doesn't know how to set php.ini options? You can still avoid 90% of the INIT_STRING/ADD_STRING dilema by simply using single quotes and concatenation (or commas when dealing with echo statements). It's a simple trick and one which shouldn't harm maintainability too much, but on a large, complicated script, you just might see an extra request or two per second.

Extending and Embedding PHP

2006-06-07T14:20:00.000-07:00

It's official!!!! After a year in development Extending and Embedding PHP is now shipping from fine book stores everywhere.

I've gotten good reviews from the half dozen people I know who've gotten their hands on it, and I am really satisfied with most of it. Do I think it could be better? Of course I do, but I don't think I was ever going to be completely satisfied.

I've learned a lot through this process and while I don't see any more titles in my immediate future (there are things coming down the pipe which are likely to change my availability), I do expect that my next book, should it materialize, will be even better.

If you pre-ordered it, you should see it soon, if you've already got your copy, let me know what you think! Did I skim over some topic too quickly? Did I belabour something else? If this book eventually finds it's way into a 2^nd edition (no promises mind you), are there topics you want to see added? Tossed out? Expanded/Compressed?

What the heck is TSRMLS_CC, anyway?

2006-06-01T13:08:00.000-07:00

If you've ever worked on the PHP internals or built an extension, you've seen this construct floating around here and there, but noone ever talks about it. Those who know what this is typically answer questions from those who don't with "Don't worry about what it is, just use it here here here and here. And if the compiler says you're missing a tsrm_ls, put it there too..." This isn't laziness on the part of the person answering the question (okay, maybe it is a little bit), it's just that the engine goes so far out of its way to simplify what this magic values does, that there's no profit in a new extension developer knowing the mechanics of it. The information is like a cow's opinion, it doesn't matter, it's Moo.

Since I love to listen to myself rattle on about pointless topics (and I havn't blogged much this month), I thought I'd cover this topic and see if anyone manages to stay awake through it. You can blame Lukas, he got me rolled onto planet-php.net...

Glossary

TSRM

Thread Safe Resource Manager - This is an oft overlooked, and seldom if ever discussed layer hiding in the /TSRM directory of your friendly neighborhood PHP source code bundle. By default, the TSRM layer is only enabled when compiling a SAPI which requires it (e.g. apache2-worker). All Win32 builds have this layer enabled enabled regardless of SAPI choice.

ZTS

Zend Thread Ssafety - Often used synonymously with the term TSRM. Specifically, ZTS is the term used by ./configure ( --enable-experimental-zts for PHP4, --enable-maintainer-zts for PHP5), and the name of the #define'd preprocessor token used inside the engine to determine if the TSRM layer is being used.

tsrm_ls

TSRM local storage - This is the actual variable name being passed around inside the TSRMLS_* macros when ZTS is enabled. It acts as a pointer to the start of that thread's independent data storage block which I'll cover in just a minute

TSRMLS_??

A quartet of macros designed to make the differences between ZTS and non-ZTS mode as painless as possible. When ZTS is not enabled, all four of these macros evaluate to nothing. When ZTS is enabled however, they expand out to the following definitions:

TSRMLS_C tsrm_ls
TSRMLS_D void ***tsrm_ls
TSRMLS_CC , tsrm_ls
TSRMLS_DC , void ***tsrm_ls

Globals

In any normal C program (just like in PHP) you have two methods of getting data access to the same block of data in two different functions. One method is to pass the value on the parameter stuck like so:

#include 

void output_func(char *message)
{
   printf("%s\n", message);
}

int main(int argc, char *argv[])
{
   output_func(argv[0]);

   return 0;
}

Alternately, you could store the value in a variable up in the global scope and let the function access it there:

#include 

char *message;

void output_func(void)
{
   printf("%s\n", message);
}

int main(int argv, char *argv[])
{
   message = argv[0];
   output_func();

   return 0;
}

Both approaches have their merits and drawbacks and typically you'll see some combination of the two used in a real application. Indeed, PHP is covered in global variables from resource type identifiers, to function callback pointers, to request specific information such as the symbol tables used to store userspace variables. Attempting to pass these values around in the parameter stack would be more than unruly, it'd be impossible for an application like PHP where it's often necessary to register callbacks with external libraries which don't support context data.

So common information, like the execution stack, the function and class tables, and extension registries all sit up in the global scope where they can be picked up and used at any point in the application. For single-threaded SAPIs like CLI, Apache1, or even Apache2-prefork, this is perfectly fine. Request specific structures are initialized during the RINIT/Activation phase, and reset back to their original values during the RSHUTDOWN/Deactivation phase in preparation for the next request. A given webserver like Apache1 can serve up multiple pages at once because it spawns multiple processes each in their own process space with their own independant copies of global data.

Now let's introduce threaded webservers like Apache2-worker, or IIS. Under these conditions, only one process space is active at a given time with multiple threads spun off. Each of these threads then act in the same manner as a single-threaded process might; Servicing requests one-at-a-time as dispatched by inbound requests. The trouble starts to brew as two or more threads try to service the a request at the same time. Each thread wants to use the global scope to store its request-specific information, and tries to do so by writing to the same storage space. At the least, this would result in userspace variables declared in one script showing up in another. In practice, it leads to quick and disasterous segfaults and completely unpredictable behavior as memory is double freed or written with conflicting information by separate threads.

Non-Global Globals

The solution is to require the engine, the core, and any extension using global storage to determine how much memory will be used by request-specific data. Then, at the spin-up of each new thread, allocate a chunk of memory for each of these players to store their data into thus giving each thread its own local storage. In order to group all the individual chuncks used by a given thread together, one last vector of pointers is allocated to store the individual sub-structure pointers into. It's the pointer to this vector which is passed around as the tsrm_ls variable by the TSRMLS_* family of macros. To see how this works, let's look at a example extension:

typedef struct _zend_myextension_globals {
   int foo;
   char *bar;
} zend_myextension_globals;

#ifdef ZTS
int myextension_globals_id;
#else
zend_myextension_globals myextension_globals;
#endif

/* Triggered at the beginning of a thread */
static void php_myextension_globals_ctor(zend_myextension_globals *myext_globals TSRMLS_DC)
{
   myext_globals->foo = 0;
   myext_globals->bar = NULL;
}

/* Triggered at the end of a thread */
static void php_myextension_globals_dtor(zend_myextension_globals *myext_globals TSRMLS_DC)
{
   if (myext_globals->bar) {
       efree(myext_globals->bar);
   }
}

PHP_MINIT_FUNCTION(myextension)
{
#ifdef ZTS
   ts_allocate_id(&myextension_globals_id, sizeof(zend_myextension_globals),
                  php_myextension_globals_ctor, php_myextension_globals_dtor);
#else
   php_myextension_globals_ctor(&myextension_globals TSRMLS_CC);
#endif

   return SUCCESS;
}

PHP_MSHUTDOWN_FUNCTION(myextension)
{
#ifndef ZTS
   php_myextension_globals_dtor(&myextension_globals TSRMLS_CC);
#endif

   return SUCCESS;
}

Here you can see the extension declaring its global requirements to the TSRM layer by stating that it needs sizeof(zend_myextension_globals) bytes of storage, and providing callbacks to use when initializing (or destroying) a given thread's local storage. The value populated into myextension_globals_id represents the offset (common to all threads) into the tsrm_ls vector where the pointer to that thread's local storage can be found. In the event that ZTS is not enabled, the data storage is simply placed into the true global scope and the thread initialization and shutdown routines are called manually during the Module's Startup and Shutdown phases. If you're wondering why TSRMLS_CC was included in the non-ZTS blocks, then I clearly havn't made you fall asleep yet. Those aren't needed there since we know they evaluate to nothing, but it helps encourage good habits to include them anywhere the function's prototype calls for them.

Putting it all together

The final piece of this thread-safe puzzle comes from the question: "How do I access data in these structures?" And the answer to that question comes in the form of another familiar looking macro. Each extension or core component defines, in one of its header files, a macro which looks something like the following:

#ifdef ZTS
# define   MYEXTENSION_G(v)     \
             (((zend_myextension_globals*)(*((void ***)tsrm_ls))[(myextension_globals_id)-1])->v)
#else
# define   MYEXTENSION_G(v)     (myextension_globals.v)
#endif

Thus, when ZTS is not enabled, this macro simply plucks the right value out of the imediate value in the global scope, otherwise it uses the ID to locate the thread's local storage copy of the structure and derefence the value from there.

Wanna know more, like how to deal with foreign callbacks where tsrm_ls isn't available? Buy my book!

Compiled Variables

2006-05-24T10:15:00.000-07:00

Last month at php|tek I gave a presentation on "How PHP Ticks" where I covered, among other things the process of compiling source code into opcodes (an intermediate psuedo-language similar to what java calls "bytecode" or what .NET calls "MSIL"). As part of this section of the presentation, I showed one of the more interresting changes between ZE 2.0 (PHP 5.0) and ZE 2.1 (PHP 5.1), namely: How variables are retreived and used in an operation. More specifically, how they provide a small, yet cumulative, speedup to applications in a way that's transparent to the end-user -- One more reason to like PHP 5.1 right?

After listening to Marcus Whitney's interview with Brion Vibber of WikiMedia in which he mentions my presentation and makes reference to this engine change, I realized that I should clarify what this feature is (and more importantly, what it isn't) before any FUD spreads.

What Compiled Variables (CVs) are

First, let's look at the anantomy of an OpArray. Say you have the following simple script:


<?php
$a = 123;
$b = 456;
$c = $a + $b;
echo $c;

Now let's see how ZE 2.0 (PHP5.0) compiles this (ZE1.x/PHP4.x comes out to nearly identical opcodes). The $0 and ~0 references you see are (for lack of a better one sentence explanation) types of temporary variables (the latter moreso than the former, but don't worry about the distinction right now). What's important to know about this block and its statements are in the accompaning comments:

FETCH_W                  $0, 'a'          /* Retreive the $a variable for writing */
ASSIGN                   $1, $0, 123      /* Assign the numeric value 123 to retreived variable 0 */
FETCH_W                  $2, 'b'          /* Retreive the $b variable for writing */
ASSIGN                   $3, $2, 456      /* Assign the numeric value 456 to retreived variable 2 */
FETCH_R                  $5, 'a'          /* Retreive the $a variable for reading */
FETCH_R                  $6, 'b'          /* Retreive the $b variable for reading */
ADD                      ~7, $5, $6       /* Add the retreived variables (5 & 6) to gether and store the result in 7 */
FETCH_W                  $4, 'c'          /* Retreive the $c variable for writing */
ASSIGN                   $8, $4, ~7       /* Assign the value in temporary variable 7 into retreived variable 4 */
FETCH_R                  $9, 'c'          /* Retreive the $c variable for reading */
ECHO                         $9           /* Echo the retreived variable 9 */

Seem like a lot of work for one plus one? It is, here's the same code snippet compiled by ZE 2.1/PHP 5.1 (or later).

ASSIGN                   $0, !0, 123      /* Assign the numeric value 123 to compiled variable 0 */
ASSIGN                   $1, !1, 456      /* Assign the numeric value 456 to compiled variable 1 */
ADD                      ~2, !0, !1       /* Add compiled variable 0 to compiled variable 1 */
ASSIGN                   $3, !2, ~2       /* Assign the value of temporary variable 2 to compiled variable 2 */
ECHO                         !2           /* Echo the value of compiled variable 2 */

These !0 variables refer to a new structure in the execution stack which stores and references to the "real" variables out in userspace. The hash value for each variable is computed at compile time (meaning that it's only done once per variable no matter how often it's used and that opcode caches save this work from being done during subsequent page views at all). The first time one of these CVs is used, the engine looks it up in the active symbol table and updates the CV cache to know where it is. All subsequent uses of that compiled variable use that pre-fetched address and don't have to look it up again. On an individual lookup, this isn't a major leap forward in speed, however consider a for loop where the test value is checked on every iteration; To put it in PHP terms, which would you rather do?

for($i = 0; $i < foo =" lookup_variable('foo');">increment();
$foo = lookup_variable('foo');
$foo->check_value();
}

$foo = lookup_variable('foo');
for($i = 0; $i <>increment();
$foo->check_value();
}

What compiled variables are not

Don't assume you're going to get a speedup on all of your code, especially if you use arrays or objects (which most code taking advantage of PHP5's new features does). The CV speedup has one minor achilles heel: It only works on simple variables. Putting it in terms of opcodes, let's consider this PHP source:

<?php
$f->a = 123;
$f->b = 456;
$f->c = $f->a + $f->b;
echo $f->c;
?>

Basicly the same code right? Just a little oopified... Let's look at the ZE2.1/PHP5.1 compilation of that:

ASSIGN_OBJ                $0, !0, 'a'     /* Assign the numeric value 123 to property 'a' of compiled variable 0 object */
OP_DATA                       123         /* Additional data for ASSIGN_OBJ opcode */
ASSIGN_OBJ                $1, !0, 'b'     /* Assign the numeric value 456 to property 'b' of compiled variable 0 object */
OP_DATA                       456         /* Additional data for ASSIGN_OBJ opcode */
FETCH_OBJ_R               $3, !0, 'a'     /* Retreive property 'a' from compiled variable 0 object */
FETCH_OBJ_R               $4, !0, 'b'     /* Retreive property 'b' from compiled variable 0 object */
ADD                       ~5, $3, $4      /* Add those values and store the result in temp var 5 */
ASSIGN_OBJ                $2, !0, 'c'     /* Assign the ADD result to property 'c' of compiled variable 0 object */
OP_DATA                       ~5          /* Additional data for ASSIGN_OBJ opcode */
FETCH_OBJ_R               $6, !0, 'c'     /* Retreive property 'c' from compiled variable 0 object */
ECHO                          $6          /* Echo the value */

What's important to see here is that the properties are refetched each time a read or write is performed on them, which at first glance looks as bad as the pre PHP5.1 way of dealing with variables. Don't let your enthusiam for compiled variables blind you though. Remember the magic __get(), __set(), __offsetget(), and __offsetset() methods which objects allow for. These overloading tricks are great, but they mean that the variable returned by one fetch may not be the variable returned by a subsequent fetch. It's unfortunate that this can't be guaranteed, but it's the reality of a dynamic language like PHP. Know your particular class isn't overloaded? You can get that speedup (at least some of it) back by using good 'ol references to turn your object variables into simple variables:

<?php
$a = &$f->a;
$b = &$f->b;
$c = &$f->c;
$a = 123;
$b = 456;
$c = $a + $b;
echo $c;
?>

Becomes:

FETCH_OBJ_W               $0, !1, 'a'     /* Retreive property 'a' from compiled variable 1 object */
ASSIGN_REF                $1, !0, $0      /* Make compiled variable 0 a reference to the retreived variable */
FETCH_OBJ_W               $2, !1, 'b'     /* Retreive property 'b' from compiled variable 1 object */
ASSIGN_REF                $3, !2, $2      /* Make compiled variable 2 a reference to the retreived variable */
FETCH_OBJ_W               $4, !1, 'c'     /* Retreive property 'c' from compiled variable 1 object */
ASSIGN_REF                $5, !3, $4      /* Make compiled variable 3 a reference to the retreived variable */
ASSIGN                    $6, !0, 123     /* Assign the numeric value 123 to compiled variable 0 */
ASSIGN                    $7, !2, 456     /* Assign the numeric value 456 to compiled variable 2 */
ADD                       ~8, !0, !2      /* Add compiled variable 0 to compiled variable 2 */
ASSIGN                    $9, !3, ~8      /* Assign the value of temporary variable 8 to compiled variable 3 */
ECHO                          !3          /* Echo the value of compiled variable 3 */

Now, this particular example is actually a few more opcodes, and a little bit more work for the engine too, but the more your code following this point uses the local/simple-variable/reference copies rather than the object copies, the balance will quickly tip towards your favor because the variables are already fetched, and they don't need to go through the expensive re-fetch process (which is worse for objects than it ever was for regular variables).

Another caveat to CVs, is that they are entirely scope-local. This should make since as $a in the globals scope is not the same as $a in a given function. What this means for your script, is that when execution enters a new function (execution scope) the CVs for that function are a blank slate and everything has to be fetched anew, even if that function was called before.

Globals and Statics

Statics and Globals are treated to CV status, but only by way of the reference trick I just mentioned:


<?php
static $bar;
echo $bar;

Turns into:

FETCH_W      static      $0, 'bar'
ASSIGN_REF                   !0, $0
ECHO                         !0

What does that mean for your use of the $GLOBALS array? That's right, the global keyword is technically faster. Now, I want to be really clear about one thing here. The minor speed affordance given by using your globals as localized CVs needs to be seriously weighed against the maintainability of looking at your code in five years and knowing that $foo came from the global scope. something_using($GLOBALS['foo']); will ALWAYS be clearer to you down the line than global $foo; /* buncha code */ something_using($foo); Don't be penny-wise and pound foolish..

No, seriously. This post shouldn't be taken as a guide-book to speeding up your apps, they're slow for other reasons.

Doctor! Doctor! It hurts when I do this!

2006-05-10T15:48:00.000-07:00

I came across a bug while working on runkit today. On testing a batch of changes I'd made I was suddenly* running into a buffer overrun error on any request which instantiates the Runkit_Sandbox class. A few valgrind and gdb rounds later and I'd traced the corruption to apc.enabled's ini_entry->value which was actually encountering a form of double-free.

Now, I have no illusions regarding the fact that runkit is a black sheep. The things it does are contrary to PHP's design and probably shouldn't be done, so my first assumption is that APC is not the one at fault and that runkit is just coincidentally stepping on APC's toes. Time to break out the caffeine and get dirty...

After a little more digging I notice this block of code in APC:

PHP_MINIT_FUNCTION(apc)
{
   ZEND_INIT_MODULE_GLOBALS(apc, php_apc_init_globals, php_apc_shutdown_globals);

   REGISTER_INI_ENTRIES();

   /* Disable APC in cli mode unless overridden by apc.enable_cli */
   if(!APCG(enable_cli) && !strcmp(sapi_module.name, "cli")) {
       zend_alter_ini_entry("apc.enabled", sizeof("apc.enabled"), "0", 1,
                       PHP_INI_SYSTEM, PHP_INI_STAGE_ACTIVATE);
   }

   if (APCG(enabled)) {
       apc_module_init(module_number TSRMLS_CC);
   }

   return SUCCESS;
}

Ignore for the moment the fact that it's claiming we're in the ACTIVATE stage (we're really in the STARTUP phase -- but that's not where the problem lies). The problem with this statement is that zend_alter_ini_entry() uses estrndup(), a call which assumes that thread storage has been spun up and that the per-request pointer list is ready for the new entry to be indexed. During MINIT, that's just not the case.

The "right" solution for this is to move the CLI mode override to RINIT and sure enough, this fixes the overrun/double-free issues just fine. The trouble is, it's also a wasteful proposition. Compound this by the fact that the error only presents itself when all of the following conditions are met:

APC being used on command line
apc.enable_cli switch not set
Thread Safety turned on (requires --enable-maintainer-zts during bulid)

Should APC take a performance hit in order to satisfy such an edge case scenario? &#@% NO! This is one bug that can be casually swept under the rug. Move along now, nothing to see here.

P.S. - In case you were wondering, the only reason I didn't notice this before is that I'd only just recently added APC to my development build. The coincidence of this bug showing up with this last batch of changes was precisely that, a coincidence.

re2c is no lemon

2006-05-08T17:51:00.000-07:00

For the longest time I've been wanting to learn how to actually write a compiler. Sure, I've been twiddling the Zend Engine here and there for a few years, but there's this gulf between tweaking and actually building up from scratch which, until today, I had just never found my way across. Enter PDO_User, my most recent extension for implementing PDO drivers in userspace. Although some uses of PDO_User might be as simple as wrapping real database drivers that don't have a PDO version (e.g. informix), there's also a class of userspace implementation that might want to perform decidedly non-SQL actions via a SQL front end.

Since writing a compiler in userspace is... difficult to say the least, I decided this was the perfect opportunity to sharpen my teeth on the task and provide a generic SQL compiler via the PDO_User class. For the lexer I chose re2c as Marcus has been pushing that and what I'd seen of the syntax made it look pretty straight forward, for the parser I went with lemon which Wez is generally ga-ga over. I considerd flex and yacc, but tossed them out since I've already got a basic understanding of their syntaxes. I wanted a start fresh.

After a couple-three days and some aborted attempts, I've managed to implement a relatively feature-rich SQL tokenizer and compiler accessible to userspace via array PDO_User::tokenizeSQL($sql[, $ignore_whitespce=true]) and array PDO_User::parseSQL($sql). Those of you intimately familiar with SQL specs will certainly find faults in my grammar, and just about everyone should notice that there's no support for floating point numbers yet (a minor oversight). What's important is: (A) I've got a working implementation to build from, and (B) It was damned easy to learn both these tools.

Some thoughts on re2c and lemon:

Not enough samples provided on the manual pages. As an OSS documentation maintainer myself I know that this is easier to complain of than actually do something about.
I got stuck early on with re2c due to having an old version (the one packaged with debian) which didn't support case-insensitive matching. If you run into this problem, check your re2c version. If `re2c -v` isn't even able to output a version, then it's certainly too old.
What the @!#& is the "error" non-terminal for in lemon? The documentation talks about using it to gracefully deal with parse errors, but doesn't talk about how to use it.

If all you want is some sample output, have a look at this:

<?php
  print_r(PDO_User::parseSQL("SELECT foo,bar as b,baz
                            FROM boop LEFT JOIN doop as d
                            ON (bid=did)"));


Array
(
  [type] => statement
  [statement] => select
  [fields] => Array
    (
      [0] => foo
      [1] => Array
        (
          [type] => alias
          [field] => bar
          [as] => b
        )
      [2] => baz
    )
  [from] => Array
    (
      [type] => join
      [table1] => boop
      [join-type] => left
      [table2] => Array
        (
          [type] => alias
          [table] => doop
          [as] => d
        )
      [on] => Array
        (
          [type] => condition
          [op1] => bid
          [condition] => =
          [op2] => did
        )
    )

  [modifiers] =>
  [terminating-semicolon] =>
)

PDO_User and the effects of Slide Driven Development

2006-05-03T14:59:00.000-07:00

Last Fall, Wez and I (and others) met up at the Zend conferrence in Burlingame, CA (US). This was primarily a pre-PDM checkin for those of us who wouldn't be able to make it to Paris to get our thoughts in on the direction of PHP6. Apart from these topics, Wez mentioned wanting to have a userspace bridge for implementing PDO drivers, but that there was entirely too much going on that was (in all honesty) more important.

I agreed on both counts. At the time I was doing my best to ignore the temptation to work on operator... Click through and you'll see just how weak willed I can be :)

Fast forward to last week in Orlando, FL (US) and the php|tek conferrence. As mentioned in earlier blog posts, I think my presentations went over okay for a first couple of talks, but if I have to make one complaint, it's that the subject matter was ill-aimed. I spoke to Sean (one of the conferrence organizers) after the closing keynote and asked him about the upcoming php|works conf in toronto. When he said that one of the focuses would be on database interaction (since it would be taking place along side db|works), my thoughts immediately jumped to the PDO userspace driver which had been running around the back of my mind for months.

By the end of my flight back I had half the implementation written and had generally come to the conclusion that the PDO API was pretty damned nice. Long story short, I've just uploaded release 0.1 of pdo_user which is ready for testing. The file you'll want to pay attention to is: README.OBJECTS as that describes the mechanics of actually implementing a driver in userspace. Another interresting spot to look is tests/globals.phpt which includes a simple, yet functional userspace driver for accessing variable contents as database resources.

I don't expect it to work perfect right out of the box, but it should get the job done. Please Report Bugs when you find them so I can get 'em fixed.

Let's go out to the movies

2006-04-28T18:25:00.000-07:00

As promised yesterday, I've gotten the videos from php|tek uploaded. Note: They're all coverage of the beer chugging contest, so don't expect to be clicking into any presentations.

Please don't code like this...

2006-04-23T10:54:00.000-07:00

As a component of my job, I'm overseeing a fledgling developer in another department as he sharpens his teeth on putting together a simple application for one of my units. The first draft of his code was....well....It's a good example of why some people associate PHP with insecure applications. After some lengthy back and forths we've cleaned up the security related problems and gotten the functionality working. In my recent code review however, I came across this gem:

SELECT userid FROM (
SELECT COMPETENCY.userid,
SUM(CASE WHEN COMPETENCY.cid = 1 THEN 1 ELSE 0 END) AS comp_01,
SUM(CASE WHEN COMPETENCY.cid = 2 THEN 1 ELSE 0 END) AS comp_02,
SUM(CASE WHEN COMPETENCY.cid = 3 THEN 1 ELSE 0 END) AS comp_03,
SUM(CASE WHEN COMPETENCY.cid = 4 THEN 1 ELSE 0 END) AS comp_04,
SUM(CASE WHEN COMPETENCY.cid = 5 THEN 1 ELSE 0 END) AS comp_05,
...You get the idea...
SUM(CASE WHEN COMPETENCY.cid = 35 THEN 1 ELSE 0 END) AS comp_35,
SUM(CASE WHEN COMPETENCY.cid = 36 THEN 1 ELSE 0 END) AS comp_36,
SUM(CASE WHEN COMPETENCY.cid = 37 THEN 1 ELSE 0 END) AS comp_37,
SUM(CASE WHEN COMPETENCY.cid = 38 THEN 1 ELSE 0 END) AS comp_38,
SUM(CASE WHEN COMPETENCY.cid = 39 THEN 1 ELSE 0 END) AS comp_39,
SUM(CASE WHEN COMPETENCY.cid = 40 THEN 1 ELSE 0 END) AS comp_40,
FROM COMPETENCY, COMPETENCY_CODE
WHERE COMPETENCY.cid = COMPETENCY_CODE.cid
GROUP BY COMPETENCY.userid) AS my_view
WHERE (comp_20 = 1 AND comp_19 = 1)

Now, I generally don't like to be TOO abrasive with new developers, don't want to scare them off or anything, but can someone buy this child a SQL book?

Simple Pleasures

2006-04-18T16:10:00.000-07:00

About a week ago a coworker came to me for recommendations on hooking up some broadband at home. I gave her the usual breakdown of the differences between Cable and DSL, what she would need in her laptop to make it connect properly, blah blah blah... So after a little back and forth she settled on a DSL package which offered a decent price and a wireless package which she was told initially included a pcmcia card (later found out it didn't, but they'd be happy to sell her one...).

Since her 266Mhz laptop (running Win98E) only had 28MB of memory we ordered her up a 128MB stick to fill the empty slot (she's trying to do this all on a constrained budget, but there are limits!). We got the memory installed and plugged in an old 10BT pcmcia card I had lying around (a useless spare part to me, but a helpful token for her while she waited for the wireless card to appear) and she was up and running.

A few days later the 802.11g card appeared and this is where things went wonky. She installed the full set of the ISP's software (which included absolutely pointless crap that I'd avoided for her initially when I put the wired card in) in order to get to the wireless card drivers. Not only did the wireless card not work, but the setup died midway, the wired connection stopped working, several random error dialogs started appearing, and the coup de gras.... The backlight on the LCD winked out completely. In short, her one-year contract with the ISP was going to be worthless because her only computer had just exploded.

So she brought the PC back to work with a heartwrenching face of woe. After two days of twiddling with it in between the stuff UC actually pays me to be doing, the laptop is officially singing. The software problems just required uninstalling the ISP's mierda, getting updated drivers from the card vendor, and a tiny bit of registry hacking, nothing phenomenal for a IT person, but not something she could have managed without a reinstall (the process of which would have likely led to the same end result). The LCD was a bit trickier, turns out the lead from the step-up transformer to the backlight lamp had snapped off (coincidental to the software problem, though perhaps aggravated by her frustration induced thwacking) at the bulb so I had to tear open the whole panel and resolder it. Again, nothing more amazing than (effectively) reseating a cable, but this was fun because I'd never dissasembled a panel to that degree before.

Best part of all, when I opened up her laptop to show a vibrant screen (actually better than it had looked before all this started) and successful connectivity to AirBears (UCB's wireless network), the look on her face just said it all.

Sometimes, doing helpdesk work kicks ass.

Good Old Times

2006-04-18T14:46:00.000-07:00

Andrei and Zak recently posted their "Good Old Times" references to getting into the PHP project so I thought I'd put mine forward for good measure. Never let it be said I'm not a follower...

At any rate, it didn't take long for Derick to suggest I refer to the CODING_STANDARDS and resubmit the patch as a unified diff against HEAD. After resubmitting, the patch was accepted and, bouyed by my response, led to another contribution in the form of the getanyrr() function which would later come to be named dns_get_record().

Three and a half years later and I've managed to touch nearly every part of the PHP Core distribution and make a lot of friends (and perhaps some not-so-friendlies) along the way. Cheers to all, you know who you are.

Bunnies bunnies everywhere

2006-04-10T11:25:00.000-07:00

I've long been convinced that coincidence is never an accident. On a day to day basis, the questions that come through ##php will gather in bunches to the point where I can often just up-key to find the last time I answered the question, and state it again. What does this have to do with bunnies? I'm glad you asked.

It started with a random link posted in IRC: http://blog.jonnay.net/archives/632-Introducing-BunnyRegex-easy-regular-expressions,-and-mini-languages-inside-of-PHP..html With a title like that, who can resist? The topic itself is fairly uninterresting, however it turned the conversation to rabbits, or at least rabbit stew.

Within moments, a coworker came by my office with a thank-you present (I'd helped her upgrade her personal laptop and get setup with DSL at home). As part of this thank-you present she offered up a stuffed animal (okay, so I have a thing for stuffed animals, sue me). Guess what species she'd selected? That's right. Wabbit.

Is it Fox Mulder spooky? No. But it's a little bit freak-o...

Speaking of rabbits: Check out my cat (who happens to be named "Bunny"):

You're not bad...for a girl...

2006-01-04T16:38:00.000-08:00

In response to http://news.php.net/php.internals/21374 I offered up a fairly simple extension dubbed "callable" (note, I never wound up dropping this in PECL) which overrides the builtin is_callable() function to call $object->__callable($methodname) in order to determine if a __call() overloaded object is actually capable of calling a particular method.

Those who wanted the funcitonality were happy to have the code and it brought the already-too-long discussion to a close saving list traffic for more important topics.

That same day, one of the list lurkers sent me an email which included the following.....quizzical paragraph:

Not to sound chauvinistic or anything and not meant to be, but you are a woman, I'm very impressed with the code, it's direct and too the point, not what I would expect from someone of your gender as most female programmers I've come across are into a lot of fluff and eye-candy and not so much in pure functionality.

Now, I'm about 95% certain that he did, in fact mean it as a compliment. And while I appreciate the....hrmmm praise isn't quite the right word...good wishes, I'm not really really clear on just how appreciative I should be. It's a little like what I would expect hearing "You're remarkably well spoken for a black person*." would be like... Or something.... How do you successfully answer a comment like that? Is it even possible?

*Not the cliche'd expression, but my liberal-guilt won't let me use it, even as an example of poor taste.

Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
	01	02	03	04	05	06	07	08	09	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31

Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
	01	02	03	04	05	06	07	08	09	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31

Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
	01	02	03	04	05	06	07	08	09	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31

Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
	01	02	03	04	05	06	07	08	09	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31

Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
	01	02	03	04	05	06	07	08	09	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31

Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
	01	02	03	04	05	06	07	08	09	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31