Monday, April 28, 2008

Sorted Details

Q: What did the layout manager say to the anxious component?
A: Contain Yourself!

Q: Is it possible for components to be arranged without a layout manager?
A: Absolutely!

Q: How did the component feel about its first animated layout?
A: It was a moving experience.

Q: What do you call the animation objects owned by a component?
A: The component's personal Effects.

Q: Why is an animating container a risky real estate investment?
A: It's a transitional area.


I've been playing around with animation and layout recently, and I wondered how I could combine the two. That is, when a container changes state and wants to re-layout its children, wouldn't it be nice if it could animate that change instead of simply blinking them all into place?

Suppose you have a picture-viewing application, where you're looking at thumbnails of the photos. Sometimes you want them to be quite small so that you can see more pictures in the view. But sometimes you want them to be larger instead so that you can see more details in any particular picture. So there's a slider that allows you to dynamically control the thumbnail sizes.

By default, in this application as well as other real applications that it's modeled on, changing the thumbnail sizes causes the application to re-display itself with all of the changed thumbnails in their new location. But what I want is for the change in size to animate, where all of the thumbnails will move and resize to their new location and dimension.

It turns out that this is not that hard to achieve, using the built-in effects of Flex.

Application: SlideViewer

First, the application (photographs courtesy of Romain Guy):

Usage: Move the slider around, which changes the sizes of the pictures. See the pictures move and resize into their new locations.

So how does it work?

How It Works

The application uses the built-in Move and Resize effects of Flex on each of the pictures. These effects, when combined in a Parallel effect, do exactly what we need - they move and resize their targets. But there are a couple of additional items that you need to account for in order for it to work correctly for our animated layout situation.

For one thing, you have to disable automatic updates to the container of the pictures during the animation. If you don't do this, the container will be trying to lay the pictures while you're trying to move them around. It's like watching an inverse of that video game anomaly, where your screen is having a seizure watching you.

Another trick is one that Flex transitions use internally; we first record the properties of all of the animating objects at their start location, using the Effect.captureStartValues() function, then we change the thumbnail sizes, force a layout to occur,l and run the effect. This approach causes our effect to automatically animate from those cached initial values to the new values that our container has imposed on the resized images.

The Code

There are three source files for this application (downloaded from here):

  • This class handles the rendering of each individual photograph on a custom background.
  • This subclass of the Tile container loads the images and creates Slide objects to hold them.
  • SlideViewer.mxml: This is the main application class, which arranges the basic GUI elements and creates and runs the layout animation when thumbnail sizes change.

This class is a simple custom component that acts as a container for the Image object created from each photograph. It also adds some custom rendering to get a nice slide-like background for the image.

First, there is a simple override of addChild(), so that Slide can cache the Image object that it is holding. This caching is not necessary, since we could retrieve this object from the component's children list at any time, but it seemed like a better way to cache a commonly-access object:

    override public function addChild(child:DisplayObject):DisplayObject
        var retValue:DisplayObject = super.addChild(child);
        image = Image(child);
        return retValue;

The real reason for the custom subclass is the custom rendering that we perform to get the fancy gradient border for each image. This happens in the updateDisplayList() function. The first part of the function is responsible for sizing and centering the contained Image object appropriately. The remainder of the function handles the custom rendering to get our nice gradient border:

    override protected function updateDisplayList(unscaledWidth:Number,
        super.updateDisplayList(unscaledWidth, unscaledHeight);
        // Set the image size to be PADDING pixels smaller than the size
        // of this component, preserving the aspect ratio of the image
        var widthToHeight:Number = image.contentWidth / image.contentHeight;
        if (image.contentWidth > image.contentHeight)
            image.width = unscaledWidth - PADDING;
            image.height = unscaledWidth / widthToHeight;
            image.height = unscaledHeight - PADDING;
            image.width = unscaledHeight * widthToHeight;
        image.x = (unscaledWidth - image.width) / 2;
        image.y = (unscaledHeight - image.height) / 2;
        // Draw a nice gray gradient background with
        // a black rounded-rect border
        var colors:Array = [0x808080, 0x404040];
        var ratios:Array = [0, 255];
        var matrix:Matrix = new Matrix();
        matrix.createGradientBox(width, height, 90);
        graphics.beginGradientFill(GradientType.LINEAR, colors,
                null, ratios, matrix);
        graphics.lineStyle(.04 * width);
        graphics.drawRoundRect(0, 0, width, height, .1 * width, .1 * width);

This class creates all of the Slide objects that will be in the view and also handles later resizing events due to movement of the slider component.

The images are added to the application through the addImages() function. This function uses a hard-coded list of embedded images. The original version of this application was written with AIR, and I used the new File APIs to allow the user to browse the local file system and load images dynamically. That's important functionality, and it's preferable to the hard-coded approach below, but it's also orthogonal to the effect I'm trying to demonstrate here, so I dumbed-down the application to just load in some known images instead. Perhaps in a future blog I'll release the code and app for the AIR version. Note also that I'm only using embedded images here to simplify things for the weblog; it's easier to deploy this thing standalone (with embedded images) rather than having it refer by pathname or URL to images that live elsewhere, just because of the limited nature of this blogging engine (Grrrrr. Don't get me started...).

addImages() iterates through the list of embedded images (stored in the bitmaps[] array - check the source file for details), creating a Flex Image component for each one with the bitmap as the source property, and then creates a containing Slide object for each resulting Image.

    public function addImages(event:Event):void
        // Create an Image object for each of our embedded bitmaps, adding
        // that Image as a child to a new Slide component
        for (var i:int = 0; i < bitmaps.length; ++i)
            var image:Image = new Image();
            image.source = bitmaps[i];
            image.width = slideSize;
            image.height = slideSize;
            image.cacheAsBitmap = false;
            if (!slides)
                slides = new ArrayCollection();
            image.addEventListener(Event.COMPLETE, loadComplete);
            var slide:Slide = new Slide();        
            slide.width = slideSize;
            slide.height = slideSize;

Note in the above function that we added an event listener, loadComplete(), that is called when each image load is finished:

    private function loadComplete(event:Event):void
        var image:Image = Image(;
        // Enable smooth-scaling
        Bitmap(image.content).smoothing = true;

We added the loadComplete() functionality so that we could enable smooth-scaling for each image. This makes the thumbnails look better in general, but it also vastly improves the animated effect when the images are being scaled on the fly. Without smooth-scaling there can be some serious 'sparkly' artifacts as colors snap to pixel values. Unless all of your pictures are of fireworks and disco balls, the scaling artifact without smoothing enabled is probably not one you'd like.

When the slider value changes in the GUI, that value gets propagated as the new size of the Slide objects through setter for the slideSize property:

    public function set slideSize(value:int):void
        _slideSize = value;
        for (var i:int = 0; i < slides.length; ++i)
            var slide:UIComponent = UIComponent(slides.getItemAt(i));
            slide.width = value;
            slide.height = value;


Now let's look at the code for our main application file, SlideSorter. This class is written mostly in MXML, with some embedded ActionScript3 code to handle starting the animation.

First, there's our Application tag. There's nothing amazing that happens here, but you can see that we define a nice gray gradient background for the application as well as a callback function for the creationComplete event, which will call the addImages() function we saw earlier to load the images:

<mx:Application xmlns:mx=""
    xmlns:comps="components.*" layout="absolute"
    backgroundGradientColors="[#202020, #404040]"

The GUI of our application is quite simple; there is just the container for the slides and the HSlider that determines the thumbnail size. The container is defined by this simple declaration:

    <comps:SlidesContainer right="10" bottom="45" top="10" left="10" id="slides"/>

The HSlider is a little more interesting. It defines a data binding expression for the slider value, to ensure that the location of the slider will be set according to the thumbnail size. This is really only for the first view of the application, so that the slider control is in the correct initial location; after that, it is the change in slider value that will change the thumbnail size, so they will automatically be synchronized through that other mechanism. Also, note that change events on the slider will call the segue() method, which is where we set up and run our animation. Note that I changed the slideDuration property from the default 300 down to 100; this makes our animation start sooner, rather than waiting for 300 milliseconds just for the slider animation to complete before we receive the change event.

    <mx:HSlider id="sizeSlider" bottom="10"
        width="100%" height="27" value="{slides.slideSize}"
        change="segue()" slideDuration="100"
        snapInterval="25" minimum="25" maximum="400"/>

Our animation for the layout transition is defined by a simple Parallel effect, which composes a Move and a Resize effect together such that they play at the same time. It is important to note that both Move and Size have built-in intelligence that can determine what values to animate from and to in the right situation. For example, when we set up our animation to collect the start values and then move all of the thumbnails to their end locations/sizes (by calling validateNow() on the container), the effects are clever enough to sense that we want to animate from those initial values to the values that they currently detect.

    <mx:Parallel id="shifter">

Finally, we have the ActionScript3 code that we use to set up and run our animation. Note that we already have "set up" our animation by declaring the shifter Parallel effect above. But we need to tell this animation which targets to act on when it runs. (In the situation of this particular demo, we could probably just do this once, since the components in the view are hard-coded at startup. But for a more general case where there might be image changes between transition runs, I wanted the behavior to be more dynamic).

This little snippet of code sets the targets to animate (the slides in our SlidesContainer object) and collects the starting location/sizing info for those targets. It then sets the final slideSize property that we want to animate to and forces a layout validation on the container to position the objects in place. (Note, however, that this layout is not yet visible. It runs the logic internally, but those locations are not changed onscreen immediately; we're safe as long as we're still in this code block). It then sets autoLayout to false; this statement tells our layout manager that we don't want it interfering with trying to move and resize our slides while we're in the middle of animating them. This allows us to do absolute positioning of the objects inside of our container even though our container normally wants to position the slides in a flow-layout style. We then add an event listener to call our segueEnd() function when the animation is complete, which we'll see below. Finally, we play() our animation, which does what we want. Internally, the Move/Resize animation collects the end values for our slides and runs an animation from the start to the end values.

            public function segue():void
                shifter.targets = slides.getChildren();
                slides.slideSize = sizeSlider.value;
                slides.autoLayout = false;
                shifter.addEventListener(EffectEvent.EFFECT_END, segueEnd);

Finally, we handle the callback when the animation finishes in our segueEnd() function. This simply resets the autoLayout property on our container, to make sure that it resumes its normal behavior.

         public function segueEnd(event:Event):void
             slides.autoLayout = true;

100 goto end

That's it. I frankly expected this to be much harder than it was. It took a bit to work through the details of how to set up the animation appropriately (Effect.captureStartValues()) and how to get the layout manager out of my way during the animation (autoLayout = false), but otherwise all of the intelligence for doing this kind of transition was built into the existing Flex framework.

Note that there are some limitations to this kind of transition. For example, the animation will detect changes only at the top-level children of the container. So, for example, I originally used a Box container to hold each slide, but I found that the images inside the slides were being resized immediately and were not being animated from start to finish. Making the change to simply add the images as direct children of the Slide component fixed this. Ideally, a more robust animated layout system would work with hierarchies of layout; that's something I'll have to think a lot more about.

Complete source code for this application is available here.

Tuesday, April 22, 2008

JavaOne, Too

Romain Guy and I will be returning to JavaOne this year to give another talk this year. We thought about presenting on horticulture, or the effect of air travel on the Amazon rain forest, but in the end we decided at random to talk about this topic: Filthy Rich Clients: Filthier, Richer, Clientier. And though it was tempting to just present stuff we'd already written (with 550 pages of material in the book, there's a lot to, well, steal for presentations), we thought it would be more fun and to do new stuff. Who needs sleep? We hope you can make it to the session.

For anyone that signed up for the session already, we hope you got word that the conference has shuffled us out of our original slot on Tuesday and put us on Thursday afternoon instead (4:10 - 5:10 pm). Maybe they just realized that everyone enjoys pre-enrolling for sessions so much that they figured you'd want another chance to log in and see how that change affected your schedule. And if you didn't pre-register for the session yet, you might want to do so; I've heard that the room is filling up (it's not clear why - maybe it was the "free money!" mention in the abstract. Or maybe it's Romain's groupies. Again.).

There's other stuff happening as well. For one thing, there's an author signing for our book. Maybe someone can explain to me why you actually want us to deface your book. It's a nice book, clean and professional looking, and you want us to write in it? I understand having a famous author sign a book they wrote; heck, it'll be worth more when they're dead. But the authors of Filthy Rich Clients? The only thing worth more when I'm dead will be my life insurance policy. Anyway, we'll be there at the bookstore, signing anything you put in front of us except a blank check.

Finally, there's an Adobe event on Wednesday night, starting at 5:00 at the Metreon. I'm giving a short presentation with James Ward. Come see some of the Flex stuff I'm working on now. And then come to the Adobe party at Jillian's. Go to the Adobe booth on the show floor for details.

I'll also be working the Adobe booth off an on, doing a session or two and milling around trying to look useful. Stop by and say hi.

Monday, April 14, 2008

Time's Up

In trying to figure out the wacky results I was seeing from running the BubbleMark application, I discovered that timer events are constrained in Flash (and therefore Flex) when running inside the browser. This limitation extends to all methods of timing, including: the Timer class, handling ENTER_FRAME events, and manually calling callLater().

Meanwhile, using these timing methods outside of the browser, either in the standalone Flash player or in an AIR application, returns very different results because Flash is no longer constrained by the browser's throttling.

I wrote a simple application, TimerBenchmark, to show the differences between the different timing mechanisms inside the browser and out. The user selects the timing mechanism desired, optionally enters a resolution value (the milliseconds between timing events) for the Timer case, and the test automatically runs. As the test runs, it counts how many timing events are executed per second and displays that result in the "FPS" field at the top of the application. For example, here's the application, runnin in your browser:

How it Works

First, let's see the code. Everything is in the TimerBenchmark MXML file, which you can download and play with. The GUI is set up with a combination of labels, radio buttons, and a TextInput control for the Timer resolution:

    <mx:Label id="fps" x="75" y="48"/>
    <mx:Label x="10" y="48" text="FPS"/>
    <mx:Label x="114" y="76" text="Timer resolution"/>
    <mx:TextInput x="218" y="74" width="39" id="resolution"
        enabled="{timerButton.selected}" text="1" enter="restart()"/>
    <mx:RadioButtonGroup id="benchmarkType" change="restart()"/>
    <mx:RadioButton id="timerButton" x="10" y="74" label="Timer" groupName="benchmarkType"/>
    <mx:RadioButton id="enterFrameButton" x="10" y="100" label="enterFrame" groupName="benchmarkType"/>
    <mx:RadioButton id="callLaterButton" x="10" y="126" label="callLater" groupName="benchmarkType"/>

Any change in the Timer resolution input or one of the radio buttons will call restart(), which will reset any running benchmark and then execute the appropriate test:

            private function restart():void
                if (timer)
                removeEventListener(Event.ENTER_FRAME, enterFrame);
                callLaterRunning = false;
            public function runBenchmark():void
                switch (benchmarkType.selection) {
                    case timerButton:
                    case enterFrameButton:
                    case callLaterButton:

All of the tests call benchmarkCallback() to do the actual work of timing the mechanisms:

            private function benchmarkCallback():void
                var nowTime:int = getTimer();
                var delta:int = (nowTime - startTime);
                if (delta > 1000)
                    fps.text = String(numFrames * (delta / 1000));
                    numFrames = 0;
                    startTime = nowTime;
                if (callLaterRunning)

benchmarkCallback() counts the number of times that it's been called since the last measurement time. Every second, it calculates the fps result, based on numFrames and the time elapsed since the 0th frame. It updates the fps label appropriately and then resets numFrames and startTime. Note that our benchmark function doesn't actually do anything useful. It's a bit like our politician, except that runBenchmark() is intentially unproductive because we only want to time how fast the timing mechanisms can run, not how fast we process anything when they get there. It's important to be able to disentangle these concepts from each other, lest we run into the confusion I encountered in the BubbleMark program, where the throttled rate of Timer was affecting the perceived computation or rendering performance of the application.

startTimerTest() is the function called when the Timer button is pressed:

            public function startTimerTest():void
                if (timer && timer.running)
                timer = new Timer(Number(resolution.text));
                timer.addEventListener(TimerEvent.TIMER, timerEvent);
            private function timerEvent(event:TimerEvent):void

This mechanism works by asking the Flash runtime to call our callback function, timerEvent(), every n milliseconds, where we set n to be equal to the value in our "Timer resolution" text input control.

The enterFrame mechanism looks like this:

            private function startEnterFrameTest():void {
                addEventListener(Event.ENTER_FRAME, enterFrame);
            public function enterFrame(event:Event):void

This mechanism works by telling the Flash player to call our callback function, enterFrame(), whenever the ENTER_FRAME event is processed (which is, by definition, once per 'frame' that Flash renders).

Finally, the callLater() benchmark is run by the following code:

            private function startCallLaterTest():void
                callLaterRunning = true;

in addition to the final two lines of our benchmarkCallback() function:

                if (callLaterRunning)

This mechanism works by asking the Flash runtime to call our function, benchmarkCallback(), at the next available opportunity, which occurs twice per frame.

Resolution Results

Here are some sample results with the application running inside the browser on my Windows Vista system:

As you can see, the Timer and EnterFrame mechanisms are both throttled at about 65 frames per second. Meanwhile, the callLater mechanism is getting serviced at about twice that rate.

The reason for the throttling is that the timing system that Flash uses inside the browser is pegged at a maximum, which is a combination of a limitation of the timing mechanism of the browser itself and the refresh rate of the monitor (which for my laptop is 60 fps). The main reason for this limitation is to avoid pegging the CPU just to run animations that really don't need to run faster than the monitor's refresh rate. After all, why waste time and CPU resources updating something that the user will only see updated at the refresh rate of the monitor?

Both the Timer mechanism and enterFrame are maxed out to this same limit. The callLater mechanism sees about twice this rate because it is actually called exactly twice per frame in Flash. It is called once when leaving the enterFrame() call and another when processing the actual frame rendering, so an application that requests a callback through callLater can expect to get that callback twice per frame.

Meanwhile, here are some results from running in the standalone Flash player:

Now we can see that the Timer approach gets close to our ideal fps number; since the resolution was set at 1 and we set the internal Flash frameRate property to 1000, the ideal result is 1000 frames per second. The enterFrame mechanism seems to be throttled at a much lower 250 fps, although this is still significantly better than the browser limit of 65 fps. The callLater mechanism is again twice that of enterFrame, which is what we would expect.

I wrote an AIR version of the benchmark as well. I won't bother to post that version, since I got the same results as the standalone Flash player. This is as expected, since the AIR version is basically running the Flash player standalone internally, freed from the browser constraints. So it is essentially equivalent to the standalone Flash player version.

Resolution Resolution

So where does that leave things? What are we to make of the different timing results?

Well, we mainly need to be aware of the issues, especially when running timing-sensitive code. What mechanism should you use? What frame rate do you want? Are you really getting the results you think you are, or is there some environmental effect that is having an impact on the timing of your application?

In most real-world situations, your application need not (and probably should not) run at a faster rate than the resolution of the screen, so the limitation of the timing mechanisms to refresh rates is perfectly reasonable. In particular, in graphical applications any updates to the scene at greater than refresh rates is simply a waste of cycles, since the user won't see that many updates. But if you need to run some function at greater than this rate for some reason, be aware of the factors called out above and maybe even do some benchmarking of your own to make sure you're getting what you think you're getting.

For More Information

Here's my previous post on Timer's effect on the BubbleMark results

Here's James Ward's post on the same topic, with some alternate benchmark applications and results

Here'sa blog post by Adobe's Tinic Uro on the throttled frame rate in Flash

Here's the original BubbleMark site

Here's the source code for my application, TimerBenchMark

Here's the Flash SWF file for the benchmark

Wednesday, April 9, 2008

Off the [Bubble]Mark

James Ward has been looking into the BubbleMark application recently, and we were talking about some of the strange results we saw in different situations. For example, I was able to take the existing application, which gets something like 65 frames per second in IE7 on my Vista laptop, recast it as an AIR application, and get about double that frame rate, or 130 fps. James posted some variations of the application on his blog; check them out to see what I mean.

(Some background, for those that didn't avail themselves of the awesome opportunity of clicking on the recent link to the site: BubbleMark is an application that moves a number of 'balls' around in a window, and bounces them off one another while doing so, and then measures how fast this critical and awe-inspiring process happens per second).

What's going on? Is that gremlin in my keyboard back to haunt me? Or have I been staring at the screen too long and I'm mis-reading the numbers? Or maybe age is creeping up on me and I'm just getting things wrong more often, as a prelude to slipping on the ice and breaking my hip.

Circular Reasoning

The fundamental problem with the current test is that it is limited by the resolution (the time between events) of the timing mechanism used by the application. In this case, the timing mechanism is a Flash Timer object that is scheduled to call our code at frequent intervals (supposedly far more frequent than the frame rate we're seeing). So while the application purports to test the number of frames that it can render per second, it actually tests the number of times per second that the Timer sends out events. In the situation where Flex can render the balls much faster than the Timer frame rate, this means that the frames-per-second (fps) reported has very little to do with the speed at which the balls are being moved or rendered.

By producing an alternate test that runs on AIR (which takes Flex out of the browser and runs Flex applications in a standalone window on the desktop), we remove a constraint where the Timer resolution tends to be more limited in the browser than in standalone applications. Since the Timer can call our ball-moving routine more often in this AIR version, and since we were taking less than a frame's worth of time to move the balls before (although that wasn't reflected in the fps result), we get a higher effective frame rate and better results.

So the AIR version is closer to the ideal, where we're actually measuring more of what the test was presumably intended for; moving and rendering the balls.

But there's still a problem here, which has been implied by the wide variation in the results. The test is confusing three orthogonal issues and trying to combine them into one result (fps). The three things actually being tested are:

  • Calculation speed: How fast can the runtime position the balls, including moving them, calculating collisions, and offsetting the collided balls? This comes down to VM and math performance, since it's just raw computations.
  • Rendering performance: There is a certain amount of time spent each frame actually drawing the balls on the screen. That seems to be insignificant in our case, compared to the other factors here, but you would think that a graphics benchmark might actually care about this item them most. How fast can we draw the balls? I don't know, because we're spending most of our time just waiting around for Timer events and calculating the balls' positions. That's like timing how long it takes you to get dressed in the morning, but starting the timer the previous night when you go to sleep.
  • Timer resolution: This is the primary issue called about above. I don't believe it was ever the intention for the benchmark to measure the rate at which the various runtimes could schedule callbacks. Because, well, that'd be a pretty dull graphics benchmark. But that's exactly what's happening here; we're not measuring graphics performance at all in the default test.

To add to the problem of these three factors getting each others' chocolate stuck in the others' peanut butter, you've got the problem that all of the different platforms (Flex, Java, .net, etc.) probably have different reliance on these three factors (are they faster at rendering but slower than computation? Do they have more limited Timer resolution? How do browsers affect these factors on each platform? Or hardware? Or operating systems?)

I think that if you wanted real results that you could use and compare from this benchmark, you'd have to de-incestuate the factors and measure them all separately. I could envision three competely separate tests coming out of this:

  • Calculation speed: Let's have a non-graphical test that simply moves/collides the balls as fast as possible and reports how many times it was able to do this per second. No need to render here; we only care about how fast we can do the math and related setting of properties.
  • Rendering performance: Render a given number of balls as fast as you can and report how often you can do it per second. In order to not run up against the Timer resolution issue, you'd have to adjust the number of balls so that you could be assured that you would be spending very little time waiting around between callback events. In fact, you could even try to render far more than could be done in a single frame, but then report a number like balls/second instead of frames/second.
  • Timer resolution: Write a test that simply issues callbacks from the Timer mechanism, without doing any work in those callbacks, to see how many times this happened per second. (I wrote this test and posted a blog about it; see that post for more information on this no-op timer test).

With results from these three tests, it would be easier to understand where a platform was being bottlenecked: at the runtime, rendering, or resolution level.

At this point, I'd love to say, "...and here's that benchmark!" But I'm not going to, because I've already spent a pile of time just trying to understand the weird results I was seeing from Bubblemark. If someone else wants to take a whack at it, that'd be great. But at the very least, I hope this helps you understand and filter the results you see from the BubbleMark tests a little better.....