Mixing UIKit with OpenGL ES on iPhone

UIKit is a rich library with classes that have been highly polished by Apple. Lot's of development time can be saved by using it in the right way in your game's UI, however, there are problems that can arise.
Standard buttons and the like tend to sit on an OpenGL view without too many issues but as soon as you start using something slightly more complex such as a UISlider, the problems start.
I have a screen 90% drawn in OpenGL with an animated 3D model of a ship rotating round in it. This is kind of important as I need the ship to update in real time whilst a sliding control underneath shows different items the player may select. I wanted to make use of the UISliders functionality for this task.
I started by having a horizontal UISlider with a content width set based on the number of items and their size. I set it to not clip subviews. This allows me to display parts of items on screen that aren't currently centred by the page snapping of the UISlider.

The first problem I encountered was when I pressed down on the UISlider it would block the OpenGL ES layer animating, so the ship and any other OpenGL animation would pause. This is really ugly to watch.

The first step to fixing this is to change the display link run loop mode from

[displayLink addToRunLoop:[NSRunLoop currentRunLoop] forMode:NSDefaultRunLoopMode]

[displayLink addToRunLoop:[NSRunLoop currentRunLoop] forMode:NSRunLoopCommonModes]

Stackoverflow: OpenGL Animation freezes with UIScrollView

This fixed the OpenGL blocking behaviour but meant that some touch events were being missed on the UISlider. So I investigated further and found a fix to this where someone put the OpenGL draw loop in a grand central dispatch thread. Like so:

if (dispatch_semaphore_wait(frameRenderingSemaphore, DISPATCH_TIME_NOW) != 0) { return; }

dispatch_async(openGLESContextQueue, ^{ [EAGLContext setCurrentContext:context]; // Render here dispatch_semaphore_signal(frameRenderingSemaphore); });

I found this at:

CADisplayLink OpenGL rendering UIScrollView behaviour

courtesy of the 'Molecules' app creator Brad Larson

This fixed the issue entirely... however, for some reason my UIView slider items would sometimes just not appear now! Sometimes they would appear after a large delay (10 seconds or so) and sometimes a variable number of them would appear. I then investigated a work around for this.

It turns out that there are several hacky ways to force the main run loop to update after calling a UIView's setNeedsDisplay function and whilst these seemed to help somewhat they were far from a concrete 100% fix for me.

Stackoverflow: more ways of controlling UIView rendering and some hacks

In the end I decided to stop using custom UIViews for rendering and just kept the UISlider for it's polished physics whilst rendering my own OpenGL buttons in the same positions as the slider's items. This gives me no further rendering issues but let's me keep using the best part of the UISlider - it's physics and touch functionality, whilst leaving the graphics control entirely with me using OpenGL.

Force Landscape framebuffer in OpenGL ES 1.1

I've fixed an orientation bug I had related to the first frame buffer created having incorrect dimensions - as if the app were launched in portrait mode.

Before in my plist:

UISupportedInterfaceOrientations

UIInterfaceOrientationLandscapeLeft
UIInterfaceOrientationLandscapeRight


Changed to:

UISupportedInterfaceOrientations

UIInterfaceOrientationLandscapeRight
UIInterfaceOrientationLandscapeLeft


In both situations the correct landscape orientation comes through the root view controller but only with the latter array order does it make the initial framebuffer with the correct dimensions!

Please note, the frame buffer is created twice on launch still. The very first time is always portrait dimensions but the second is now landscape (960x640 on iPhone 4). This was not happening until I changed the order of the supported orientation array above.

This bug was in OpenGL ES 1.1 using the now deprecated EAGLView.

Articles on Programming Optimisation

I've linked below some great articles on optimisation. Some of the later ones are quite low level and more just points of interest rather than proving especially useful these days, at least for my purposes.

Some key rules to follow
http://assemblyrequired.crashworks.org/2008/12/22/ea-stl-prevents-memory-leaks/#more-92

Lots of general tips and great blog
http://assemblyrequired.crashworks.org/category/programming/

Good one on branching and data layout for game systems/containers again
http://altdevblogaday.com/2011/11/10/optimisation_lessons/

Optimizing for cache coherence – excellent, and a good reminder of why thinking about the data flow in a program is so important.
http://supercomputingblog.com/optimization/taking-advantage-of-cache-coherence-in-your-programs/

From:
http://gamedev.stackexchange.com/questions/853/c-low-level-optimization-tips

A very, very low-level tip, but one that can come in handy:

Most compilers support some form of explicit conditional hinting. GCC has a function called __builtin_expect which lets you inform the compiler what the value of a result probably is. GCC can use that data to optimize conditionals to perform as quickly as possible in the expected case, with slightly slower execution in the unexpected case.

if(__builtin_expect(entity->extremely_unlikely_flag, 0)) {
// code that is rarely run
}
I've seen a 10-20% speedup with proper use of this.

http://gamesfromwithin.com/data-oriented-design

Use const when possible – it can make the compiler optimise things.

http://diaryofagraphicsprogrammer.blogspot.com/2008/11/iphone-arm-vfp-code.html

_restrict keyword for telling the compiler pointers won’t alias (one pointer -> one instance of data ratio)
http://cellperformance.beyond3d.com/articles/2006/05/demystifying-the-restrict-keyword.html

:? Sometimes compiles to be faster than if else
http://assemblyrequired.crashworks.org/2009/01/04/fcmp-conditional-moves-for-branchless-math/

purecall and virtual function overhead
http://thetweaker.wordpress.com/2010/06/03/on-_purecall-and-the-overheads-of-virtual-functions/

Asynchronous Resource Loading

After reading about Alex Okafor's difficulties with CGContextDrawImage on separate threads for loading graphical resources I decided to just load the textures on the main thread and save myself the hassle of changing my png lib.

The main bottle neck at the moment is the level geometry data and models so I've shifted that to load on a separate thread and permit me to add a loading animation. I decided to use posix threads for compatibility reasons rather than Apple's NSThreads api.

I thought I'd save texture space by using a single frame of animation and rotating it in code but this didn't work very well. The texture pixels were slightly off at different rotation values, so I opted for animation frames to keep a polished 'locked in' look to the animation.

The loading screen now has a loading bar for overall progress with an animation that gives the user some feedback whilst only stalling slightly initially for the texture loads, which are non-threaded. I feel this is polished enough and fully threading the loading screen is not worth the time investment at this stage.

Update 01/10/11:

I tried adding the OpenGL shared context stuff anyway and it all worked fine with my texture loading code, wasn't much of an effort to add either. All loading is now entirely done on a second worker thread. Shout out to BigMac for the sanity check!

Git Source Control with Xcode

Finally got this working after some issues. In the end I found the easiest way was to leave Xcode out of the creation of the repository (I was adding to an existing project).

If you're not a terminal wizard you may find it helpful to enable viewing of hidden files on your mac by typing the following into terminal:

defaults write com.apple.finder AppleShowAllFiles -bool true

This let me get a good look at all the .git folder files. Turn it off by calling the same command with false at the end rather than true.

Git isn't too difficult once you know what you're doing. I started by navigating to the folder I wanted source control for (in my case this was not the entire Xcode folder as it contains lots of other data, I only wanted code to be source controlled) and use the following git commands.

git init
git add .
git commit -m 'First commit'

This creates a git repository in the folder, adds all the files in the folder and writes a first commit with a comment. Next I added the remote copy 'origin' and pushed the local git repository to my master version on a server.

git remote add origin
git push origin master

I now use Xcode to commit changes to my local repository and simply push to the server using the above git command.

Baked lighting and Environment shadows

I decided to kill the dynamic lighting in favour of baked vertex colours. The speed improvement from this should let me keep 60fps whilst having a naturally looking lit environment. I never really needed 'dynamic' lights anyway and they didn't enhance the look of the game that much more than 2D texture lights have done. I've implemented shadow maps, which have transformed the way the world looks. Buildings and architecture pop right out of the screen now to great effect, looking more 3D than previously.

Levels are no longer designed via simply placing tiles, instead all the static geometry in a level is modelled from existing objects placed together in Blender. This means I can still use the tiles but have the flexibility to add new objects as I please. I'm also using Blender to bake in the light vertex colours and create the shadow map. I will continue to use my old editor to place triggers and game items/entities.

There are some left-over texture seams and no shadow on the player's vehicle yet (hopefully I will be able to overlay the shadow multi-texturing onto combatant ships too in the end) but quite a visual improvement nonetheless.

Some of the mesh processing needs to neaten up various things still like texture seams and shadow map texture artifacts. I'll also soon set a minimum ambient light level based on the sun diffuse light being used - as some polys not facing the light source are currently rendered in complete darkness. Finally, I could consider blending ambient occlusion in with the existing shadow map.

Art pipeline, from asset to in-game

I've added some new generic metal alien building models to the game. These are going to be used to add a bit of feel for the theme of the enemies. The background setting is that they've set-up small outposts in various areas they've invaded. Half of them are destructible with destroyed models, which allows me to add another gameplay element/mission objective such as destroy all enemy buildings or similar.

It's so important to have a fast asset pipeline as an indie developer. In my case, as it is at present with me doing all the art, it allows for certain shortcuts. Developing tools to automate an efficient asset build pipeline has so far not being considered worthwhile enough for this project but the increasingly large amount of typed-in data is certainly pushing me to have a proper data-driven tool chain for my next project of similar scale.

First textures are created/modified if need be for the new models. Typically I just draw them straight into the relevant mega texture. Layered art backups are kept when appropriate for more complicated textures, such as scenery textures with baked lighting.

The textures are done, next we create some new models in Blender. I'll typically save all the models using one texture or belonging to one world in the same Blender file.

The next stages should ideally be automated by a tool but I have yet to find the time to build one and so I enter new data for the models (enumerator/ID, material, GameEntity type etc) by hand in the code. I also use a mesh optimiser tool I made to extract and build the exact data I need for my game from the Blender export file.

Now the new assets are ready for the editor. A little further data entry is required to get the new models to appear in the editor's GUI before they can be placed into a scene.

Save the level, boot the game up and load the level. More data entry and possibly coding required if the new game entities are to have new behaviour or functionality (this is always the case when adding a new weapon or enemy atm).

Finally, tick off the task in the spreadsheet! An ESSENTIAL design and planning aid in any project, let alone an indie game of any significant scale. Being able to get an instant visual feedback on progress made is important from a motivational point of view as well as for planning. Looking at the overall process for my current asset pipeline, they're are obvious areas for optimisation and improvement. There are currently many parts of the process where I need to enter data, where a tool could do it. It's all about the cost/benefit ratio between the time to develop a nice tool to sort this stuff and me just quickly entering values. As it is now, I don't imagine a tool is going to favourably enhance timescale on this project but creating one for multiple future projects is certainly a sound idea.

Mighty Dogbroth awakens

A good friend and fellow developer recently started his own GameDev blog. It's got some good content - check out The many guts of a think-box impact at http://dogbroth.tumblr.com/.

gluUnProject fun

The old level editor I'd made was a bit of a hack that had been created to get a few concept/test levels out for Annihilation Arena II in early development. The lack of design had really began to show and it was time to make a new editor. The main difference was a push to make it use the exact same data and engine as the actual game, bringing the two projects closer together. Since I was starting afresh I thought it a good idea to do things properly from the get go and make it display everything more or less as it would look in game, in full 3D.

Having re-worked the XML level and model file formats to be far prettier and more adaptable for the future and built the core systems for handling editor objects etc. I moved onto the new graphical display for the level editor.

AAII uses a sort of 2.5D isometric view, which means some sort of coordinate transformation has to be done converting mouse coords to in-game 3D coords. Luckily for me I had the full glu libraries available since I was developing the editor in OSX rather than iPhone (there's actually iPhone versions of gluUnProject around the web if you care to find them) and this meant I could simply use gluUnProject rather than do the maths manually (although it's actually not the trickiest maths if you've got time).
Perhaps it was from working too late whilst fatigued but this coordinate conversion has managed to eat up a good few hours of time. It's very important to set up the correct OpenGL state before using gluUnProject if you want to get any sense out of it. I spent a lot of time searching around for examples for this function and whilst they are out there, they didn't take into account various things and often left users still stumbling to get it working for their own situations. So here is some commented code, highlighting the key things, which I hope helps anyone else who is finding screen to object space coordinate conversion using this function a nightmare.

// Enable depth test
glEnable(GL_DEPTH_TEST);
glDepthMask(1);
glDepthFunc( GL_LEQUAL );

glViewport(0, 0, backingWidth, backingHeight);
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
gluPerspective(35, backingWidth/backingHeight, 1.0, 1000.0);
 
//Push the gluLookAt projection matrix onto the stack
glMatrixMode(GL_PROJECTION);
glPushMatrix();
gluLookAt(m_fcamEyeX-65.0f, m_fcamEyeY-65.0f, 100.0,
   m_fcamEyeX, m_fcamEyeY, 0.0f,
   1.0f, 5.0f, 0.0f);

// Update and render the scene
Update();
Render();

if(mouseClicked)
{
   // Remember this must be done after rendering our scene as otherwise
   // there will be no polygons there to detect depth for!
   MouseToOpenGLCoords();
}

So next I check if the mouse was clicked that frame and if so convert it's coordinates to OpenGL world coordinates. We need to call glReadPixels to supply the zDepth of the polygon that was clicked on in the view. Clicking on empty space will result in a zDepth of 1.0 i.e. maximum depth of the view frustum.

void MouseToOpenGLCoords()
{
   GLdouble dMatrix[16];
   GLdouble dProjection[16];
   GLint iViewport[4];
   GLfloat m_fDepth;

   // We need to grab the projection and model view matrices along with the viewport
   // for the conversion
   glGetDoublev(GL_PROJECTION_MATRIX, dProjection);
   glGetDoublev(GL_MODELVIEW_MATRIX, dMatrix);
   glGetIntegerv(GL_VIEWPORT, iViewport);

   m_fWindow[0] = m_mousePoint.x;
   m_fWindow[1] = m_mousePoint.y;

   // Read the zDepth of the coordinate where the screen was clicked. Will simply return
   // 1.0 if you don't click on a polygon.
   glReadPixels(m_fWindow[0],m_fWindow[1],1,1,GL_DEPTH_COMPONENT,GL_FLOAT,&m_fDepth);

   // use gluUnProject to get the OpenGL world coordinates
   gluUnProject(m_fWindow[0], m_fWindow[1], m_fDepth,
  dMatrix, dProjection, iViewport,
  &m_dOpenGLCoord1[0], &m_dOpenGLCoord1[1], &m_dOpenGLCoord1[2]);
}

AAII Quadtree improvements

 

The Quadtree class has been made slightly more robust. Although Annihilation Arena II uses a 3D engine, it's really only 2.5D as it uses a fixed isometric view point. This changes only for cinematic NIS's on occasion but for the most part a quad tree is ideal for visibility culling as it's been relatively simple to implement and works fine within the 2.5D view point.

A Quadtree is a tree data structure with four children. They are useful for a number of things but AAII uses them to calculate which static level geometry is in screen space. To see if an object is visible we test which quad at the top of the tree it is contained in and iterate down. This means we quickly eliminate testing in large areas such as the upper-right quad, in-fact, we can probably eliminate testing on three thirds of the entire level area in the first iteration.

The Quadtree of a small level in Annihilation Arena II

 

New website up and functioning!

The JDTec website now has a brand new look and gives information on all of the JDTec products. We thought we'd keep the Main page as a blog so that we can give up to date information on bug fixes and changes to the products.

We will also periodically be posting updates on development progress for upcoming projects mixed in with some indie game dev tips in general.

All feedback to do with the website or any of the products available should be sent to info@jdtec.co.uk.