elementFromPoint(x,y) was defined to return the element at the given coordinates within the view port (the visible area) of the web page, or null (if the coordinates are outside of the viewport). The coordinates are measured relative to the origin of the view port. This is how iOS 5 finally works.

Before (iOS 4.x and older), elementFromPoint(x,y) completely ignored the view port. It measured the coordinates relative to the origin of the document. And even elements outside of the visible area could be found. This behavior seems to make much more sense than the new iOS 5 behavior, but according to the official JavaScript specification, it’s not the correct behavior.

The different behavior between iOS 5 and older iOS versions can cause some serious problems. The coordinate systems are no longer compatible, so when the web page or App has to run under old an new iOS versions, it is necessary to find out the correct coordinate system that is used.

In a native iOS App you could simply check the iOS version, and based on its value you can decide which coordinate system you need to use when calling elementFromPoint(x,y). But when writing a web page, this is not so easy: the iOS version is not exposed to the web page (it might be part of the UserAgent information, but because almost all browser do allow to use a fake userAgent information, this information is not reliably at all). Also on the Mac and on other platforms different WebKit releases might be used which do use different coordinate systems for the elementFromPoint(x,y) call as well. Therefore it makes sense to find a way to identify the coordinate system independent of the iOS version, and if necessary correct the coordinates.

At first when we compare the two coordinate systems, we notice that the coordinates are offset by the scroll location. If we scroll the web page so that the top left corner of the page is visible, both coordinate systems are the same. The origin of the view port is identical to the origin of the web page. And the scroll offset is also 0 in both directions. If you scroll down 100 px, the origin (the coordinate (0,0)) of the viewport is located at the coordinate (0,100) of the web page. So the scroll offset is exactly the offset between the two coordinate systems. Therefore, transforming one coordinate system into the other is very easy. We only need to add or subtract the actual scroll offsets.

function documentCoordinateToViewportCoordinate(x,y) {
  var coord = new Object();
  coord.x = x - window.pageXOffset;
  coord.y = y - window.pageYOffset;
  return coord;
}

function viewportCoordinateToDocumentCoordinate(x,y) {
  var coord = new Object();
  coord.x = x + window.pageXOffset;
  coord.y = y + window.pageYOffset;
  return coord;
}

These JavaScript functions take a coordinate of one system and transform them into a coordinate of the other system.

But in order find out if and which of the functions we need to use, we have to find out, in which coordinate system the call elementFromPoint(x,y) expects the coordinates. To do this we use the fact that elementFromPoint() returns null when the coordinates are outside of the view port, when it expects coordinates measured relative to the viewport (as noted above, when the coordinates are relative to the origin of the document, elementFromPoint() will always return an element, even when outside of the visible area, so we can distinguish between the two cases).
Good test coordinates would be (0, window.pageYOffset + window.innerHeight -1) and (window.pageXOffset + window.innerWidth -1, 0), for vertical scrolling and horizontal scrolling. As noted above, when no scrolling is done, both coordinate systems are identical, and we don’t need to take care about anything. But if the page is scrolled, we need to check which system is used. The test coordinates take the actual scroll offset and add the width or height of the visible area (this is the innerWidth and innerHeight of the “window” object) and subtract 1. This makes sure that the coordinate addresses the very last pixel line or column of the visible area measured relative to the document origin. This is always a valid document-based coordinate which lies within the document boundaries (a coordinate outside of the document boundaries would return null even with the elementFromPoint() call for the document-based coordinate system). If the page is scrolled by at least one single pixel, the test coordinates would lie outside of the viewport, when interpreted as relative to the viewport, so elementFromPoint() would return null. When elementFromPoint() would interpret them relative to the document, these coordinates are always valid and would always return an element. And this is how we can easily detect, which coordinate system elementFromPoint() is using.

function elementFromPointIsUsingViewPortCoordinates() {
  if (window.pageYOffset > 0) {     // page scrolled down
    return (window.document.elementFromPoint(0, window.pageYOffset + window.innerHeight -1) == null);
  } else if (window.pageXOffset > 0) {   // page scrolled to the right
    return (window.document.elementFromPoint(window.pageXOffset + window.innerWidth -1, 0) == null);
  }
  return false; // no scrolling, don't care
}

We can combine this to one custom elementFromPoint() function that is using a document-based coordinate system as input and will internally do all the magic for us:

function elementFromDocumentPoint(x,y) {
  if (elementFromPointIsUsingViewPortCoordinates()) {
    var coord = documentCoordinateToViewportCoordinate(x,y);
    return window.document.elementFromPoint(coord.x,coord.y);
  } else {
    return window.document.elementFromPoint(x,y);
  }
}

And the counterpart for viewport-based coordinates:

function elementFromViewportPoint(x,y) {
  if (elementFromPointIsUsingViewPortCoordinates()) {
    return window.document.elementFromPoint(x,y);
  } else {
    var coord = viewportCoordinateToDocumentCoordinate(x,y);
    return window.document.elementFromPoint(coord.x,coord.y);
  }
}

So instead of using elementFromPoint() directly, you simply use elementFromViewportPoint() or elementFromDocumentPoint() instead, depending of the coordinates you have to deal with. It will then work correct in old and new WebKit releases.

Please note: if you use the code of my older blog post “Customize the contextual menu of UIWebView” in your projects, you need to update this as well, because it also uses the elementFromPoint() call. But this should be really easy to do.

In general, when you have to use JavaScript code to do certain tasks on a web page, you need to inject this code into the web page when it is loaded. The UIWebView provides a delegate method “webViewDidFinishLoad:” which is called whenever a web page has finished loading. This is the correct location to inject your JavaScript code into the web page. And to inject JavaScript code into a web page you can use the method “stringByEvaluatingJavaScriptFromString:” of the UIWebView class.

In most cases you can just use the view controller which manages your UIWebView object as delegate for the UIWebView. When defining the view hierarchy and the view controller in Interface Builder, you can set the delegate for the UIWebView directly within Interface Builder, so you even don’t need to write any code for this. But you can also set the delegate in code, for example in the “viewDidLoad” method of the view controller. And in addition, the delegate (the view controller) has to implement the “webViewDidFinishLoad:” method, in order to get notified when the web page has finished loading. Please note that the “loadRequest:” method of UIWebView loads the page asynchronously, so when this method returns you can not assume that anything from the page has already loaded. Which means it is essential that you wait until the method “webViewDidFinishLoad:” is called.

The following example shows how this can be done in a view controller implementation (this example assumes that the JavaScript code is stored in a file “JavaScriptFile.js” within the resources of your App):

- (void)viewDidLoad
{
  // If not done in the InterfaceBuilder, you need to set the delegate for the UIWebView object
  // so that you're notified when the page has loaded (via "webViewDidFinishLoad:")
  [webView setDelegate:self];

  // ...
}

- (void)webViewDidFinishLoad:(UIWebView *)webView
{
  // Loading and injecting your JavaScript code into the webView, so you can actually use it
  // in the context of the web page that was loaded...
  NSString *path = [[NSBundle mainBundle] pathForResource:@"JavaScriptFile" ofType:@"js"];
  NSString *jsCode = [NSString stringWithContentsOfFile:path encoding:NSUTF8StringEncoding error:nil];
  [webView stringByEvaluatingJavaScriptFromString:jsCode];

  // ...
}

This is more or less everything you need to care about when programming JavaScript helper functions and injecting them into the web view objects. You can use the method “stringByEvaluatingJavaScriptFromString:” to call your helper functions afterwards and getting the results from your JavaScript functions as Cocoa string.

But there’s another detail you need to care about which is related to XCode itself. And this seems to be something many users are not aware of and what can cause your App to fail.

When building an App, XCode is doing this is several phases, like compiling the source code files, linking the object files and frameworks and copying all resources into the App bundle. The problem here is that XCode recognizes a JavaScript file as “Source Code” and therefore adds this file into the “Compile Sources” phase. But this is not correct in our case because the JavaScript file can not be compiled and linked to the object code of our App. The JavaScript file will be executed in the context of a web page within the UIWebView, so it’s not part of the “native” executable code of the App. The JavaScript code must be treated as “Resource file” and added as Resource file into the App bundle. Then we can load the file as resource within the App and inject its content to the web view.

XCode lets you inspect and edit the build phases of your project. Usually you don’t need to care about this because XCode is doing everything correct automatically. But not when adding JavaScript files. So you have to make sure that all JavaScript files which are listed in the “Compile Source” phase are moved to the “Copy Bundle Resources” phase. If you forget this, the JavaScript file won’t be copied to the resources of the App and when the App is running, it won’t be able to find and load the JavaScript files. So, always make sure that the JavaScript files are listed in the correct build phase.

The following screenshots will show you, where you find the “build phase” settings in your XCode project. The red circles indicate where you need to click to open lists or tables to find the settings.

This shows the location in XCode 3.x:

And this shows the location in XCode 4.x:

Scaling UIImages

Scaling images is important for example when you want to create thumbnail images which are used as an overview for real documents or larger pictures. The standard UIImage class doesn’t provide a method to scale images, but scaling an image isn’t very complicated. You can do it with just 4 lines of code…

UIImage *originalImage = ...;
CGSize destinationSize = ...;

UIGraphicsBeginImageContext(destinationSize);
[originalImage drawInRect:CGRectMake(0,0,destinationSize.width,destinationSize.height)];
UIImage *newImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();

At first we have to create and start an image context with the size of the resulting image. All drawing operations which take place after the UIGraphicsBeginImageContext() call will be done in the newly created image context. This means, we are no longer drawing on the screen. All we need to do now is to draw the original image in the size of the final image. The drawing takes place in the newly created image context. Afterwards we can use UIGraphicsGetImageFromCurrentImageContext() to get an image from the image context. Finally we need to call UIGraphicsEndImageContext() to make sure that the system will release the image context again, because we don’t need it anymore. And of course the drawing will take place on the screen again after calling this function.

Creating an UIImage from a UIView

If you need an image from the content of a UIView object (like a whole web page that is displayed in a UIWebView object) or even a complex hierarchy of UIView objects so you can save it as PNG or JPEG image or use it as thumbnail image, you can do this also with just a few lines of code…

UIView *view = ...;

CGSize size = [view bounds].size;
UIGraphicsBeginImageContext(size);
[[view layer] renderInContext:UIGraphicsGetCurrentContext()];
UIImage *newImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();

The code is very similar to the code from above to scale the image. But instead of drawing an image into the new image context, we use the “renderInContext:” method to draw the content of the view (which includes its subviews) into the image context. The resulting image will have the same size as the original UIView.

Retina Display

With the new iPhone 4 and the 4th generation iPod Touch, the iOS does also support high resolution screens. But the larger screen resolution is not used to display more content, instead it is used to display a sharper image with more details. Basically the iOS uses the same coordinate system on all screens with a size of 480×320 points. The only difference of the retinal displays is that there’re additional pixels between these points. Most drawing functions of the iOS automatically use these additional pixels to draw sharper and more detailed content. But there are exceptions when the iOS can not create more details itself. For example for bitmap images (UIImage objects), there’s no way to automatically generate a more detailed image for the retina displays. So iOS 4.0 introduces a new “scale” property for images. This property can be 1.0 (the default value) which is used for standard resolution and 2.0 for the more detailed retina resolution with twice as much details. If the “scale” property has a value of 1.0, one pixel of the image will be drawn on 2*2 (= 4 pixels) on the retina display, if the “scale” property has a value of 2, one image pixel is drawn on one screen pixel.

The above code will work with all iOS releases (including iOS 3.x which is the latest version that can be used on the first generation iPhone and iPod Touch devices), but it will always use the default “scale” property of 1.0 for images. Which means the code can not improve the image quality on the new retina displays. In order to get more detailed images on retina displays, we need to use “UIGraphicsBeginImageContextWithOptions()” instead of “UIGraphicsBeginImageContext()”, so we can define the “scale” property for the new image context as well. But “UIGraphicsBeginImageContextWithOptions()” is only available since iOS 4, so we have to check if we can use this function. If it is not available or if the device does not have a retina display, we still need to use the default scaling of 1.0 …

if ([[UIScreen mainScreen] respondsToSelector:@selector(scale)]) {
     if ([[UIScreen mainScreen] scale] == 2.0) {
        UIGraphicsBeginImageContextWithOptions(size, YES, 2.0);
     } else {
        UIGraphicsBeginImageContext(size);
    }
} else {
    UIGraphicsBeginImageContext(size);
}

Combining everything into a Objective-C Category

Now it’s time to combine all of the above and create a category for the UIImage class. I’m implementing three simple methods to create a new UIImage from a UIView or UIImage object.

• “imageWithImage:scaledToSize:” creates an image from a UIImage and scales it to a given size
• “imageFromView:” creates an image from a UIView
• “imageFromView:” creates an image from a UIView and scales it to a given size.

MyImage.h:

@interface UIImage (MyImage)
+ (UIImage*)imageFromView:(UIView*)view;
+ (UIImage*)imageFromView:(UIView*)view scaledToSize:(CGSize)newSize;
+ (UIImage*)imageWithImage:(UIImage*)image scaledToSize:(CGSize)newSize;
@end

MyImage.m:

#import "MyImage.h"

@implementation UIImage (MyImage)

+ (void)beginImageContextWithSize:(CGSize)size
{
    if ([[UIScreen mainScreen] respondsToSelector:@selector(scale)]) {
        if ([[UIScreen mainScreen] scale] == 2.0) {
            UIGraphicsBeginImageContextWithOptions(size, YES, 2.0);
        } else {
            UIGraphicsBeginImageContext(size);
        }
    } else {
        UIGraphicsBeginImageContext(size);
    }
}

+ (void)endImageContext
{
    UIGraphicsEndImageContext();
}

+ (UIImage*)imageFromView:(UIView*)view
{
    [self beginImageContextWithSize:[view bounds].size];
    BOOL hidden = [view isHidden];
    [view setHidden:NO];
    [[view layer] renderInContext:UIGraphicsGetCurrentContext()];
    UIImage *image = UIGraphicsGetImageFromCurrentImageContext();
    [self endImageContext];
    [view setHidden:hidden];
    return image;
}

+ (UIImage*)imageFromView:(UIView*)view scaledToSize:(CGSize)newSize
{
    UIImage *image = [self imageFromView:view];
    if ([view bounds].size.width != newSize.width ||
            [view bounds].size.height != newSize.height) {
        image = [self imageWithImage:image scaledToSize:newSize];
    }
    return image;
}

+ (UIImage*)imageWithImage:(UIImage*)image scaledToSize:(CGSize)newSize
{
    [self beginImageContextWithSize:newSize];
    [image drawInRect:CGRectMake(0,0,newSize.width,newSize.height)];
    UIImage *newImage = UIGraphicsGetImageFromCurrentImageContext();
    [self endImageContext];
    return newImage;
}
@end

Optimizations

Scaling images can temporarily require lots of additional memory, depending of the size of the images. The original image is first drawn in an image context. This image context must provide a bitmap buffer internally in which the drawing takes place. This buffer needs some memory. Also when the final image is created from this image context, some more memory is required. Only afterwards it is possible to free up the memory again by releasing the image context and maybe the original image.

For retina displays, the bitmap buffers for images where the “scale” property has a value of 2.0 are four times larger as for non-retina displays (scale = 1.0). So a single full screen image requires approximately 2.5 MB of memory. And because multiple bitmap buffers are required while scaling, the memory requirements can be much higher than without retina display.

This is not a big deal for the iPhone 4 which has much more main memory (512 MB) than all other devices. But for the 4th generation iPod Touch, the increased memory usage for the image buffers for the retina display is a problem. The 4th generation iPod Touch only has 256 MB of main memory, so it’s almost always running on its memory limits.

So to reduce the memory usage when creating a small thumbnail image from a UIView (which means you’re only interested in a scaled-down(!) image from a UIView), you should first create an image from the UIView with a “scale” property of 1.0. This will reduce the memory usage for the newly created image from the view. And because you’re about to scale down this image, it doesn’t matter that it doesn’t have all the details it would have with scale=2.0. When you then scale down this image using a “scale” property of 2.0, the final (smaller) image can still can get all the details it needs from the image you’ve created before.

If you have to create many images form large UIViews, you may consider to try to reduce the memory requirements this way (especially for iPod Touch with its limited amount of RAM). If you only need to deal with a few smaller views, you probably don’t need to care about this.

When developing Apps for the iPhone and iPod Touch, it’s likely that you’re using UITableView objects to display the content of your App. UITableView is one of the most important Views of the iPhone OS and used for all kinds of lists and data that can be provided as list. In most cases you would store your data in an NSArray or NSMutableArray object and each row of the table represents an item of this array. In case your app allows the user to reorder the table entries, you would have to implement the following delegate method of the UITableView class:

- (void)tableView:(UITableView *)tableView
         moveRowAtIndexPath:(NSIndexPath *)fromIndexPath
                toIndexPath:(NSIndexPath *)toIndexPath

This delegate method is called whenever the user has enabled the “edit” mode of the table and has moved a table entry to another location. Within this method, you have to reorder the items of your own data storage (the array) so that the new representation of the table matches the order of the items within your array again. But if you look at the methods of the NSMutableArray class, you won’t find a method to do this directly. There’s no method to move an object of the array from one index to another. But because this is a task which we need very often (almost all iPhone Apps do have tables and in many cases it makes sense that the users of our App should be able to reorder the table entries), we will add a new method to the NSMutableArray class which is doing this, so we don’t have to implement the same code over and over again.

Subclassing NSMutableArray to add the new method would be possible, but is not the way I would choose here. NSMutableArray is an abstract class and a subclass would not inherit the data storage of the parent class. This means when subclassing, we would have to create and maintain our own data storage and would also have to implement all of the basic methods to set and get array items. Many of the basic data storage classes of the Cocoa Frameworks are abstract classes, so be aware when you plan to subclass these classes. Remember that you have to provide your own data storage if you do so.

But Objective-C provides other ways to add methods to an existing class, so subclassing is not necessary. We can define a category for the new method. A category can be used to add new methods to existing classes without subclassing them. This has the benefit that these new methods are even available for objects which are created by code which we can’t control, like objects created and returned by the system frameworks. But the downside is that a category can not add any additional member variables or additional data storage to the class. The reason is simple, the memory that is occupied by an object of the base class must remain the same. The other frameworks and code which doesn’t know about the new methods are still allocating only the memory for the base class when they create a new object. So a category can only add new behavior (methods), but no new data (member variables).

For our task, a category is just what we need: we don’t need additional variables or data storage, we just need to implement a new behavior (method) to move an array object from one index to another index. We name this new method “moveObjectFromIndex:toIndex:”

The implementation of the category would look like this:

MoveArray.h:

@interface NSMutableArray (MoveArray)

- (void)moveObjectFromIndex:(NSUInteger)from toIndex:(NSUInteger)to;

@end

MoveArray.m:

#import "MoveArray.h"

@implementation NSMutableArray (MoveArray)

- (void)moveObjectFromIndex:(NSUInteger)from toIndex:(NSUInteger)to
{
    if (to != from) {
        id obj = [self objectAtIndex:from];
        [obj retain];
        [self removeObjectAtIndex:from];
        if (to >= [self count]) {
            [self addObject:obj];
        } else {
            [self insertObject:obj atIndex:to];
        }
        [obj release];
    }
}
@end

The code is very simple: we first remove the object from its original location and insert it at the new location or add it to the end of the array when the new location is beyond the array bounds. As you can see, though we do not subclass NSMutableArray, we can still use the “self” keyword just like we would have done this in a subclass. Also note when defining a category there’s no block where you can define member variables (within “{…}” like you can do this when subclassing), but instead you can only give the category a name (in this case “(MoveArray)”). This is because you can’t change the memory allocation for the class (see above).

Now, whenever we need to move an array object from one index to another (in a NSMutableArray object), we can simply call

[array moveObjectFromIndex:from toIndex:to];

As you can see, though we haven’t sub-classed NSMutableArray, the new method is called exactly in the same way as any other methods of the NSMutableArray class. This makes categories a very powerful and flexible feature.

For our initial UITableView example, implementing the UITableView delegate method “tableView:moveRowAtIndexPath:toIndexPath:” would be very simple now. In most cases we could just do the following:

- (void)tableView:(UITableView *)table
               moveRowAtIndexPath:(NSIndexPath *)sourceIndexPath
                      toIndexPath:(NSIndexPath *)destinationIndexPath
{
    [array moveObjectFromIndex:[sourceIndexPath row]
                       toIndex:[destinationIndexPath row]];
}

But with a “bookmarklet” (a special bookmark which is based on JavaScript code) you can do this as well. Here’s how the bookmark URL for the bookmarklet should look like:

    javascript:location.href='web'+location.href.substring(4);

Because you can’t create Bookmarks from scratch in Safari on the iPhone, you should just create a bookmark of a random page and change its URL to the above line and the title to “Open in iCab Mobile”.

If you open this bookmarklet from within the Safari bookmarks, the currently displayed web page is passed to iCab Mobile and opened there.

The bookmarklet simply changes the URL scheme from “http” to “web” and from “https” to “webs”. And because iCab Mobile supports the “web” and “webs” schemes as a replacement for “http” and “https”, the iPhone OS will pass these modified URLs to iCab Mobile.