Processing Organization

Data analysis involves two major components, reading in and geolocating each of the N frames, then the creation of image products and the extraction of pixel time series for a number of designed pixel instruments. The latter step is like Argus sampling. Because each collect is different from the previous, this requires specification of a number of inputs, described in the next section.

The analysis uses a number of structures to simplify the variable space. These include:

• gcp - ground control points, described in Appendix A
• insts - pixel instruments, described in Appendix B
• inputs - described below
• meta - metadata, built during analysis and saved for the record in cx (appendix C)
• stack - temporary stack data structure.
• Images - collection of regular rectified Argus image products.

Geo-location requires solving the image geometry for each frame. This is too complicated to describe in depth here so the user is referred to texts like Hartley and Zisserman [2003] or a soon to be completed companion paper that will be included in the toolbox (“Surf Zone Characterization Using a Small Quadcopter: Technical Issues and Procedures”). But it is important to have at least a broad understanding of the process.

Solution requires knowing things about the physical camera and lens, called the intrinsic parameters, and also things about the installation, called the extrinsic parameters. The intrinsic parameters are found by lens calibration in the lab and are well handled by free toolboxes available on the web. We use the excellent Caltech Calibration Toolbox with those results implemented through a lens calibration profile (lcp) structure that is created by the m-file makeLCPP3.m. You will need to modify this routine with entries for your camera. There is a similar calibration toolbox in Matlab, if you have access to the Matlab Computer Vision Toolbox.

There are six extrinsic parameters, the x, y, and z locations of the camera and the three viewing angles, azimuth (taken here as the compass-like rotation clockwise from the positive y-axis), tilt (zero at nadir, rising to 90° at the horizon), and roll (rotation about the look direction, positive in the counter-clockwise direction as viewed from the camera). If you know all of these parameters accurately, you can geo-locate object in images. If you do not, you need to do a least squares solution for those you don’t know using ground control points (GCPs, points in the image that you know the world location of and can also locate by cursor in the image). Since each GCP located by clicking in the image provides 2 pieces of input (U and V coordinate), we must see enough GCPs so that twice that number exceeds the number of unknowns. Thus for six unknowns we require 4 GCPs (2*4 > 6). In fact, these points cannot also be collinear and they really should be spread across the image for a good solution. For terrestrial work this is usually possible, but for surf zone work we may see only one or two identifiable points on the shore, near the edge of the field of view. Thus we often must find alternate sources for these parameters so that we can reduce the number of degrees of freedom below 6 (and hopefully remove the non-collinear restriction).

It is rare to find sufficiently accurate information of the azimuth and tilt of an airborne camera so these variables almost always must be solved for. However the camera location is often available in the imagery, for example by using exiftool on a snapshot (usually taken just before a video). Latitude-longitude information on a Phantom 3 seems to be accurate to a few meters and is a good choice. Vertical position can be less accurate (for example, it is often expressed relative to the takeoff point, rather than in the ground coordinate system) but could be used if no better GCPs are available. Finally, it is reasonable to assume for a good stabilized gimbal as on the Phantom 3 that roll is stable and perhaps taken as equal to zero within a first approximation. Thus it is possible to reduce to as low as two unknowns which can be solved with just two GCPs anywhere on the image (in fact, the solution will be found with just one GCP, but not in a least squares sense).

The choice of which of the six unknowns is taken to be known is selected by the input variable inputs.knownFlags in which each element equals 1 for a known variable and 0 if that variable is to be found by least squares solution. Thus

knownFlags = [1 1 0 0 0 1]

would mean that the x and y camera locations and the roll would be considered fixed (supplied values in the input file are used exactly as supplied) while zCam, azimuth and tilt (variables 3, 4, 5) will be solved for (supplied values are used as seeds for the nonlinear search).

While clicking on GCPs in an image is a reasonable approach to solve for the geometry of one initial image, it would be exceedingly tedious to require this for every frame of a movie where the viewing angles vary slowly due to UAV drift. Instead, we would prefer to use control points that can be automatically found in each frame, for example something that is brighter than its surroundings. We call these reference points and let the user identify a number of these in the initialization process. This process is described below.

1. User Inputs

2. Initialization

3. First Frame Processing

4. Remaining Frame Analysis

5. Final Analysis