Wednesday/Thursday morning at the conference

I gave my presentation first thing in the morning (see the post below).

I attended the Geodatabase Road Ahead session- AnyGeo has a good, quick summary. While there's a lot going on under the hood, it seems like the SDE featureset is fairly stable. The expanded support in the network dataset for multiple shape types of blockages, increased cost areas, and time-dependent costs will be very useful; the next logical step is to have an implementation of multiple cost types (i.e., roads and tolls). The new 'child to parent' replication method will be extremely useful in simplifying federated database workflows (each county can directly push their work to a complete state dataset if they can handle most of the ETL actions through a schema cross-mapping). At the end of the session, I asked about 9.x version interactions- the parent will need to be 9.4, but the child won't (the state can establish the shiny new DB and the counties can transition as their budget allows).

In the afternoon, I took advantage of a couple of the non-session activities available. A couple of my co-workers are developing a new print map template at work, and I went to the Map Critique area with the current draft of the base map layers (the bread of the 'map sandwich'). I got a lot of feedback and had a good conversation with ESRI cartographic staff.

I also dropped by the Hands-On Training Lab and took a free 45-minute course (it was a topic I already had some user knowledge but not a thorough background in). The format was fantastic- a 30 minute video presentation (which I admittedly used the 'play faster' option to get through in 20) and then a set of exercises to give you the physical interaction. I hope they develop more of these in the future and I should look to using the format for my own in-house training.

I also got to try out the 9.4 Beta software- it looks pretty good. In particular, I was interested in the 'basemap group layer'- you can have multiple basemap layers in the map document (say the background in one and borders/annotation in another); they'll both draw continuously as you pan (the base map, which was fairly data heavy, took a second to render for areas where drawing hadn't occurred before). Also, when a layer is placed in a basemap group, it won't interact with the identify button- I actually like that behavior (too often, the top-most layer is relatively non-important boundary). I like the interface- it really does allow for a lot customization and hides everything you don't want.

This morning I caught the "Enterprise DB Tips & Tricks" session. I'm not the DB guy at work, but I work with him extensively and want to at least have a good understanding of the issues he has with maintaining the SDE (and occasionally give him a semi-informed suggestion). While I haven't personally dealt with the issues presented, the information did help tremendously- it put into overall system context the operations that I see at work.

My Talk: Extending Python Scripts

So, I gave a talk yesterday "Extending Python Scripts" (it's 5MB). I'm a big fan of using Python for geoprocessing (I'm naturally psyched about the expansion of Python capabilities coming up in 9.4).

The ability to make maps in 9.4 through Python highlights one of its current shortcomings- an analysis process must provide output that is readable to user. If you've designed the analysis, great; however, if you're deploying the process for someone else, the outputs need to be readily usable in multiple situations.

A similar situation occurs in ArcMap. One of the ways ESRI solves this is by including charting functionality and bundling Crystal Reports, an external helper tool.

In the same way, we can incorporate external libraries into our Python scripts to let us process the results of the analysis into a more usable form. This particularly useful when designing scripts for use in a server context; you are much less certain of what the user has available for tools to re-process the analysis.

In my talk, I covered 5 libraries that I've used in geoprocessing scripts:
-ReportLab, a PDF library
-xlwt, an Excel writing utility
-PIL, an image processing utility
-urllib, built-in library to grab stuff via the web
-smtplib, built-in library to send e-mails

Of these, I feel that the urllib is the most powerful. With it, you can grab data from ArcGIS Server REST sources as well as other websites. This allows you to do create mash-ups within the geoprocessing applications!

The other libraries all in some way deal with providing output. Excel and PDFs are more user friendly than shapefiles or dBase tables and more widely accepted in consumer client situations. smtplib provides the capability to e-mail the results, rather than have them displayed at the time of analysis; we've used this at public displays, where people can e-mail a report of their house to themselves while talking with our planners.

My example scripts are available at the Model and Script Tool Gallery.

ESRI UC Plenary Thoughts

The plenary can probably be best broken into 3 themes: what we're doing with GIS, what has been released, and what's coming in the future.

The award winners, highlighted by CenterPoint Energy and the State of Maryland, are clearly showing what we're doing. Both of these organizations are working with the existing tools, maybe not to the limits of what the technology can do, but to the fullest extent that they are able to. A lot of what their providing is online in one form or another; it's clear that while the desktop clients are the main feature-rich interface, ESRI fully encourages online ('cloud') access for consumer use and general sharing.

A lot of time was spent showing what 9.3.1 has provided. It's a recently released product; a lot of groups are naturally slow to transition to it. It provides a lot of benefits in speeding up web mapping; it's also being complemented by the expanded data serving capabilities of ArcGIS Online (and the searching abilities of the GeoPortal extension to server, which I'm surprised *didn't* make the presentation, unless I somehow missed it). So that message is clear- for now, get up to 9.3.1.

By the same token, the ArcGIS Explorer portion was a major presentation- the 900 release is presumed to be in a 'few weeks' and the Explorer client is being placed as the desktop consumer client when an application running on a local computer is needed. I haven't had a chance to work with it, but it would be interesting to see what advantages it can provide in a small-area, local jurisdiction context over web mapping or ArcGIS desktop (the demos exhibited where of a global nature).

The 9.4 presentation was the natural 'wow' factor. ESRI really is spending a lot of resources into both the Desktop and Server products; the Desktop client in particular is getting a much-neede overhaul (I'd argue that overhaul came for Server at 9.3 with the REST/JavaScript/Flex/Silverlight setup). I didn't attend Developer's Summit, but I'm given to understand that a lot of the features shown had been announced then; what we were seeing was in some ways a refinement from this spring. Still, they are promising a lot. The entire experience of finding, getting, viewing and using data will be much improved- one app handles everything (and the UI is designed to let you concentrate on the particular action at hand). You can also see the influence of the server development coming into the desktop client, as the base map/operational layers separation is implemented to improve draw times and the geoprocessing tasks go asynchronous from ArcMap (I'd hate to be an analyst who mistakenly tries to have two transformative tasks simultaneously run on the same data, like two calculate fields; I wonder how the locking mechanism is going to work). Another server/web paradigm coming into desktop is a gigantic emphasis on searching- searching for data, tools, etc. This should make it easier for the technical staff to support users as most 'where is my data' questions can start to get resolved into a search.

Editing is also getting improved through refinements and focusing. We can intelligently construct frameworks around editing to ensure that we put in the stuff that needs manual input and let the computer handle the rest; both on the Desktop and in the many Server configurations. Python is fully inheriting the AML role as a scripting language, gaining cartography tools and enhanced analysis capabilities. This probably means tweaking the existing work you do, but generally for the better- it will take some time to accommodate (it's not surprising that you can have 9.3.1 and 9.4 installed on the same machine- the switchover time will probably at least a couple of weeks for most people as they get used to 9.4).

Most of the time on server GIS focused on the Mobile environment and the geodatabase. As I said above, 9.3 placed web mapping on a new, solid foundation. The mobile clients look to mature at 9.4 (especially the Windows laptop/tablet client). The creation of an open API for the geodatabase is a welcome sight for anyone storing data in an ESRI format and needing to get it to work elsewhere. The 'vector/raster' divide in GIS/RS package capabilities is continuing to narrow; now that the functionality is mostly in place, I wonder how ESRI plans to market themselves in the RS world.

The final new area was the support for increasingly complex forms of data. 3-D features will be much easier to work with and do analysis on (the distance-dependent rendering quality will make it much easier to to deal with complex scenes), and the instant importing/locating of multi-patches (such as imported Sketchup models) will be a great help for anybody who deals with the architectural realm- Sketchup doesn't make it easy to deal with a coordinate system, so having an easy way to place those models is nice. I'm glad the time analysis features are getting upgraded. An area of growth in our data is making features time-aware, as new laws, alter what a particular piece of data means. For example, zoning text amendments, while not changing the actual zone, can alter what can be done significantly; being able to see the change in the difference of development potential over time based on legislative history would be invaluable. Having a time-based lookup table is the easiest way to store the criteria for this analysis. I was also personally surprised by the georeferenced video layers- that could be a very cool add-on in GIS presentations, if not a really interesting way to bring real-time into GIS (think about using this with the CCTV network in Britain- rather than monitoring a ton of screens, you implement a mosaic of the fields of view).

Certainly, the presentation left as many questions as answers. The big ones in my mind is, of course, the timetable for 9.4. Many of the features they demo'd looked fairly mature, but there were also features mentioned but not really shown- was it just too much for one presentation or are they not ready for prime time? Should we be expecting SP2 or 9.3.2 with some of the more mature features?