CMUG 09-16: Catonsville

Apologies for the late post; I was temporarily blocked by blogspot.

We'll be having our fall meeting this coming Wednesday (9/16) at CCBC-Catonsville in Building Q's theater lounge.

We've got 3 great presentations lined up:

• Jeff Cox, ESRI's Authorized Training Program in Anne Arundel County


• Jaimer Alveraz, Where Are They All Coming From?: Mapping the Community College of Baltimore County’s Student Population


• As always we'll have a technical demonstration; tentatively, it's planned to be on ESRI's new ArcGIS Explorer
  

The full agenda and driving directions can be found here.
See you there!

James & Sarah

ESRI UC: Thursday afternoon & Friday morning

Geoprocessing: The Road Ahead

Thursday afternoon featured one of the sessions I had been most interested in attending- Geoprocessing- the Road Ahead. From what was featured in the Plenary and the Desktop Road Ahead, it's been clear that a lot of effort has been placed in enhancing the geoprocessing analysis tools and making them easier to use.

The presentation expanded on the themes already described in other sessions:
1) A new user experience
2) ModelBuilder enhancements
3) Python incorporation
4) New tools

As seen in the Plenary and anytime you looked at 9.4, a new Geoprocessing menu is present, pre-loaded with some of the most commonly used tools. This menu is customizable, so you can place your own favorites in the menu. The tools will be findable under search and has had the search terms expanded; searching for 'cookie-cutter' will bring up the Clip tool. Taking another nod from the server environment, the Results window (you use that, right?) has been moved to its own, hideable window. Instead of a command-line window, a full featured python window is available.

ModelBuilder has gotten a few enhancements which will enable for more complexity in the models. Iteration types have been expanded, with built in support of 'all files in folder' and 'all feature classes in workspace' type for each loops. The model-specific tools (Select Layer..., etc.) can be found in a special menu in ModelBuilder (Model Tools)- making it easier to find these utilities (and simultaneously remove them from the interactive geoprocessing experience). The user interface is also improved; an undo/redo button set (including model layout changes) has been added and tool tips for parameters now appear when hovering over a tool. Particularly useful to me is the 'pop-up connection' feature, which will let you specify which parameter an input should be directed to when making the connection between them.

As has been announced, Python is the scripting library for ArcGIS. A lot of work has been done to make the 9.4 ArcGIS Python library more 'pythonic'- it's now going to be a module (arcpy), so we won't need to instantiate a geoprocessing object. There are three main packages: mapping, geoprocessing, algebra (as in map algebra). ArcGIS will also be able to expose external libraries (such as the r statistical package) as tools- so you can place your custom code as simple menu items. As part of making the arcpy module, intellisense support and pyDoc documentation has also been enhanced.

Some features of the mapping library were highlighted. Essentially, all properties of an MXD document will be exposed, and custom maps can be made via scripting. We saw PDF support- I'd like to know what other formats will be supported- all of the export options?

The new analysis tools were featured. Right now, there are 90 of them; the development team thinks it might be above 100 by the time 9.4 ships. A big new grouping of tools are 'Editing tools'- think some of the editing features (e.g., trim, extend) being applied en masse on an entire feature class or selection set.

A lot of tweaking has been done with the Spatial Analysis tools to improve performance and usability, and classic map algebra has been reintroduced. New is the concept of a 'fuzzy overlay'- instead of producing a determinate output, a probabilistic result is generated; this support set theory's AND, OR, and GAMMA (links) functions. Image analysis has also gotten easier with 9.4; in particular, it's easier to create a classification training set and analysis can be done on just the data in the view, making it easy to prototype the analysis.

Like Spatial Analyst, Geostatistical Analyst has also had the UI and defaults improved to make it more ready to use and provide meaningful results. Also, ecological sampling tools have been provided- including a tool to determine additional sampling locations based on kriging standard areas (this was developed by the EPA to improve analysis capabilities over the classic random points).

The 3-D features have been improved; aside from the new tools covered in the Plenary, all the 3-D analysis tools support terrains as well as TINs. Also, some tools to help improve native LiDAR processing have been included, such as being able to identify outliers and blunders in point data and remove points under polygons (water masks).

A lot of the improved functionality in Network Analyst were featured in the Geodatabase Road Ahead session. 3-D routing will also be available. Service Area and Closest Facility processing will be available via REST; this will make a lot of Flex applications more fun!

ArcGIS Explorer: Beyound the Basics

My office has played with ArcGIS Explorer (AGX) off and on over the past couple of years. With the AGX 900 release, it looks like it's time again. AGX is ESRI's desktop product that they're targeting at 'Geographic Information Users'- the people who need to browse GIS data but necessarily do complex analysis. The new features of AGX (Geopresentations/2D-3D/etc) were briefly mentioned (those would be the basics we're going beyond); this session covered how to customize and extend the application to make it fit your particular organization's needs. Some data setup tips (that are applicable in ArcGIS Desktop and KML export as well) were covered. First, in order to keep the download small, the AGX core software is missing a few things, like fonts and full projection support. These can be separately downloaded as expansion packs. Second is the reminder that you should spend some time on layer configuration- 3-D layers need to be authored in ArcGlobe, HTML pop-ups should be enabled and set how you want users to see their data.

All I can really say is that everything is extraordinary simple to set up and implement. You can build new base maps, implement custom splash screens, and customize the toolbars without any programming experience- the first can be done within Explorer, the latter two through an Application configuration manager that ships with it. If you want to develop your own tools, you can create 'add-ins' in .NET- the SDK has a wizard to start the process and take care of the more annoying programming overhead, so you can get right to writing the add-in's functionality. Add-ins will support Buttons, Dockable Windows, Galleries, and Extensions; they're bundled in .eaz files for easy transport and saved in a 'well-known' folder location (i.e., everybody has the same folder structure, so we know were to look).

Server Optimization

Optimization by it's nature is a 'tips and tricks' type of session, and it's better to look at the slide deck than rely on anybody's summary. One of the interesting things I took out of it is that we really are running a small setup (web/SOM/SOC box pulling in data from a separate SDE server). A lot of the cases shown involved optimizing multiple and even 8-server+ configurations.

Closing session

Ah, the end. If you've been at a closing session at and ESRI conference before (UC or Dev Summit), this should be familiar. The winners from the Map Gallery were announced (Walks in Portland, Irish names, San Luis Obispo all pretty prominent). Jack re-iterated the concept of GeoDesign. Through the questions raised at the conference and at the Q & A, a big theme emerged- Community, providing ways to enable GIS users of all levels to connect continuously and share what they do. It sounds like ESRI will be spending a lot of time in that in the future.

A new part of the Q&A process involved voted-on video questions. The top vote getter was what were ESRI's plans after 9.4. While there is still nearly a year to go before 9.4 is released, the big evolution that is being considered is moving to marketing ESRI (the company) as a platform or product. I'm honestly surprised they haven't already; at least in my area (DC), they do a fair amount of professional services, training, and other consulting.

Some improvements to the conference format were suggested. For those of you who found the rooms to warm, you'll be disappointed that someone thanked the staff for not freezing them this year (I concur, even if I found the rooms a little cooler than I like- I'm just used to DC's 90 degrees - 90% humidity summer climate). A way to pre-screen, recommend, or vote on paper presentations was also suggested; the paper sessions can sometimes be of variable quality. Likewise, better searching for the sessions would be useful; at the end of my paper presentation (a somewhat technical one on Python), I had someone approach me and ask why I wasn't in the program. It turns out he was looking through the Technical Workshops section- the ESRI provided 'our software and how to use it' sessions, instead of the Paper Presentations.

A couple of questions about ESRI as a company and how it positions itself in (and thus structures) the GIS industry. Jack's response on being a private company is classic- "running a private company is horrible but it's better than the other options". Jack sees the company as providing the services to to the GIS professionals (and some prosumers), who will then develop the maps and tools used by the consumers (rather than targeting the consumers of GIS data directly). ESRI is very excited to support NGS's Geomentoring program to place GIS into primary and secondary education, and encouraged us to volunteer.

This was a busy conference, and while I learned a lot, I also came away with some questions that I hope to explore a bit more thoroughly in the future.

Wednesday/Thursday morning at the conference

I gave my presentation first thing in the morning (see the post below).

I attended the Geodatabase Road Ahead session- AnyGeo has a good, quick summary. While there's a lot going on under the hood, it seems like the SDE featureset is fairly stable. The expanded support in the network dataset for multiple shape types of blockages, increased cost areas, and time-dependent costs will be very useful; the next logical step is to have an implementation of multiple cost types (i.e., roads and tolls). The new 'child to parent' replication method will be extremely useful in simplifying federated database workflows (each county can directly push their work to a complete state dataset if they can handle most of the ETL actions through a schema cross-mapping). At the end of the session, I asked about 9.x version interactions- the parent will need to be 9.4, but the child won't (the state can establish the shiny new DB and the counties can transition as their budget allows).

In the afternoon, I took advantage of a couple of the non-session activities available. A couple of my co-workers are developing a new print map template at work, and I went to the Map Critique area with the current draft of the base map layers (the bread of the 'map sandwich'). I got a lot of feedback and had a good conversation with ESRI cartographic staff.

I also dropped by the Hands-On Training Lab and took a free 45-minute course (it was a topic I already had some user knowledge but not a thorough background in). The format was fantastic- a 30 minute video presentation (which I admittedly used the 'play faster' option to get through in 20) and then a set of exercises to give you the physical interaction. I hope they develop more of these in the future and I should look to using the format for my own in-house training.

I also got to try out the 9.4 Beta software- it looks pretty good. In particular, I was interested in the 'basemap group layer'- you can have multiple basemap layers in the map document (say the background in one and borders/annotation in another); they'll both draw continuously as you pan (the base map, which was fairly data heavy, took a second to render for areas where drawing hadn't occurred before). Also, when a layer is placed in a basemap group, it won't interact with the identify button- I actually like that behavior (too often, the top-most layer is relatively non-important boundary). I like the interface- it really does allow for a lot customization and hides everything you don't want.

This morning I caught the "Enterprise DB Tips & Tricks" session. I'm not the DB guy at work, but I work with him extensively and want to at least have a good understanding of the issues he has with maintaining the SDE (and occasionally give him a semi-informed suggestion). While I haven't personally dealt with the issues presented, the information did help tremendously- it put into overall system context the operations that I see at work.

My Talk: Extending Python Scripts

So, I gave a talk yesterday "Extending Python Scripts" (it's 5MB). I'm a big fan of using Python for geoprocessing (I'm naturally psyched about the expansion of Python capabilities coming up in 9.4).

The ability to make maps in 9.4 through Python highlights one of its current shortcomings- an analysis process must provide output that is readable to user. If you've designed the analysis, great; however, if you're deploying the process for someone else, the outputs need to be readily usable in multiple situations.

A similar situation occurs in ArcMap. One of the ways ESRI solves this is by including charting functionality and bundling Crystal Reports, an external helper tool.

In the same way, we can incorporate external libraries into our Python scripts to let us process the results of the analysis into a more usable form. This particularly useful when designing scripts for use in a server context; you are much less certain of what the user has available for tools to re-process the analysis.

In my talk, I covered 5 libraries that I've used in geoprocessing scripts:
-ReportLab, a PDF library
-xlwt, an Excel writing utility
-PIL, an image processing utility
-urllib, built-in library to grab stuff via the web
-smtplib, built-in library to send e-mails

Of these, I feel that the urllib is the most powerful. With it, you can grab data from ArcGIS Server REST sources as well as other websites. This allows you to do create mash-ups within the geoprocessing applications!

The other libraries all in some way deal with providing output. Excel and PDFs are more user friendly than shapefiles or dBase tables and more widely accepted in consumer client situations. smtplib provides the capability to e-mail the results, rather than have them displayed at the time of analysis; we've used this at public displays, where people can e-mail a report of their house to themselves while talking with our planners.

My example scripts are available at the Model and Script Tool Gallery.

Re-blog: Other session reports I'm interested in

Road Ahead- Desktop/Server/Mobile/: http://geo.geek.nz/events/arcgis-desktop-9-4-arcgis-server-9-4-and-arcgis-mobile-9-4-the-road-ahead-at-the-esr-international-user-conference-2009/ (good quick overview)

Server Road Ahead: http://ambergis.wordpress.com/2009/07/15/esri-uc-arcgis-server-9-4/ (covers pretty much all I would cover)

Flex SIG: http://veryspatial.com/2009/07/15/esri-uc-day-2-roundup-couple-of-sessions/. It *was* a packed session. I wasn't thrilled with the 'programmers can't design' attitude- it's not my strong suit, but a lot of geographers do have at least some cartography experience.

Desktop: The Road Ahead

A lot of new technology gets demonstrated at the plenary very briefly. To compensate and complement this, there are a series of 'Road Ahead' sessions. Given the wide arrays of technologies shown, it's not surprising that there are several separate Road Ahead talks. I'm planning to catch the Desktop, Server, Geoprocessing and Geodatabase sessions. I saw the first two today (Unfortunately, the first Geoprocessing Road Ahead is the same time as my paper; it's a good thing there are 2 talks for each session).

The Desktop Road Ahead was the first session of the morning. The continuing theme is that 9.4 is an 'evolution' of the ArcGIS platform, not a major change. At the plenary, it was noted that there had been some debate as to whether to call this release 9.4 or 10 (indicating a significant change). If ArcGIS was still overwhelming a desktop product, the '10' argument would probably have won out; while ArcMap is still recognizably ArcMap, some serious changes have happened in the UI & under the hood.

The first thing they covered was the big question unanswered at the plenary- the timeline for 9.4. They're expecting a large, open Beta program this fall. I asked about eligibility at the Beta Island on the floor; it sounds like everyone who has an attendee at UC is going to be automatically enrolled in the Beta (I don't know if they have enough "bug buster" shirts for everyone, though). A pre-release candidate should be due early 2010, and 9.4 release in spring, with fully internationalized versions in 5-6 major languages (I missed the list, but the slide was showing Kanji) a few weeks later.

A key concept in the design of ArcMap was to preserve the current GIS workflows; unlike Office 2007, which fundamentally changed how things are done within that program. The menus and toolbars have been 'tidied up'; The Add Data entry on the file menu has grown to accommodate geocoding and x,y tables (though the button on the toolbar didn't seem to reflect this), the 'Tools' menu has been reduced to just the customization options and logically renamed and a Geoprocessing menu has been added. Geoprocessing tools are now accessible as buttons throughout the the UI, whether on a toolbar or in a menu item; this includes custom tools like models and script tools.

The entire look and feel should be very familiar if you've used Visual Studio, especially VS 2008. This is an interesting contrast to the use of the 'ribbon' interface in ArcGIS Explorer- it seems to underscore the notion of the expertise differential between the two program's users (everyone uses Office/AGX while not everyone uses VS/ArcMap). Anything that generates a persistent window can dock it's window inside the containers by dragging the window to the container and then selecting the location with positioning arrows. Panes can also be set to auto-hide, so that the map area is normally maximized. This includes a 'Catalog' window for browsing for data By default, ESRI wants us to work in an environment where the maps, data, other documents and perhaps even tools or other supporting GIS elements are stored together in one directory for the project- there are some browsing shortcuts that play off this notion. The catalog window in ArcMap doesn't create a separate lock on a gdb when accessing it, eliminating a common frustration of users- having ArcCatalog and ArcMap (separate applications) work on the same data is impossible. Also included is a 'Search' window that can be used to search for data (metadata or tags), maps and tools (styles/symbols also have a search interface, so you don't need to add them & then hunt for the right symbol). Another welcome addition is background geoprocessing, so you can start tasks and get back to your work (I asked: only one background GP process at a time).

Drawing performance has been significantly improved- I'm going to take a guess that they've moved over to the faster graphic engine that's responsible for the optimized map services in 9.3.1. That would explain the support for Maplex and Representations in the map services. A major new feature is the concept of a 'basemap' group layer- you place your background layers into a basemap group layer and it automatically caches the features and accelerates their drawing- the base map provides a continuous pan experience. I haven't gotten the full details on it, but I followed up with the staff member presenting it later- the basemap layers are pressed into images and feature querying is no longer possible, so I'm guessing that ArcMap is making a much more limited request from the geodb than a normal redraw takes (just enough to to process symbology, maybe- shape & categorizing data?). The display engine still needs some work, though- at several times in the presentation, a refresh had to be manually invoked.

A lot of tweaks have gone into working with features. Interactive selection has been beefed up, so that you can select by point, rectangle, or arbitrary area. The Selection tab is being replaced by 'Layers', which will make management of selectability and selection sets more obvious and better integrated with how the data is classified- in fact, I think the Layers tab will be the default tab for many people, rather than the TOC display. The editing environment now makes use of templates, again based on a classification set, which can control which fields are shown during editing and pre-fill values. These editing templates are applicable in both web and desktop contexts. Snapping is now a separate toolbar and more easily provides control and feedback on what features you're snapping to (and seems a lot quicker). Snapping is also available for all tools, so georeferencing scanned plats, for example, has just gotten a lot easier. The editing tools work in ArcScene in a 3-D environment and the editing tools are now unified for features, annotation and dimensioning.

One thing noted is that Mapping & Automation don't necessarily sound like they go together. If you've had to do a series of ~300 park maps you'd know otherwise. Thus the expansion of Python scripting into mapping and the major new feature in the Layout, data-driven pages. From talks after the presentation, it sounds like the data-driven pages is the tip of the iceberg that can be done with python. Essentially, data driven pages creates a series of maps based on a layer. Any layer- a uniform grid, points, arbitrary polygons. To get a uniform grid, there is now a "Create mapping grid" tool which creates a grid based on the extent/presence of a feature class. Just like any other multi-page output function, there are data-driven fields- the default ones are a little limited (relating only to environment, position within series, and the feature controlling the data), but can be expanded using python access to layout graphics. Python can also access properties of the map and layer documents; the big example is a bulk repointing of data locations (Find/replace *.shp -> SDE :) ). More will be in the Geoprocessing Road Ahead session, but it is absolutely clear in my mind that Python is the new AML.

They wrapped the talk with showing the RoadMap for developers/administrators. License management will be more flexible- license checkout will be supported, so you can log in first thing in the morning, check out a license, and hog it even if you don't have ArcMap. Win2K is not supported, Win7 will be. If you're doing .Net development, you'll need VS 2008 or 2010; VB 6 is dropped and VBA will be a separate install to start the withdrawal process.

From a development standpoint, this should be an easier transition- both 9.3.1 and 9.4 can be installed concurrently, so you can still use the 9.3.1 libraries. A *lot* of customization can be done through simply modifying the layer properties and ArcMap document, so some of the basic VBA tasks have disappeared. Python/Geoprocessing will be the environment of choice for analysis. For more complicated GUI modifications, "add-ons", written in .NET/Java but avoiding a lot of the registration overhead, can be developed and placed at a set location. I'm assuming ArcMap will load them, wrap them in a sandbox that handles application registration, and use them as if they were normal custom code. Unfortunately, I haven't seen an example of those yet.

All in all, I'm excited- it's a shame it's nearly a year out.

ESRI UC Plenary Thoughts

The plenary can probably be best broken into 3 themes: what we're doing with GIS, what has been released, and what's coming in the future.

The award winners, highlighted by CenterPoint Energy and the State of Maryland, are clearly showing what we're doing. Both of these organizations are working with the existing tools, maybe not to the limits of what the technology can do, but to the fullest extent that they are able to. A lot of what their providing is online in one form or another; it's clear that while the desktop clients are the main feature-rich interface, ESRI fully encourages online ('cloud') access for consumer use and general sharing.

A lot of time was spent showing what 9.3.1 has provided. It's a recently released product; a lot of groups are naturally slow to transition to it. It provides a lot of benefits in speeding up web mapping; it's also being complemented by the expanded data serving capabilities of ArcGIS Online (and the searching abilities of the GeoPortal extension to server, which I'm surprised *didn't* make the presentation, unless I somehow missed it). So that message is clear- for now, get up to 9.3.1.

By the same token, the ArcGIS Explorer portion was a major presentation- the 900 release is presumed to be in a 'few weeks' and the Explorer client is being placed as the desktop consumer client when an application running on a local computer is needed. I haven't had a chance to work with it, but it would be interesting to see what advantages it can provide in a small-area, local jurisdiction context over web mapping or ArcGIS desktop (the demos exhibited where of a global nature).

The 9.4 presentation was the natural 'wow' factor. ESRI really is spending a lot of resources into both the Desktop and Server products; the Desktop client in particular is getting a much-neede overhaul (I'd argue that overhaul came for Server at 9.3 with the REST/JavaScript/Flex/Silverlight setup). I didn't attend Developer's Summit, but I'm given to understand that a lot of the features shown had been announced then; what we were seeing was in some ways a refinement from this spring. Still, they are promising a lot. The entire experience of finding, getting, viewing and using data will be much improved- one app handles everything (and the UI is designed to let you concentrate on the particular action at hand). You can also see the influence of the server development coming into the desktop client, as the base map/operational layers separation is implemented to improve draw times and the geoprocessing tasks go asynchronous from ArcMap (I'd hate to be an analyst who mistakenly tries to have two transformative tasks simultaneously run on the same data, like two calculate fields; I wonder how the locking mechanism is going to work). Another server/web paradigm coming into desktop is a gigantic emphasis on searching- searching for data, tools, etc. This should make it easier for the technical staff to support users as most 'where is my data' questions can start to get resolved into a search.

Editing is also getting improved through refinements and focusing. We can intelligently construct frameworks around editing to ensure that we put in the stuff that needs manual input and let the computer handle the rest; both on the Desktop and in the many Server configurations. Python is fully inheriting the AML role as a scripting language, gaining cartography tools and enhanced analysis capabilities. This probably means tweaking the existing work you do, but generally for the better- it will take some time to accommodate (it's not surprising that you can have 9.3.1 and 9.4 installed on the same machine- the switchover time will probably at least a couple of weeks for most people as they get used to 9.4).

Most of the time on server GIS focused on the Mobile environment and the geodatabase. As I said above, 9.3 placed web mapping on a new, solid foundation. The mobile clients look to mature at 9.4 (especially the Windows laptop/tablet client). The creation of an open API for the geodatabase is a welcome sight for anyone storing data in an ESRI format and needing to get it to work elsewhere. The 'vector/raster' divide in GIS/RS package capabilities is continuing to narrow; now that the functionality is mostly in place, I wonder how ESRI plans to market themselves in the RS world.

The final new area was the support for increasingly complex forms of data. 3-D features will be much easier to work with and do analysis on (the distance-dependent rendering quality will make it much easier to to deal with complex scenes), and the instant importing/locating of multi-patches (such as imported Sketchup models) will be a great help for anybody who deals with the architectural realm- Sketchup doesn't make it easy to deal with a coordinate system, so having an easy way to place those models is nice. I'm glad the time analysis features are getting upgraded. An area of growth in our data is making features time-aware, as new laws, alter what a particular piece of data means. For example, zoning text amendments, while not changing the actual zone, can alter what can be done significantly; being able to see the change in the difference of development potential over time based on legislative history would be invaluable. Having a time-based lookup table is the easiest way to store the criteria for this analysis. I was also personally surprised by the georeferenced video layers- that could be a very cool add-on in GIS presentations, if not a really interesting way to bring real-time into GIS (think about using this with the CCTV network in Britain- rather than monitoring a ton of screens, you implement a mosaic of the fields of view).

Certainly, the presentation left as many questions as answers. The big ones in my mind is, of course, the timetable for 9.4. Many of the features they demo'd looked fairly mature, but there were also features mentioned but not really shown- was it just too much for one presentation or are they not ready for prime time? Should we be expecting SP2 or 9.3.2 with some of the more mature features?