CCI Open Data Portal
CCI Open Data Portal
What do you need from an Open Data Portal, delivering Essential Climate Variables from satellite data? And though this approach works acceptably well for data that doesn`t change all that often, it is far from acceptable from data that does (or could) change more frequently. In fact, the closer updates to data get to realtime changes, the less optimal this approach is because it places a heavier burden on consumers (who must poll the API for data chances more frequently) and for the data portal itself (which must handle and respond to more frequent requests from API consumers). Other - more efficient - approaches to accessing data can be used when data updates occur more frequently.The Data Portal is designed to provide access to a variety of higher education data points concerning West Virginia`s public colleges and universities. These approaches - like server-sent events and Websockets (which are both part of the HTML5 specification), or registering a callback URL (or Webhook ) - benefit both the data consumer and the data producer. These values not only make imminent sense, they are enshrined in the guiding principles behind the open data movement. With this in mind, its a very good thing that we see governments updating their data sets frequently - particularly when these updates are driven by users who surface issues of accuracy or completeness that a government corrects with a subsequent data release.Datasets are organised around specific thematic data portals, if you are not sure which portal to use then start at the ICES Data portal, which gathers all the content for you. However, for all of the money being spent on open data platforms, none of them make it easy to track the revision history of a data set over time. There are cases where this can be one of the most important aspects of a data set - how has it been enhanced and updated over time. I realize I`m being picky, and I`m largely ignoring the fact that more and more governments are standing up open data portals.Please note that the available data provided by the database queries for the year 2009 and before only comprises information of member TSOs of the Continental Europe region (former UCTE). Diff`ing things is such a common practice in the world of technology that the complete absence of this functionality in commercial open data portals is pretty frustrating. More and more governments are using GitHub for their data , and Github makes it super easy to track the revision history of a file - that`s what GitHub does Hopefully open data vendors will take a cue from GitHub and build this functionality into their platforms.Download spatial, tabular, and metadata option for fast extraction of data and user friendly interface. But the amount of money spent by governments on open portals can be significant - these products should meet the needs of both governments and data users. There are a lot of instances, yet, where BI solutions could be pointing to the portal instead of backend databases, there is more training that could be done to make use of, for instance, OData portal to pipe into Excel (a common tool for any dogfooding). If some could seamlessly integrate or something higher up, it would fix the dog food problem and the power user taking the data and running. Chicago is definitely ahead of the curve in its utilization of its open data portal.
I`m not sure the last two are strictly the fault of open data portals - version control is a joke where I work and some of our government systems are so old or living in silos that people have built a career out of protecting that getting daily or weekly feeds takes an act from elected officals. Discoverability and usability is a key difference between a portal and flat-file methods.After partnering with the business enforcement agency, the list of eligible drivers are posted on the portal ( -Economic-Development/Public-Passenger-Vehicle-Licenses/tfm3-3j95 ). Now, the city and cab companies look at the portal to operationalize their business process. I have often found myself using the portal to access shapefiles, even though I have access to an ArcGIS server.Operational use: Chicago built and uses WindyGrid ( -windygrid-taking-situational-awareness-to-a-new-level-259 ), which combines multiple data sources. Often, for performance, it`s easier to reference the source DB. However, we do reference the portal when the portal is the most authoritative source. Despite these stories, I suspect I am more aware of how the public have used the portal than how city employees have used it.But beyond just the examples you`ve shared - which are awesome, particularly the taxi company example - my comment about dogfooding is meant to refer to the API access to data provided by open data portals. Anthea is discussing federal government, where my note in paragraph 2 is far less applicable. And you`re right, the dirty secret of data portals is that they`re primarily used as fancy FTP servers.This is one of the features that gets touted most often by companies selling portal software as a game changer (I`ve been on the receiving end of many of these), and yet almost no one is using these APIs for open data to power government apps - at least not that I can find. If you haven`t read Anthea Watson Strong`s post on dogfooding , I`d highly recommend it - I think it accurately captures where we are currently in the open data movement. Sometimes, the portal is the most authoritative source, especially when the system is managed offsite or by outside agencies.
But, I often think there are two fundamental lines of discussion around open data: national programs versus local (city/state) initiatives. I can definitely say that city staff use the data portal for Kansas City (). For example, the Office of Performance Management gets most of their data for analysis directly from the portal. We built it this way to make sure that the data is useful, accurate, and recent. The Open Data portal has made it much easier for staff to use and communicate data effectively and efficiently.The platform Open Data provides is extremely useful - and is something staff seems to rely on once they learn how to use it efficiently. For example, the ability to upload a spreadsheet and map it without having to use GIS software is a fantastic resource for analysts across the organization. KCStat and the Open Budget site are nice, but those are both products that are provided by Socrata that are tightly coupled to their open data portal software. I believe that Open Data vendors are inflating the need and value of their service. Why should we spend a lot of $$$ to dumping into a portal when it`s already available.The reason being that insiders can outline creative ways to expose or otherwise display data that would help others to see elements for the first time or from a different perspective. Such a portal would be valuable if subscribers could view solutions others have utilized and request deployment in their community or some modification. Lack of Support for Realtime Data: I`m hoping this will be pushed forward pretty quickly with more cities connecting up sensors to capture data.Get Out of the Way and Give Me My Data!: I agree and believe the primary users of a data portal are machines, the secondary users are custodians and the tertiary users are anyone else (which includes the casual hacker or two browsing for data they`ll connect to machines ;) ). Personally I`m looking for realtime data projects to push the boundaries of CKAN and hope to have some news on that soon.I think there will be some exciting applications for those working with realtime data and I think portal software such as CKAN is ‘ready` for the challenge. You complain that open data portals have useless bells and whistles (sorting, grouping, graphing, visualising) yet lack features (realtime support, revision history etc). IMHO (having worked with quite a few government bodies on open data stuff now), the most important thing government can be doing is releasing data. There are other groups (civic hackers etc) who can build nice features, but no one else can release that data. The idea for the Big Data Portal came from companies probably a lot like yours.