Open Data Day Hong Kong: Event & Follow-up

Twenty people gathered at a good work room provided by Odd-e on Connaught Road West. Very nice of Odd-e to donate the space.  Four of us pitched 4 issues and we quickly formed groups.  Take a look at the hackpad created before, during and after the Open Data Day.

  1. List of vacant primary and secondary schools in Hong Kong- how many and how could the public more easily know what is going on with a vacant schools (3 people)
  2.  Cops / Bad Cops – reporting to the police / reporting by the police (3-4 people)
  3. Impact of Environmental Policy on actual air quality improvements (6-7 people)
  4. A crowdsourcing database of missing people in Hong Kong (4-5 people)
  5. Submitted after the day about pay-walls and how they enable misrepresentation and possible fraud.

Sammy Fung, the day’s organizer, was a floater during the day.  We are all very grateful for Sammy’s efforts to make the day possible.

We were well supplied with coffee, drinks, snacks and we worked throughout the day.  There was some exchanges between the groups on ‘how to do x or y’.  Generally though the conversations were within the groups until the end of the day presentations.  Good progress was made by 6pm.  Below are my observations and short summaries of what was accomplished during the day.  These are drafts.  I’ll very likely change them once I have feedback.  There are photos and other details on the Open Data Day Hong Kong 2016 Facebook page.

The theme connecting the 4 issues is ‘lack of trust’.  There was no plan or prior discussion on the issues to be used for the Open Data Day.  It’s an indication of how one group of HK citizens feel.  Who are these people?  Young, middle-age and frankly old.  Male and Female.   All have some sort of technical, statistical, analytical experience or inclination.  Twenty people’s views cannot be extrapolated out to 7 million.   How much trust do these 20 HK citizens have in the HK Government?   Not much is the answer.  Based on the tentative data gathered and analysed around the 4 issues, the HK public should listen carefully and ask questions when the HK Government claims it’s doing its best to be open, transparent and truthful.

(1)  Take a look at Public Accounts Committee of the 5th Legislative Council P.A.C. Report no. 65, part 8., chapter 3, page 135, clause 25.   The EDB admits it misrepresented to a Legco member’s question the total number of vacant schools.  The EDB responded there were 108 vacant schools when there were 234 vacant schools at that time, 2015-2016.   How many vacant schools are there and can we verify school vacancy without relying solely on data supplied by HK government departments?

Ans:  The Education Bureau’s has a website of ‘all schools’ with 11,077  and a dataset of ‘all schools’ with 3,507.  Why such a huge difference?   The answer is when is a school a school.  All 11,077 are schools.  All 3,507 are schools.  The EDB’s definition of ‘school’ is both vague and precise.  The discrepancy is the bigger list includes all of the tutor, cram or learning centre schools and other special schools, whereas the shorter list is only kindergartens, primary, secondary.  The numbers never quite match up.  The shorter list provides longitude and latitude of ‘schools’ and these can be mapped into Google Maps.  Text analysis of the descriptions from Google Maps may reveal if this is currently a ‘school’.   Automating the process is the key challenge.  After a few cycles the number of vacant schools should become clearer.  We don’t know for sure but it’s likely many more than the 29 the Education Bureau reported on 17 February 2016.  More will follow …

(2) The HK Police release glowing praise letters they receive.  The HK Police seem less inclined to release complaining letters they receive.   However, some complaining letters are released.  By applying text analysis on the praising letters and on the complaining letters released to the HK public what does it reveal?

Ans. The team managed to  completely automate the collection and the analysis process.  This is impressive for a day’s work.  The interpretation is coming.  More will follow …

(3) The air quality has improved over the past 10 years we are told by various government groups and NGOs.  By collecting and analysing the daily air quality released to the public does it reveal statistically significant improvement in the air quality over a period of time?

Ans. The data was collected and analysed.  There is measurable change.  Is it statistically significant change or simply random?  More will follow …

(4) Hong Kong is a city with millions of people living and passing through every year.  Some people go missing.  Could using crowd-sourcing along with the HK Police missing persons site help family and friends find the missing people?

Ans. Mixing and matching data scraped from public websites was done.  The HK Police were claiming ‘personal privacy’ concerns when a member of this group enquired a few days before Open Data Day.  What will be the reaction to scraped data combined with details supplied by family and friends?  More will follow …

(5) David Webb, an investor activist, in Hong Kong provided, ‘Deception behind the Companies Registry paywall‘ for the Open Data Day.  David writes on his useful, free, open and transparent Webb-site Reports,

“On International Open Data Day, we reveal a network of knock-off companies using the CIBC, Credit Suisse and BNP brands, based in HK with subsidiaries in the UK and New Zealand. If those registries were not free and open, the deception would remain undiscovered. We call on HK Registrar Ada Chung to tear down this paywall.”

Hong Kong with massive government reserves collects small amounts of money from its citizens.  Why go to such trouble for unneeded revenue?

The Hong Kong Government’s Office of the Chief Information Officer hosts the Data 1 site, DATA.GOV.HK.  The webpages do look much better than they did a few years ago.  The datasets seem about the same but a careful comparison may reveal improvements.  I note there are 15 applications ‘showcased’ as examples of:

“creative web and mobile applications and solutions developed by the Government and community with DATA.GOV.HK datasets. These examples will demonstrate the potentials of the public sector information provided in digital formats.” (from Applications)

All are interesting applications of open source public data.  However, fifteen applications seems rather paltry.  Why can’t the OGCIO provide a more comprehensive list of the applications which uses these datasets?  Let’s hope for improvement by Open Data Day Hong Kong 2017.

Leave a Comment

Open Data Day Hong Kong 2016

March 5, Saturday, is Open Data Day around the world. There will be a hackathon from 10am until 7pm at Odd-e Hong Kong, 35-36 Connaught Road West, Hong Kong. Take a look at the Facebook page for the details. A maximum of 30 people can participate. As of the first of March, 8 people have registered. Open Data Day Hong Kong 2016

Register for the Open Data Day Hong Kong 2016 at Eventbrite

The Hong Kong government has a website with datasets available to the public.

“You can download, distribute, reproduce, re-use or hyperlink to DATA.GOV.HK datasets for commercial or non-commercial purposes free-of-charge.”

Data.gov.hk – maintained by the HK Government OGCIO

Leave a Comment

Storytelling to Capture Knowledge: Mobilising the Learning Organisation (or How Fanta was invented)

I wrote this piece for Singapore’s Information and KM Society (IKMS) quarterly news magazine, GLOBE, (Growth Leadership Organisation Business Education) 4th Quarter, 2014.  Madanmohan Rao was the editor.  I wish I had a quarter of the energy Madanmohan has but I don’t.  He has put together resources for knowledge management that I use on a regular basis.

GLOBE, iKMS Quarter 4 edition 2014, pages 50-53, “Storytelling to capture knowledge: Mobilising the learning organization”, Bill Proudfit.

The theme for the Globe quarterly was KM and the Learning Organisation.  The LO has been around for long enough to become accepted wisdom.  Of course, we all want to learn but…it’s so hard to learn…how to apply learning to make something truly new and different is even more difficult.

Fanta has been my favourite soft drink since I was a young boy growing up in the deep American south.  I wondered how could anyone figure out how to make this great tasting stuff.  Here is a story, maybe somewhat true, I’ve never been sure how much of it is and how much of it isn’t.   Its a story which I’ve used over and over to illustrate how innovation requires having your back-up-against the wall and disaster seems just about to be a certainty but then, miraculously, an event occurs, a hero arrives, a problem is solved and the solution (sometimes) leads to great success.

Leave a Comment

Edward Snowden | Letter from Hong Kong

Dear Edward,

A few days before you revealed the NSA’s hacking of telecommunications and Internet traffic around the world I was giving a presentation here in Hong Kong at the Open Data Hong Kong’s 3rd meetup on, ‘What is Open Data’. I started with a quote from Rufus Pollack, co-founder of the Open Knowledge Foundation saying in 2012, “Today we find ourselves in the midst of an open data revolution”. That revolution has ended up on Hong Kong’s doorstep with you fleeing here and stories now across the pages of most of the territory’s newspapers for the past several days.

The U.S. government is making decisions behind closed doors to manufacture for itself what is being done is legal but at the same time choosing to hide this manufactured legal truth from its own citizens. Personally, I am not that worried about my privacy. By choosing to use free internet based services such as Google Gmail, Google+, Facebook, WordPress and many others I grant the right for these services to know what I’m doing online in return for the free services they provide. I’ve long suspected that Internet service companies and various governments are monitoring most anything I’m doing online. However, if the US government monitors all electronic and telecommunication networks I do want to know that it is being done and with sufficient detail to understand the breadth and scope of the monitoring. I do not want to be lied to on an almost daily basis starting with President Obama and going down his chain of command.

The people who approve and are nominally in charge of these monitoring programmes likely do not have the technical expertise to understand how these programmes work. These programmes are not automatic, not done by anonymous machines and most simply are not magic. I suspect that in any 24-hour period these monitoring programmes only work at best around 75% to 85% effectiveness and it may be much less some of the time. You have been one of the thousands of people writing code, monitoring routines, and making hourly, daily, weekly and longer adjustments to a wildly complex group of systems that at any moment may stop working. You and others who do make these monitoring programmes work do not share the same philosophy as the people in charge and no amount of signing confidentiality agreements is going to make your change you philosophy. I want to thank-you for standing up and doing what is right at great personal risk. We are all better off that people like yourself, Daniel Ellsberg, Julian Assange, Bradley Manning and many others choose to stand-up and tell the truth.

I hope you got to see some of the Dragon Boat Festival, 龍船節, racing yesterday. Qu Yuan’s, 屈原, story is both sad and uplifting. The people so loved him they wanted to keep him safe. Hong Kong is a wonderful city and I know if will fight for openness and transparency and I hope it will keep you safe.

Many thanks,

Bill

Comments (2)

Data.One Analysis Summary & Report

As part of the Open Data HK Make.01 Hackathon I worked with a team on reviewing the HK Government’s Data.One site. We produced this report. The team’s effort was quite remarkable and I’ve described the hackathon in an earlier post on this blog. The report is made up of files posted on Google Drive and some public links. This very simple infograph may help you navigate around the main parts of the report. I recommend starting in the middle with Data.One Analysis Summary & Report.

Data.One Analysis Project – Report & Summary Graph
Data.One Analysis Summary & Report
UX/UI & Instruction/Education Parts 1 & 4
World Bank Assessment Tool Part 2 Content Relevance
PSI Datasets Parts 2 & 3 Content Relevance & Format of Files
Global PSI Data Catalogues
Data.One website

This report has also been made available to the HK Government’s OGCIO PSI (Office of the Chief Information Officer – Public Sector Information) team which maintains the Data.One website and Charles Mok, IT Sector, Legislative Councillor. This report is a jumping off point for further discussion on how to improve open data awareness and use in Hong Kong.

Summary
A report on the HK Government Data.One site was prepared as part of the hackathon organized by Open Data HK on 14, 18-19 May, 2013. The report targets user experience and interface, content relevance and usefulness for citizens, usability and format of the datasets made available and the instructions and education available for the general public and potential developers. The report gives examples from 19 public sector information open data websites around the word. The report makes recommendations on how to improve the Data.One site specifically as well as awareness and knowledge of open data in Hong Kong. An assessment tool from The World Bank was used to judge the completeness of the datasets being made available in Hong Kong. The formats of the datasets were evaluated and suggestions made on how to make them more useful for potential use by developers.

Leave a Comment

Open Data Hong Kong ~ Make.01 ~ Hackathon

I was a bit aprehensive about getting involved with anything called hackathon. For me and for a lot of others I suspect the word has connotations of electronically sneaking into an organizations computer system and stealing data. However, ‘hack‘ means something just done well as a verb and done playfully as a noun, at least according to that great resource of modern English usage the Urban Dictionary. So I went along on this past Saturday wondering if any of the people from the Catalyst night would be there and what were we going to get up to for the day. Would we do something well and playful?

I ran into someone at the Cheung Sha Wan MTR station and after loading up on tuna buns, soy milk and coffee heading out for The Good Lab. Arriving around 1pm and there were a few people in a large, bright and varied work space with kitchen, work-tables, work-benches, chairs of various shapes and sizes. It quickly filled up with about 40 people. I was involved in two projects. I found my fellow team-member, we got into the wifi and had a few conversations with people wandering around looking for possible projects. We then set to work. I was working on reviewing open data public sector information websites around the world, Data.One Analysis Project. My team-member was working on a form for crowd-sourcing potential datasets around the HK government websites, Opening Data. Most of the time people were heads-down working with some small group meetings. It was possible to eves-drop on some conversations. This was a good way of knowing what skills people had and maybe asking them a question. Around 6:30pm the group reported on progress and asked for help if required. Pizza was delivered and we ate and chatted. We kept working until 11pm.
Bill and Haggen So May 2013 Hackathon photo credit: Yolanda Jinxin Ma
Up around 8am on Sunday and made my way back to Cheung Sha Wan by 10am. Most of the same people were there plus some more. Yesterday’s team-member was joined by 2 others. We figured out what we needed to do and worked until lunchtime. There was a feeling of anxiety in the crowd. Downstairs for a good Chinese lunch and we talked about what was wrong with Hong Kong with a recent arriver from Spain. Back to work until a bit after 6pm and we started giving presentations on the results. There were some truly amazing results and knowledge sharing on how it was done. People were very interested in anything dealing with maps and how to use the not so friendly mapping CSV files available from the Data.One site. The list of projects is here. Hopefully, they will be updated in the coming weeks. Here are three that I believe deserve a special mention (but they were all really good):

Legco Meeting Log Parser ~ extract the Hong Kong Legislative Council meeting transcript and voting record from PDF and make it available. It begs the question why this isn’t made available in document format with audio and video transcripts.

Reporting Tool for Request for Access to Information ~ a centralized form with sharable tracking of requests for information to the appropriate HK government bureau or department. Hopefully this will motivate our government to have true Freedom of Information legislation in the coming years.

Hong Kong food security and mainland’s two standard on food quality ~ a way of putting on a map where food is coming from out of China into Hong Kong. Food security and safety is a huge issue in China and Hong Kong. The HK government should be sharing as much information as possible with where our food comes from and what are the past problems.

A member of the OGCIO PSI team and Charles Mok, IT Sector Legislative Councillor came over around 6pm. We had some Raspberry Pi prizes donated by Pindar Wong. The prize was chosen by popular acclamation choosing Legco Meeting Log Parser and Charles Mok gave out the prize. An interesting RTHK video of an interview with Pindar and Charles is here.

Did we do something well and was it playful? The work-products from our projects were excellent. The energy level was high. People were working really well and collaboratively and the atmosphere was a lot more playful then I’ve experienced in the dreaded corporate cubicle world a la Dilbert. So now I know what a hackathon is about.

Comments (1)

Open Data Hong Kong ~ Catalyst Night ~

Open Data Hong Kong is a group that was formed out of some talks at the Hong Kong Barcamp held at HK Polytechnic University this past February. A community of over a 100 people has formed quickly based first on a Google+ group and a couple of meetups with presentations and chatter on the 2nd floor of Delaney’s in Wan Chai and other gatherings around town. A Facebook page and the OpenDataHK website were setup recently. Establishing a dialogue between the users of open data in Hong Kong and the HK government is one of the goals for the group.

It’s impressive that there is so much interest in Open Data. What is it? The best resource I’ve found is the Open Data Handbook. You can listen to me go on about it here on a local public radio show recently here. The Open Data Hong Kong website has useful information on events and links to other sources on open data in and around Hong Kong that will keep growing. The Hong Kong government has had an open data initiative since 2011 called Data.One. The Hong Kong University Journalism & Media Study Centre ~ Data Journalism Lab ~ is a hotbed of activity on the data journalism side of open data in Hong Kong.

The Catalyst Night on 14 May is the kickoff for HKOpenDataMake.01, a hackathon event that will bring together developers, programmers, designers, thinkers and just the plain hangers-on to do and think about open data in Hong Kong. More than 50 people have signed up. The HK Government Office of the Chief Information Officer, Public Sector Information team will be attending the catalyst and talking about their plans for the Date.One initiative. At the Catalyst night the goal is to figure what to do over the next weekend. Teams will form and project goals will be set. Potential projects can be seen here. The teams will work, think and play around with datasets, tools and ideas and come up with results by Sunday afternoon. Presentations will be made on Sunday afternoon and Raspberry Pi prizes for the best results will be given out by Charles Mok, IT sector Legislative Council member.

Comments (1)

Older Posts »