Hacker News network of users (based on commenting behaviour)

There has been a lot of discussion lately on trust measures, reputation and groups on Hacker News. So, I decided to run some experiments on the crawled dataset of stories released by Xirium (http://news.ycombinator.com/item?id=182374).

Of course, the real database of voting history of users would have been much better, but this is all I could get...

Calculation

The trust values were calculated as follows:

1. For each story the submitter gets +1 from all commentors. I know this is naive, but bec. of lack of voting history (on the story), I had to go with this assumption.
2. Each commentor gets ( votes_on_that_comment / total_votes_on_all_comment ) for each comment on that story, from all other commentors. (Again I know this is naive).
3. Trust values are added up for (object, subject) pairs across all stories.
4. The votes on the story were recorded as votes from a virtual user 'HNCROWD' for the submitter. After adding up, the trust value from HNCROWD for a user reflects the 'Karma' of the user on the website.

The resulting file is downloadable in CSV format here (http://www.sendspace.com/file/mw59f7).

So with these values I tried running some experiments:

1. Clustering:
An interesting experiment would be see if there are clusters of users among commentors. I used the Markov Clustering Algorithm (http://micans.org/mcl/) for clustering graphs as it does not need the number of clusters as initial input.

Unsurprisingly enough, most of the Hacker News community belongs to a single cluster. This makes sense as Hacker News is quite a focussed community interested in practical hacking related to the web, entrepreneurship and startups.

Other explainations are that users who comment are themselves quite interested in the stories and the community and are hence closely connected and similar. The users who are dissappointed with the website, might not be commenting at all... Again, using voting statistics would have been better.

2. Trust-Rank:
Second, I tried applying a variation of the TrustRank algorithm ( http://www.vldb.org/conf/2004/RS15P3.PDF ) to the trust values data.

The result here was also unsurprising. The ordering of users was very similar to what is generated using Karma on Hacker News website.

Further work:
1. The method of calculating trust values (based on comments) is very basic and needs to improved (like taking into account threads and opposing opinions).
2. I want to see if this information is actually useful in tasks like News-Story Recommendation.

Conclusions:

Without the availability of voting data, it is hard to say if users on a focussed site like Hacker News have diverging interests. I am sure, as the community grows people of different interests are bound to join. But, the whole idea of a democratic voting site only allows stories that are interesting to the most active users to be selected. And so, other users will find the website boring, and not contribute and maybe leave. This might be an example of a community maintaining itself...

Giving highly trusted users down-modding power will strengthen this emergent behaviour, and the community will become more focussed (towards these users) than it is now. This might be both good and bad depending on if you are in this majority...

P.S. Thanks to Xirium for sharing the dataset.

FOWA: Paul Graham - Future of web startups

This is part of a series of posts with my notes on various talks I attended at the FOWA conference in London. Here are my notes from Paul Graham's talk on the future of web startups. You can read the whole essay here.

Future of Web Startups - Paul Graham (YCombinator)

Paul Graham of YCombinator postulated that Web Startups will also follow the pattern of any other technology that we have seen many times before. Initially, someone invents an innovative product/technology which is expensive. But soon after, someone figures out a way to do that cheaply and ideas and number of ways of doing it, explode.

The cost of starting a web startup has also decreased dramatically in the new web age, and this will have implications on what will happen in the future. His predictions were:

1. There will be a lot of them. As the cost is decreasing, the only threshold left is that, do you have the balls to work in a startup.
2. Standardization of things that are produced en-masse. As there will be many startups, there will be a standardization of the way, the deals are done.
• Standardization of Terms of business agreements. At the moment, every VC/Angel can come up with his own terms and deals. But, when the competition will increase, the funding packages and agreements will become standardized with set deals.
3. New attitude towards Acquisitions. Companies like Google have solved the stigma of acquisition and they use acquisitions to recruit smart people and ideas. They know this from their own experience.
4. More competition. As there are more startups, people will have to take more risk and do more crazy stuff do differentiate themselves.
5. Younger Nerdier Founders. More technically minded people who are generally wary of the business side of things, will be able to build new products. They can release good products on the web and get users first, before going to any investor.
6. Technology hubs like Silicon Valley are still needed, as they enable face-to-face meetings and easier visibility.
7. Also needed are Judges who can pick winners. VC firms and Acquirers both need people who are knowledgeable of the industry and startups. There will be more need for people who can pick the winning companies. Paul also postulates that companies will need to have a Chief Acquisition Officer, whose job is to find who to buy and then buys it.
8. College will change.
• The meaning of 'After College' will change from 'When you Graduate' to 'When you leave'.
• It will matter less and less, where you went to college. Peope from smaller colleges will have similar chances.
• Greatest value of college is who you meet there.
• Instead of studying to get grades, students will study to learn because thats the only way they will learn.
9. More wealth created. More competitors to satisfy users' needs (which are assumed to be unlimited).
10. Faster advances in technology.

I agree with most of the points that Paul Graham makes. Here is my opinion on what he predicts:

1. Location: Even though Tech hubs like SV are still needed, it has become much easier to start/seed new ones. All that is needed is a couple of big successes in an area, and that place will start teeming with entrepreneurs/investors/lawyers/etc. This is what Microsoft did in Seattle, and what is happening in cities like Bangalore and Hyderbad in India. But those first few companies will have a hell of a hard time, when they start...
2. Competition: I think as the number of startups in any space will increase, the key factors deciding success will be growth and responsiveness. Startups that can move fast, add new features quickly, and have shorter cycles will gain traction quicker. Also, startups will have to be more risk-taking, as PG pointed out.
   
3. Measures of Success: As the cost of starting companies decreases, the pressure to generate revenues will also decrease. So, the startups will focus less and less on making money, and more and more on growing the number of users. The measure of success now will change to popularity, growth and brand, rather than revenue.
   It is noteworthy that this is not the same as the Bubble in 2000, as the startups are not spending money as they did then (on huge marketing campaigns and hiring). The startups on the web will become more like charity businesses...