Hacker News network of users (based on commenting behaviour)

There has been a lot of discussion lately on trust measures, reputation and groups on Hacker News. So, I decided to run some experiments on the crawled dataset of stories released by Xirium (http://news.ycombinator.com/item?id=182374).

Of course, the real database of voting history of users would have been much better, but this is all I could get...

Calculation

The trust values were calculated as follows:

1. For each story the submitter gets +1 from all commentors. I know this is naive, but bec. of lack of voting history (on the story), I had to go with this assumption.
2. Each commentor gets ( votes_on_that_comment / total_votes_on_all_comment ) for each comment on that story, from all other commentors. (Again I know this is naive).
3. Trust values are added up for (object, subject) pairs across all stories.
4. The votes on the story were recorded as votes from a virtual user 'HNCROWD' for the submitter. After adding up, the trust value from HNCROWD for a user reflects the 'Karma' of the user on the website.

The resulting file is downloadable in CSV format here (http://www.sendspace.com/file/mw59f7).

So with these values I tried running some experiments:

1. Clustering:
An interesting experiment would be see if there are clusters of users among commentors. I used the Markov Clustering Algorithm (http://micans.org/mcl/) for clustering graphs as it does not need the number of clusters as initial input.

Unsurprisingly enough, most of the Hacker News community belongs to a single cluster. This makes sense as Hacker News is quite a focussed community interested in practical hacking related to the web, entrepreneurship and startups.

Other explainations are that users who comment are themselves quite interested in the stories and the community and are hence closely connected and similar. The users who are dissappointed with the website, might not be commenting at all... Again, using voting statistics would have been better.

2. Trust-Rank:
Second, I tried applying a variation of the TrustRank algorithm ( http://www.vldb.org/conf/2004/RS15P3.PDF ) to the trust values data.

The result here was also unsurprising. The ordering of users was very similar to what is generated using Karma on Hacker News website.

Further work:
1. The method of calculating trust values (based on comments) is very basic and needs to improved (like taking into account threads and opposing opinions).
2. I want to see if this information is actually useful in tasks like News-Story Recommendation.

Conclusions:

Without the availability of voting data, it is hard to say if users on a focussed site like Hacker News have diverging interests. I am sure, as the community grows people of different interests are bound to join. But, the whole idea of a democratic voting site only allows stories that are interesting to the most active users to be selected. And so, other users will find the website boring, and not contribute and maybe leave. This might be an example of a community maintaining itself...

Giving highly trusted users down-modding power will strengthen this emergent behaviour, and the community will become more focussed (towards these users) than it is now. This might be both good and bad depending on if you are in this majority...

P.S. Thanks to Xirium for sharing the dataset.

Quick: email current page using Gmail

Quick post further to my previous post where I shared a quick bookmarklet that allows you to email the current page. Here is a link that allows you to email using your Gmail account:

Gmail »

Again, just drag and drop this link to the bookmarks toolbar in Firefox.

Enjoying History

Recently, I am beginning to get more and more interested in History. I have found that I can spend hours reading Wikipedia and other historical websites. I am even bringing books on history back home from the library (which is a big deal for me)...

But what is most fascinating to me, is that I used to dislike history in high school. I found it so boring back then, that I could hardly read two pages of my history textbook without feeling sleepy. It usually happened like this: I started to read the book on my study table, but started getting sleepy, so I moved to my bed so that I could read comfortably there. Sitting on the bed soon changed into leaning, then reclining, then reading while lying down, and finally putting down the book and actually sleeping. :-)

Back then, history was about dates and a bunch of numbers which didn't mean anything. We had to remember these dates along with some events of the past, that had nothing to do with us. What does the story of some random king, in an unknown country in a remote part of the world (that I will probably never visit in my life), have to do with me? Who cares if people fought with swords or bows and arrows, or if they thought fighting on horseback was the cutting edge in military technology... It had nothing to do with me...

What changed? Am I reading a different history now from what I was reading back then? Maybe. But mainly, I think what changed was the way I looked at History. And the seed was planted by something that my father said to me long ago.

We were chatting about political history once, and I remarked how same things happen again and again. I said the history of the world is like a cycle. For eg. when rulers abuse their Dictatorial or Authoritarian rule, it always leads to revolt by the people and some form of Democracy is adopted. This has happened in ancient Greece, Rome, France and America (during revolutions), and freedom movements in many former colonies like India. But soon, democracy gets corrupted, where criminals, the rich and powerful and those who know how to manipulate the common people, rule. They don't allow capable and honest people to rise. Democracy then becomes an impediment, rather than a savior of the people. When this happens, usually a person with will and ambition emerges and starts fighting against the status quo. He takes the power in his hands, and then passes his judgment onto the society that is struggling to save itself. Julius Caesar, Napoleon, Hitler and Mao Tse Tung and examples of such people. Their personal beliefs may be different, but they all represent the same will to gain power. After a while, this absolute power corrupts (as Lord Acton famously said) and again leads to Authoritarian rule that exploits people. So, is this a cycle that is deemed to repeat again and again?

My father asked me to take a closer look. He said that history is not exactly a cycle but a spiral. Things that happen in the past, affect those that happen in the future. Knowing past events affects the minds and the judgment of people making decisions in the present (even if to a small degree). So, even if flow of events looks similar on a large scale, they are actually different. So, to prevent the same things happening, we must understand history, and learn from it. But how do we understand history?

To truly understand history, and why people behaved the way they did, we have to get into the minds of these actors. What did these people see in the world around them? What were their choices? What was it that motivated them, and what was it that they lived for? What were the forces acting on them, and circumstances which they had to face?

When we start imagining history as something alive, as something that we can see and be part of, it suddenly becomes a whole lot exciting. We can be almost sure, that someone somewhere was faced with decisions similar to what we are faced with today. What did he/she choose, and why? What were the consequences? Were they in control of their destinies, or were they just pawns in the overall scheme of the world?

Imagine yourself in history, and history will tell you amazing things. It will tell you great stories of extraordinary people in ordinary circumstances, and also great stories of ordinary people in extraordinary circumstances. History is a chronicle of how people just like us lived, how they fought against a hostile world, how they molded their destinies, and how they died. Were they able to achieve anything? Were they happy in the end?

In the end, it is all so that we can answer our own questions... Will we be able to achieve anything? Will we be able to make the right decisions? How will others in the future look at our lives? What will our history be like?

Invention of Zero and Brahmagupta

I just found out that Zero was invented very close to my home town...

BrahmaGupta who was a great mathematician from Ujjain, an ancient city (est. 6th Century BC) about 40 miles from my hometown Indore (there was no Indore then). This is circa 600 AD.

Arabic scholars adopted Zero and its mathematics from texts written by Brahmagupta (mainly 'Brahma-sphuta-siddhanta' or the 'Corrected Treatise of Brahma'), and was later passed onto Europe through them. The Sanskrit word ‘Shunya’ meaning nothing, empty or void, became ‘Siphar’ (origin of the word Cipher) in Arabic, which became Italian ‘Zephiro’ and hence ‘Zero’ in French and English.

Brahmabupta provided all the rules for adding, subtracting, multiplying and dividing positive and negative numbers. But he made one mistake: he said 0/0=0, which is wrong. As there was no concept of ‘limits’ and differentiation at that time (at least for Brahmagupta), he did not consider that
lim(x->0) x/x = 1. Though even now, mathematicians are unsure about what the value of 0/0 should be. In modern day computer programming languages, it is refered to as NaN or ‘Not A Number’, and dividing by zero is prohibited.

On Astronomy, Brahmagupta argued that the Earth was round and not flat, a point on which he was ridiculed by many Islamic scholars who read his work later.


Aryabhatta

Before Brahmagupta, around 500 AD, another Indian mathematician and astronomer Aryabhatta had said that

"Sthanam sthanam dasa gunam"

or 'place to place in ten times in value'. This might actually be the origin of our modern decimal-based place value notation!

Aryabhatta also influenced the birth of Trigonometry, and he was the first to describe sine's and cosine's, and prepared tables for them.

As with 'zero', the words 'sine' and 'cosine' are derived from what Aryabhatta called them: 'jiya' and 'kojiya'. Arabic scholars called it 'jiba' and 'kojiba'. They were then misinterpreted by Gerard of Cremona while translating an Arabic geometry text to Latin; he took 'jiba' to be the Arabic word 'jaib', which means "fold in a garment", and translated it into L. sinus.

Following Aryabhatta, another mathematician from Ujjain, Varahamira worked on Trigonometry in-depth. He is attributed with first developing basic rules like:

• sin(x) = cos(/2 - x)
• sin²(x) + cos²(x) = 1

But, even though Mathematics and Science were quite advanced in India at the time, this progress suddenly ground to a halt after 10th and 11th Centuries. This may be because India was under constant threat of attack from the North and West from descendants of Mongol warlords, and Islamic invaders. With the exception of Akbar, most Muslim rulers of India, favored Persian and Arabic knowledge and culture over traditional ancient Indian knowledge. Hence, such astronomers and mathematicians lost the patronage they used to get during the Gupta kings in the first millenium.

The Gupta period (between 300AD and 600AD) is considered as the 'Golden Age' for Indian science and mathematics. Kings like the legendary Chandragupta Vikramaditya were great patrons of science and art.

Research has always been heavily dependent on state support. Even now, the universities that get most government research grants are also the best in the world. See the earlier post on how Stanford University and the Silicon Valley prospered because of the funding they got for defense projects.

The Wikipedia entry on the history of zero is a good read.

BBC Radio 4 also covered the history of Zero here.

Emailing the link of the current page

Remember the times when you want to send an interesting link or URL to a friend, but the only way is Copying the URL from the browser's toolbar and then Pasting it into your email application. It is frustratingly tedious...

So, here is a link that opens the default email app and copies the URL of the current page to the email. We only need to Drag and Drop this link to the browser toolbar, and then use this as a button:

Email »

We can then edit the email and send the message...

Edit: Apparently 'dragging and dropping' works in Firefox but not in IE. In IE7, the link can be added as a Favorite into the 'Links' folder. It then appears in the Links Toolbar. When clicking on the page it does change the page, but we can go back using the 'Back' link. I suddenly like FF even more... :)

Path of an entrepreneur

I have been pursuing the path of an entrepreneur for a year and a half now. I had a dream of building my own intelligent search engine. I left a well settled job in India to come all the way to Scotland to learn more about how to do it. I think that was the best decision I made in my life.

Since then I studied, graduated, pitched, got funding, built my idea into a prototype and got people to use it. I learnt a lot of things along the way. I learnt that starting a company is like swimming upstream in a river. You can't wait and rest. You keep struggling against tremendous odds, and try to hold on to a few opportunities and fragile hope that you get in between. But then when you reach a point when you can think about where you are and what you have done, there is a sense of contentment and pride in you. It doesn't matter if you succeeded or failed, those times are never forgotten...

I still have a lot to learn, and a long way to go.

Microsoft + Yahoo = Microwho?

Microsoft wants to buy Yahoo, and that too for no less that $44.6 Billion. Yahoo had been posting bad results for a while now. There is news that Yahoo is about to layoff anywhere between 1000 and 2500 employees (7% of their employees). Former CEO Terry Semel who vacated his post for Jerry Yang, is also leaving his position as the Chairman of the Board.

What a day for Yahoo!!

Microsoft aims at gaining ground against Google by getting Yahoo into its fold. They have been trying to build MSN Live Search engine as a rival to Google, but with little success. On the surface, it looks like the MS/Yahoo merger might be beneficial to both of them. But that might not be so...

Microsoft's MSN and Yahoo are actually competitors in most of their application markets. Be it finance (Yahoo Finance vs MSN Finance/MSNBC/CNBC), email (Hotmail vs Yahoo Mail), web search (Live search vs Yahoo search), news, online games, maps or yellow pages, Yahoo and Microsoft are competitors.

So what will happen if they actually merge with each other. One of their two services will have to be let go. This is bad for competition and innovation. Also, many Yahoos have grown up hating anything to do with Microsoft. Some of these key people will leave Yahoo if a merger happens. And where will they go? Probably Google...

I think, if Microsoft really wants to compete with Google, they first need to shed their corporate (and bureaucratic) structure (atleast for their search division), and need to become more like a startup. Google has done very well by keeping the spirit of small teams and allowing individual innovation to prosper. I think Microsoft should also adopt the 20% time for personal projects idea (or something similar) that Google pioneered.

Instead of spending $45 billion in acquiring Yahoo they could use that money to foster a startup innovation culture within their own home.

Thanks to fauigerzigerk for suggesting the title in the YC News thread.