Question

Provide a specific example of a large dataset, and how it can be used. What are some of the challenges of working with large datasets, and how you think you can overcome these challenges?

Answer 1

A large dataset example is : The year end market capitalization of all the companies listed on Bombay Stock Exchange. This dataset will comprise of ~3000 (number of companies) * 25 (Years of data available) observations.

In my current career, I have used it to look at size-specific investment returns of the companies listed in Emerging Market like India. I have looked at how the concentration of largest firms relative to the total market capitalization of Indian markets has evolved over the years. Apart from market capitalization, I have alsolooked at annual turnover figures.

Some of the challenges working with large datasets are computational. These can be overcome by using a good program like SAS. STATA. Other major issue is missing values. This can be overcome by dropping those observations for which a particular variable is missing.