Data science process: 2. Retrieving Data

Data science process is a series of activities that help us move from a business problem to a useful solution based on data. 1. Define research goals 2. Data retrieval 3. Data preparation 4. Data exploration 5. Model building 6. Presenting results and automation

2. Retrieving Data

• After defining the research goal, the next important step in the Data Science process is collecting or retrieving the data that we will use for analysis.

• Without the right data, no analysis can be accurate ‒ so this step needs careful planning and checking.

Data can be 1.Internal data and 2.External data

1) Internal data: This data is already present within the company database.

2) External data : This data is available in public domain, social media APIs or with partners.

• Data retrieval can be done by collecting internal and external data both and then by performing some early quality checks on the data.

1. Start with Data Available within the Company

• Most of the companies already store a large amount of data that we can use.

• Data can be stored in ‒

o Databases: These are used mainly for storing and managing data using MySQL, PostgreSQL.

o Data marts: These are small focused parts of data warehouse created for specific departments.

́o Data warehouses: Large systems that collect and organize data from different sources so it can be analyzed.

o Data lakes: These store raw, unprocessed data (structured or unstructured) for future use.

For example ‒ A retail company may have a customer database that stores names and address of the customers. A sales data mart for tracking sales by region. A data warehouse that combines both overall analysis.

Common challenge

•  Even inside a company, finding the right data can be difficult because:

1) Data might be stored in different departments.

2) Employees who knew where data was stored might have changed jobs or left the company.

3) Different teams might use different formats for storing similar data.

4) Proper documentation is not maintained.

5) Sometimes company politics may make it difficult to get the right data.

•  One has to always talk to the company's IT or database team to understand what data is already available and how one can access it.

2. Look for External Data Sources (Shop Around)

• If the company does not have enough data, we can look for external sources. Possible external sources can be ‒

1) Public data: Many Governments and Organizations share open data online. For example ‒ data.gov.in(Indian Government), WHO, UNO,World Bank datasets.

2) Commercial data providers: Some companies sell or share specialized data.

3) Social media platforms: Platforms like Twitter, LinkedIn, and Facebook provide APIs (Application Programming Interfaces) that allow you to collect public data, such as posts, trends, or user engagement metrics.

4) Collaborations: Sometimes organizations share data with partners to improve services ‒ for example, a food delivery company like Swiggy or Zomato sharing restaurant data with a map provider.

3. Perform Data Quality Checks Early

• Once data is collected it is important to check the quality of the data before moving ahead. If the collected data is incorrect, incomplete or inconsistent, our analysis and model will give us wrong results ‒ no matter how advanced algorithms are used.

• Some of the common quality checks are ‒

1) Are there missing values?

2) Are there duplicate values?

3) Are the data types of the data correct?

4) Are there any unrealistic values?

• Early data quality check during data retrieval stage is important because careless importing or checking can create big problems later.

• This step of data quality checks should not take much time. Once we have enough proof that the collected data matches the information in the original source, we can stop checking. In the next stage ‒  data preparation ‒  we can perform a more detailed and thorough check of the data.

Review Questions

1. Explain what is meant by data retrieval in the data science process.

2. Why is data retrieval considered an important step in data science?

3. What are the main sources of data used in data science projects?

4. What challenges might a data scientist face during the data retrieval stage?

5. Why should data quality checks be performed during the data retrieval stage?