Ordinary least squares, or linear least squares, estimates the parameters in a regression model by minimizing the sum of the squared residuals.

What is Ordinary Least Square Method?

OLS Assumptions are the Conditions that we need to consider them before performing Regression Analysis.Some OLS Assumptions are:  <ol><li>Linearity</li><li>No Endogeneity</li><li>Normality</li><li>Zero Mean of Error Terms</li><li>Homoscedasticity</li><li>No Autocorrelation</li><li>No Multicollinearity</li></ol>

Assumptions for OLS

<ul><li>A Linear Regression is Linear because the Equation is Linear.</li><li>To verify the Linearity between Independent and Dependent Variable apply Scatter plot.</li><li>If the Data points form a pattern that looks like a Straight Line then Linear Regression is Suitable.</li></ul>

1. Linearity

If the pattern doesn't looks like a Straight Line, then we need to apply<ul><li class="ql-align-justify">A Non Linear Regression or</li><li class="ql-align-justify">Exponential Transformation or</li><li class="ql-align-justify">Log Transformation.</li></ul>

Non-Linear Pattern

<ul><li>Technically, Endogeneity occurs when a predictor variable (x) in a regression model is correlated with the error term (e) in the model.</li><li>This usually occurs due to the Omitted Variable Bias (when we forget to include a relevant variable in the Model).</li><li>As the thing that we don’t explain with our model goes into the error, the error would be correlated to the variable that we have omitted.</li><li>Omitted Variable Bias is hard to find. When in doubt, include the Variable in the model and try it yourself. </li></ul>

2. No Endogeneity

<ul><li>We need to consider that our Error Term is normally distributed.</li><li>We only need Normal Distribution while making Statistical Inferences.</li><li>T-tests and F-tests work because we have assumed normality.</li><li>If Error Term isn't normally distributed then we can use Central Limit Theorem.</li></ul>

3. Normality of Error Terms

<ul><li>If the Mean of the Error Terms is not expected to be zero then the line is not the Best fitting one.</li><li>Having an Intercept Solves the problem. </li></ul>

4. Zero Mean of Error Terms

<ul><li>Homoscedasticity means to have “Equal Variance”. The error terms should have equal variance with one another.</li><li>When the error terms don’t have “Equal Variance”, then Heteroscedasticity happens.</li></ul>

5. Homoscedasticity

In order to prevent Heteroscedasticity, we need to<ul><li>Look for Omitted Variable Bias.</li><li>Look for Outliers.</li><li>Apply log Transformation.</li></ul>

Heteroscedasticity 

<ul><li>Autocorrelation is a mathematical representation of the degree of similarity between a given time series and a lagged version of itself over successive time intervals.</li><li>It is similar to the Correlation between two different Time Series but Autocorrelation uses same time series twice.</li><li>One is Original form and other one is Lagged version of Original form.</li></ul>

6. No Autocorrelation 

To detect autocorrelation<ul><li>Plot all points and check for patterns or</li><li>Use Durbin - Watson test.</li></ul>There is no remedy for Autocorrelation. Instead of linear regression, we can use<ul><li>Autoregressive Models.</li><li>Moving Average Models.</li></ul>

Autocorrelation Detection

<ul><li>We observe Multicollinearity when two or more Independent Variables in a model are “Highly Correlated”.</li><li>Correlation could be Positive (or) Negative.</li><li>This will lead result in less reliable predictions.</li><li>Multicollinearity can be noticed easily by finding the correlation between each two pairs of Independent Variables.</li></ul>

7. No Multicollinearity

To fix Multicollinearity: <ul><li>If the Dataset is small, we can drop one independent variable.</li><li>If the Dataset is large, we will use Ridge and Lasso Regression.</li></ul>

Fixing Multicollinearity

These are some of the Assumptions to be pondered while Applying Ordinary Least Square Method and Performing Regression Analysis.

Education

OLS (Ordinary Least Square) Assumptions

First, we need to get an outline from an image. I decided to use Canny edge detection to extract the edges from an image (in the example above, a drawing of Sherlock Holmes). This will give us a 2D binary array indicating the edges of the image. Now, we need to represent these points as complex coordinates oriented about the origin. The x-axis will represent the real component, and the y-axis will represent the imaginary component.

1. Convert outline to a complex signal

Now, we need to order the points to make the progression as smooth as possible. In practice, I’ve found ordering to minimize the difference from one point to the next is a good approach!

2. Reorder to minimize distance between sequential points

We’ll now apply the Fourier transform to extract the magnitude and phase information associated with each frequency. As visualizing all the oscillators is slow, we’ll filter out all the frequency components above self._n_oscillators .

3. Call the Fourier transform

In other articles, I’ve written about the application of the Fourier transform to determine the frequency components of real or complex signals. We’ve discussed frequency, phase, and magnitude in…

Teach (and learn) the Fourier transform geometrically!

Without buy-in from your company’s rank and file, even the cleverest AI-derived model will sit idle and “data-driven decision-making” will just go around in circles. Companies need to start seeing regular people as part of their data strategy. Data teams must work with regular people every day, develop a feel for their problems and opportunities, and embrace their hopes and fears surrounding data, then focus on equipping people with the tools they need to formulate and solve their own problems.

The Key Takeaway

To drive the importance of regular people home, consider the process of completing a data science (big data, analytics, artificial intelligence) project. In general, this requires five steps: understanding the problem, collecting and preparing the data, analyzing that data, formulating the findings, and finally, putting those findings to work. At each step, regular people have a critical role to play — as collaborators, as customers, and as creators of the data used — and there are serious consequences for not including them. Doing each step well depends on regular people.

The Importance Of Regular People

Many managers, unconsciously perhaps, have debilitating pre-conceptions about people. They view them as part of the problem — out-of-date, ill-suited to the rigors of data, and resistant to the new ideas. Large numbers know that data is increasingly important, have great ideas for making improvements, and want to create opportunities for themselves. Engaging them is simply not that difficult.Leaders and companies need to reboot their outlook and see people as part of the solution. Managers need to “start small,” asking people where they see opportunity.

See Regular People as Part of The Solution

Real gains come when companies and leaders start to see data as a means to empower people — a way for them to minimize the mundane parts of their jobs, take a measure of control, unleash their creative juices, learn new skills, lean into the satisfying parts of their jobs, and advance their careers. You will have to adopt a pro-active attitude, provide some encouragement and training, and help.

Data As A Means To Empower People

Companies simply must realign a substantial portion of their data science, quality, architecture, and monetization programs to engage regular people.To do this, data teams must work with regular people every day, develop a feel for their problems and opportunities, and embrace their hopes and fears surrounding data. They must focus less on big data and more on equipping people with the tools they need to formulate and solve their own problems. Data teams must seek joy not in a clever model, but in business results and the successes of those they serve.

Re-Orient Your Data Programs to Get Everyone Involved

Every data project should start with two questions:Who will this effort touch?How do we get them involved as soon as possible?Then ask those people to work with you — and have a good answer when they ask, “what would you like me to do?”

Two Questions Before Any Data Project

A small team of regular people, having been trained and supported by an embedded data manager, sorts out the data it needs to complete its work, measure the quality of that data, identify, then proactively attack the root causes of their data quality issues, making them go away, forever. There is a resonant theme for data teams here — a shift from an “inside-out” to an “outside-in” perspective. And it will lead to a redeployment of personnel: towards strategic problems, towards small data, towards rooting out quality issues, and towards empowerment.

A Better Way Than Cleaning Up Of Data

Regular people are fully involved with data everyday — they are customers of data created upstream and they are creators of data that others will use; they use data to make decisions and complete their work; they are guardians of the company’s data assets; and they can be small data scientists and collaborators, customers, and data creators in larger data science, artificial intelligence, and digital transformation initiatives.

Clarify Expectations and Get on With It

Companies need to design projects around the employees who actually use them.

Stashed ideas

Your Data Initiatives Can’t Just Be for Data Scientists

It is important to understand the <a href="https://towardsai.net/p/machine-learning/main-types-of-neural-networks-and-its-applications-tutorial-734480d7ec8e">topology </a>of event-driven architecture to deep dive into various sub-patterns of event-driven architecture. This architecture pattern is based on two main <a href="https://towardsai.net/p/machine-learning/main-types-of-neural-networks-and-its-applications-tutorial-734480d7ec8e">topologies </a>: <ul><li>Mediator </li><li>Broker </li> </ul>

Topology of Event-Driven Architecture

The central mediator plays an important role in mediator <a href="https://towardsai.net/p/machine-learning/main-types-of-neural-networks-and-its-applications-tutorial-734480d7ec8e">topology </a>. It is generally used when it is required to orchestrate various steps with an event through a central mediator. It is helpful for events that have multiple events and needs some level of orchestration to process the event.Example: Design GPS Programm in a Vehicle For overall vehicle safety, navigation, and cost estimation plans, GPS data is considered the starting point for designing a proper GPS for a vehicle. They are intended to provide proper business and safety goals.

Mediator Topology

The mediator component acts as an orchestrator for every initial step. It identifies the type of event or type of data and sends it to a specific Ch<a href="https://towardsai.net/p/machine-learning/building-neural-networks-from-scratch-with-python-code-and-math-in-detail-i-536fae5d7bbf">ann </a>el. Several tools are present to handle mediator topology. Example- jBPM, Mule ESB, Apache Camel, Spring Integration, etc.

Mediator

Event Queue is used to carry the event to the event mediator. It is a kind of <a href="https://sponsors.towardsai.net/">enterprise </a>messaging queue system. There are several messaging systems like RabbitMQ, Kafka, etc.

Queue

The event channel is utilized by the event mediator to pass particular events related to each step in the primary event to the event processors. It can be either message topic or message queue.

Channel

It is a component or service application that contains business logic for the execution of business needs on receiving of events. Nowadays these are microservice-based components. It can be a channel queue consumer as well.

Processor

In Broker topology, the message flow is distributed across the event processor on receiving of event. It is different from a mediator because it is not central. It is useful in chain types of processing. supExample: Design Online Food Ordering System Online food ordering is the process of ordering food from a website.<ul><li>User selects restaurant and orders food.</li><li>Pays for the ordered food.</li><li>Restaurant receives notification</li><li>Delivery partner receives notification</li><li>Restaurant handovers food to the delivery partner</li><li>Delivery partner delivers food to the user</li></ul>

Broker Topology

Deep Dive into Event-Driven Architecture

• Kaggle : Kaggle is a platform where professionals can learn, practice, and sharpen their data analytics and data science. The platform provides tons of data that are public and allows the users of the platform to share code so that they can learn the best practices within the data space.

• FiveThirtyEight : FiveThirtyEight is an interactive news and sports platform that has some incredible information for <a href="https://www.analyticsinsight.net/data-science-is-helping-the-world-recover-from-covid-loss/">data visualization </a>projects. The platform makes a lot of their data available to the public, which means they can download and use the information according to their own convenience.

• Google Dataset Search : Google Dataset Search is one of the most comprehensive dataset search engines that are available. It claims to hold more than 25 million online <a href="https://www.analyticsinsight.net/top-10-best-companies-in-india-to-join-as-data-scientists-in-2022/">datasets </a>and assists scientists and researchers in better locating datasets. It is armed with a function, which can sort data types, update dates, and so much more.

• Data.gov : Data.gov allows its users to download and explore data from multiple US government agencies. The information can range from government budgets to climate data. It is documented quite evidently so that it becomes easier for the users to navigate them.

• AWS Public Datasets: AWS Public Datasets allow the users to download the data and work with it on their individual devices. They can analyse the data in the cloud using EC2 and Hadoop via EMR. Amazon has a page that lists all the datasets for its users and also gives free access to all the new accounts.

• UCI Machine Learning Repository : UCI Machine Learning Repository is one of the oldest sources of datasets on the internet. The datasets are generally contributed by the users, and thus have varying levels of documentation and cleanliness. Users can download UCI Machine Learning Repository without any registration.

• Quandl : Quandl is a repository of economic and financial data. Most of this information is free, but some require purchasing. The platform is extremely useful for building models to predict economic indicators and stock prices.

• data.world : data.world describes itself as the social network for <a href="https://www.analyticsinsight.net/top-10-profitable-data-science-tools-used-in-business-in-2022/">data </a>professionals. It is a platform where they can search for copy, analyze, and download datasets. In addition to this, they can upload their data and use it to collaborate with others.

• Buzzfeed News : Buzzfeed provides datasets, analysis, libraries, tools, and guides that are used in the articles available on GitHub. It is a quite popular platform and is used by millions of data professionals.

• Academic Torrents : Academic Torrents is a new site that is geared around sharing the <a href="https://www.analyticsinsight.net/top-biggest-data-science-conferences-to-attend-in-2022/">datasets </a>from specific scientific papers. It is new to the dataset platform market and allows its users to browse data directly on the site.

As the need for data science increases, individuals are looking for authentic datasets platforms for their data science projects. This article lists the top datasets platforms that can be used in 2022.

OLS (Ordinary Least Square) Assumptions

Teach (and learn) the Fourier transform geometrically!

Your Data Initiatives Can’t Just Be for Data Scientists

Deep Dive into Event-Driven Architecture

10 Platforms to Get Datasets for Data Science Projects in 2022

It's time to

READ

LIKE

A PRO!