Pandas is a Python language package, which is used for data processing. This is a very common basic programming library when we use Python language for machine learning programming. This article is an introductory tutorial to it. Pandas provide fast, flexible and expressive data structures with the goal of making the work of “relational” or “marking” data simple and intuitive. It is intended to be a high-level building block for actual data analysis in Python.Pandas is suitable for many different types of data, including:<ul><li>Table data with heterogeneous columns, such as SQL tables or Excel data.</li><li>Ordered and unordered (not necessarily fixed frequency) time series data.</li><li>Any matrix data with row and column labels (even type or different types)</li><li>Any other form of observation/statistical data set.</li></ul>The Pandas Index object contains metadata describing the axis. When creating a Series or DataFrame, the array or sequence of tags is converted to Index. You can get the Index object of the DataFrame column and row in the following way:

Python data processing with pandas

<UL><LI>Published on May 24, 2018 at 1:06 pm</LI><LI>Updated on June 11, 2018 at 1:04 am</LI></UL><div dir="ltr"> 6,437 article views</div><div dir="ltr"> 7 min read</div>Pandas is a Python language package, which is used for data processing. This is a very common basic programming library when we use Python language for machine learning programming. This article is an introductory tutorial to it. Pandas provide fast, flexible and expressive data structures with the goal of making the work of “relational” or “marking” data simple and intuitive. It is intended to be a high-level building block for actual data analysis in Python.<h2>Pandas Introduction</h2>Pandas is suitable for many different types of data, including:<UL><LI>Table data with heterogeneous columns, such as SQL tables or Excel data.</LI><LI>Ordered and unordered (not necessarily fixed frequency) time series data.</LI><LI>Any matrix data with row and column labels (even type or different types)</LI><LI>Any other form of observation/statistical data set.</LI></UL>Since this is a Python language package, you need to have a Python language environment on your machine. In this regard, please search for your own method on the Internet. For instructions on how to get pandas, please refer to the official website: <a href="https://pandas.pydata.org/pandas-docs/stable/install.html" rel="noopener" dir="ltr">pandas Installation</a>.In general, we can pipperform the installation by:<PRE>sudo pip3 install pandas
</PRE>Or install pandas through <a href="https://pandas.pydata.org/pandas-docs/stable/install.html#installing-pandas-with-anaconda" rel="noopener" dir="ltr">conda</a> :<PRE>conda install pandas
</PRE>Currently, the latest version of pandas is <a href="https://pandas.pydata.org/pandas-docs/stable/whatsnew.html#v0-22-0-december-29-2017" rel="noopener" dir="ltr">v0.22.0</a> .You can find the source code and test data of this article on Github at: <a href="https://github.com/paulQuei/pandas_tutorial" rel="noopener" dir="ltr">pandas tutorial</a> , readers can go for it.In addition, pandas and NumPy often used together, the source code in this article will be used NumPy. It is recommended that readers have some familiarity with NumPy before <a href="https://mindmajix.com/python-training" rel="noopener" dir="ltr">learning Python</a> pandas.<h3>Core data structure</h3>Pandas is the core Seriesand DataFrametwo data structures.The comparison of these two types of data structures is as follows:<td>Name</td><td>Dimensions</td><td>Instructions</td><td>DataFrame</td><td>2dimensional</td><td>Table structure, with tags, variable size, and can contain heterogeneous data columns</td>A DataFrame can be thought of as a Container of Series, ie a DataFrame can contain several Series.“Note: Before the 0.20.0 release, there was a three-dimensional data structure named Panel. This is also the reason for the pandas library name: pan (el)- da (ta)- s . However, this data structure has been abandoned because it is rarely used.”<h3>Series</h3>Since Series is a one-dimensional structure of data, we can create this data directly through an array, like this:<PRE># data_structure.py
import pandas as pd
import numpy as np
series1 = pd.Series([1, 2, 3, 4])
print("series1:\n{}\n".format(series1))
The output of this code is as follows:
series1:
0 1
1 2
2 3
3 4
dtype: int64
</PRE>This output is described as follows:<UL><LI>The last line of output is the type of data in the Series, where the data is all int64types.</LI><LI>The data is output in the second column. The first column is the index of the data, which is called in pandas Index.</LI></UL>We can print the data and indexes in the Series separately:<PRE># data_structure.py
print("series1.values: {}\n".format(series1.values))
print("series1.index: {}\n".format(series1.index))
The two lines of code output are as follows:
series1.values: [1 2 3 4]
series1.index: RangeIndex(start=0, stop=4, step=1)
</PRE>If you do not specify (as above), the index is in the form of [1, N-1]. However, we can also specify the index when creating the Series. The index does not necessarily need to be an integer and can be any type of data, such as a string. For example, we map seven notes in seven letters. The purpose of the index is to use it to obtain the corresponding data, such as the following:<PRE># data_structure.py
series2 = pd.Series([1, 2, 3, 4, 5, 6, 7],
 index=["C", "D", "E", "F", "G", "A", "B"])
print("series2:\n{}\n".format(series2))
print("E is {}\n".format(series2["E"]))
The output of this code is as follows:
series2:
C 1
D 2
E 3
F 4
G 5
A 6
B 7
dtype: int64
E is 3
</PRE><h3>DataFrame</h3>Let’s take a look at the creation of the DataFrame. We can create a 4×4 matrix through the NumPy interface to create a DataFrame like this:<PRE># data_structure.py
df1 = pd.DataFrame(np.arange(16).reshape(4,4))
print("df1:\n{}\n".format(df1))
The output of this code is as follows:
df1:
 0 1 2 3
0 0 1 2 3
1 4 5 6 7
2 8 9 10 11
3 12 13 14 15
</PRE>From this output, we can see that the default index and column names are of the form <code dir="ltr">[0, N-1]</code>.We can specify the column name and index when creating the DataFrame, like this:<PRE># data_structure.py
df2 = pd.DataFrame(np.arange(16).reshape(4,4),
 columns=["column1", "column2", "column3", "column4"],
 index=["a", "b", "c", "d"])
print("df2:\n{}\n".format(df2))
The output of this code is as follows:
df2:
 column1 column2 column3 column4
a 0 1 2 3
b 4 5 6 7
c 8 9 	10 	11
d 	12 	13 	14 	15</PRE>We can also specify column data directly to create a DataFrame:<PRE># data_structure.py
df3 = pd.DataFrame({"note" : ["C", "D", "E", "F", "G", "A", "B"],
 "weekday": ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"]})
print("df3:\n{}\n".format(df3))
The output of this code is as follows:
df3:
 note weekday
0 C 	Mon
1 D 	Tue
2 E 	Wed
3 F 	Thu
4 G 	Fri
5 A 	Sat
6 B 	Sun</PRE>Please note:<UL><LI>Different columns of DataFrame can be different data types</LI><LI>If you create a DataFrame with a Series array, each Series becomes a row, not a column</LI></UL>E.g:<PRE># data_structure.py
noteSeries = pd.Series(["C", "D", "E", "F", "G", "A", "B"],
 index=[1, 2, 3, 4, 5, 6, 7])
weekdaySeries = pd.Series(["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"],
 index=[1, 2, 3, 4, 5, 6, 7])
df4 = pd.DataFrame([noteSeries, weekdaySeries])
print("df4:\n{}\n".format(df4))

The output of df4 is as follows:

df4:
 	1 2 3 4 5 6 7
0 C D E F G A B
1 Mon Tue Wed Thu Fri Sat Sun</PRE>We can add or remove column data to DataFrame in the following form:<PRE># data_structure.py
df3["No."] = pd.Series([1, 2, 3, 4, 5, 6, 7])
print("df3:\n{}\n".format(df3))
del df3["weekday"]
print("df3:\n{}\n".format(df3))
The output of this code is as follows:
df3:
 note weekday No.
0 C 	Mon 1
1 D 	Tue 2
2 E 	Wed 3
3 F 	Thu 4
4 G 	Fri 5
5 A 	Sat 6
6 B 	Sun 7
df3:
 note No.
0 C 1
1 D 2
2 E 3
3 F 4
4 G 5
5 A 6
6 B 7</PRE><h3>Index object and data access</h3>The Pandas Index object contains metadata describing the axis. When creating a Series or DataFrame, the array or sequence of tags is converted to Index. You can get the Index object of the DataFrame column and row in the following way:<PRE># data_structure.py
print("df3.columns\n{}\n".format(df3.columns))
print("df3.index\n{}\n".format(df3.index))
The two lines of code output are as follows:
df3.columns
Index(['note', 'No.'], dtype='object')
df3.index
RangeIndex(start=0, stop=7, step=1)
</PRE>Please note:<UL><LI>Index is not a collection, so it can contain duplicate data</LI><LI>The value of the Index object cannot be changed, so it can access data securely</LI></UL>DataFrame provides the following two operators to access the data:<UL><LI>loc: Accessing Data Through Row and Column Indexes</LI><LI>iloc: Accessing data through row and column subscripts</LI></UL>For example:<PRE># data_structure.py
print("Note C, D is:\n{}\n".format(df3.loc[[0, 1], "note"]))
print("Note C, D is:\n{}\n".format(df3.iloc[[0, 1], 0]))
Note C, D is:
0 C
1 D
Name: note, dtype: object
 
Note C, D is:
0 C
1 D
Name: note, dtype: object
</PRE>The first line of code accesses elements whose row indexes are 0 and 1 and whose column index is “note”. The second line of code accesses the row indices 0 and 1 (for df3, the row index and row subscript are exactly the same, so here are 0 and 1, but they are different meanings), the column subscript Is an element of 0.<h2>Conclusion</h2>This article is a first introductory tutorial for pandas, so here we covered the most basic operations. You can <a href="https://datascienceplus.com/hands-on-tutorial-on-python-data-processing-library-pandas-part-2/" dir="ltr">read next article</a> in the series here. We hope that you understood the tutorial well and if you have any queries, please drop your comment in below comment box. We will get back to you as soon as possible.Note: I learnt this pandas tutorial from this great resources <a href="https://pandas.pydata.org/pandas-docs/stable/" dir="ltr">Pandas – Powerful Python Data Analysis Toolkit</a> and <a href="https://jmetz.github.io/2016/01/30/python-data-analysis-3.html" dir="ltr">Python Data Analysis</a> by J.Metz. It is a great tutorial and I highly recommend to read it if you are more interested in Pandas and Python data analysis!<div>Author</div><img alt="" src="https://secure.gravatar.com/avatar/69d84e5bedf1bc0a57bbb6d7c9a484bc?s=60&amp;d=mm&amp;r=g" height="60" width="60" srcset="https://secure.gravatar.com/avatar/69d84e5bedf1bc0a57bbb6d7c9a484bc?s=120&amp;d=mm&amp;r=g 2x"><h3>Disclosure</h3><UL><LI>Kiran Gutha
does not work or receive funding from any company or organization that would benefit from this article. Views expressed here are personal and not supported by university or company.</LI></UL><div dir="ltr"> Tags</div>

Pandas is a Python language package, which is used for data processing. This is a very common basic programming library when we use Python language for machine learning programming. This

computerscience

Computer Science

softwareengineering

Software Engineering

<ul><li>Published on May 24, 2018 at 1:06 pm </li><li>Updated on June 11, 2018 at 1:04 am </li> </ul>

Pandas is a Python language package, which is used for data processing. This is a very common basic programming library when we use Python language for machine learning programming. This article is an introductory tutorial to it. Pandas provide fast, flexible and expressive data structures with the goal of making the work of “relational” or “marking” data simple and intuitive. It is intended to be a high-level building block for actual data analysis in Python.

Pandas is suitable for many different types of data, including:

<ul><li>Table data with heterogeneous columns, such as SQL tables or Excel data. </li><li>Ordered and unordered (not necessarily fixed frequency) time series data. </li><li>Any matrix data with row and column labels (even type or different types) </li><li>Any other form of observation/statistical data set. </li> </ul>

Since this is a Python language package, you need to have a Python language environment on your machine. In this regard, please search for your own method on the Internet. For instructions on how to get pandas, please refer to the official website: <a href="https://pandas.pydata.org/pandas-docs/stable/install.html">pandas Installation </a>.

In general, we can pipperform the installation by:

Or install pandas through <a href="https://pandas.pydata.org/pandas-docs/stable/install.html#installing-pandas-with-anaconda">conda </a>:

Currently, the latest version of pandas is <a href="https://pandas.pydata.org/pandas-docs/stable/whatsnew.html#v0-22-0-december-29-2017">v0.22.0 </a>.

You can find the source code and test data of this article on Github at: <a href="https://github.com/paulQuei/pandas_tutorial">pandas tutorial </a>, readers can go for it.

In addition, pandas and NumPy often used together, the source code in this article will be used NumPy. It is recommended that readers have some familiarity with NumPy before <a href="https://mindmajix.com/python-training">learning Python </a>pandas.

Pandas is the core Seriesand DataFrametwo data structures.

The comparison of these two types of data structures is as follows:

A DataFrame can be thought of as a Container of Series, ie a DataFrame can contain several Series.

“Note: Before the 0.20.0 release, there was a three-dimensional data structure named Panel. This is also the reason for the pandas library name: pan (el)- da (ta)- s . However, this data structure has been abandoned because it is rarely used.”

Since Series is a one-dimensional structure of data, we can create this data directly through an array, like this:

The output of this code is as follows: series1: 0 1 1 2 2 3 3 4 dtype: int64

This output is described as follows:

<ul><li>The last line of output is the type of data in the Series, where the data is all int64types. </li><li>The data is output in the second column. The first column is the index of the data, which is called in pandas Index. </li> </ul>

We can print the data and indexes in the Series separately:

The two lines of code output are as follows: series1.values: [1 2 3 4] series1.index: RangeIndex(start=0, stop=4, step=1)

If you do not specify (as above), the index is in the form of [1, N-1]. However, we can also specify the index when creating the Series. The index does not necessarily need to be an integer and can be any type of data, such as a string. For example, we map seven notes in seven letters. The purpose of the index is to use it to obtain the corresponding data, such as the following:

The output of this code is as follows: series2: C 1 D 2 E 3 F 4 G 5 A 6 B 7 dtype: int64 E is 3

Let’s take a look at the creation of the DataFrame. We can create a 4×4 matrix through the NumPy interface to create a DataFrame like this:

The output of this code is as follows: df1: 0 1 2 3 0 0 1 2 3 1 4 5 6 7 2 8 9 10 11 3 12 13 14 15

From this output, we can see that the default index and column names are of the form <code>[0, N-1] </code>.

We can specify the column name and index when creating the DataFrame, like this:

The output of this code is as follows: df2: column1 column2 column3 column4 a 0 1 2 3 b 4 5 6 7 c 8 9 10 11 d 12 13 14 15

We can also specify column data directly to create a DataFrame:

The output of this code is as follows: df3: note weekday 0 C Mon 1 D Tue 2 E Wed 3 F Thu 4 G Fri 5 A Sat 6 B Sun

<ul><li>Different columns of DataFrame can be different data types </li><li>If you create a DataFrame with a Series array, each Series becomes a row, not a column </li> </ul>

The output of df4 is as follows: df4: 1 2 3 4 5 6 7 0 C D E F G A B 1 Mon Tue Wed Thu Fri Sat Sun

We can add or remove column data to DataFrame in the following form:

The output of this code is as follows: df3: note weekday No. 0 C Mon 1 1 D Tue 2 2 E Wed 3 3 F Thu 4 4 G Fri 5 5 A Sat 6 6 B Sun 7 df3: note No. 0 C 1 1 D 2 2 E 3 3 F 4 4 G 5 5 A 6 6 B 7

The Pandas Index object contains metadata describing the axis. When creating a Series or DataFrame, the array or sequence of tags is converted to Index. You can get the Index object of the DataFrame column and row in the following way:

The two lines of code output are as follows: df3.columns Index(['note', 'No.'], dtype='object') df3.index RangeIndex(start=0, stop=7, step=1)

<ul><li>Index is not a collection, so it can contain duplicate data </li><li>The value of the Index object cannot be changed, so it can access data securely </li> </ul>

DataFrame provides the following two operators to access the data:

<ul><li>loc: Accessing Data Through Row and Column Indexes </li><li>iloc: Accessing data through row and column subscripts </li> </ul>

Note C, D is: 0 C 1 D Name: note, dtype: object Note C, D is: 0 C 1 D Name: note, dtype: object

The first line of code accesses elements whose row indexes are 0 and 1 and whose column index is “note”. The second line of code accesses the row indices 0 and 1 (for df3, the row index and row subscript are exactly the same, so here are 0 and 1, but they are different meanings), the column subscript Is an element of 0.

This article is a first introductory tutorial for pandas, so here we covered the most basic operations. You can <a href="https://datascienceplus.com/hands-on-tutorial-on-python-data-processing-library-pandas-part-2/">read next article </a>in the series here. We hope that you understood the tutorial well and if you have any queries, please drop your comment in below comment box. We will get back to you as soon as possible.

Note: I learnt this pandas tutorial from this great resources <a href="https://pandas.pydata.org/pandas-docs/stable/">Pandas – Powerful Python Data Analysis Toolkit </a>and <a href="https://jmetz.github.io/2016/01/30/python-data-analysis-3.html">Python Data Analysis </a>by J.Metz. It is a great tutorial and I highly recommend to read it if you are more interested in Pandas and Python data analysis!

<ul><li>Kiran Gutha does not work or receive funding from any company or organization that would benefit from this article. Views expressed here are personal and not supported by university or company. </li> </ul>

Hands-on Tutorial on Python Data Processing Library Pandas – Part 1

Python data processing with pandas

Related collections

Other curated ideas on this topic: