How to normalize data in pandas crosstab
Pandas Crosstabs
Python
pandas groupby filter by column values and conditional aggregation
In this post, we will learn how to filter column values in a pandas group by and apply conditional aggregations such as sum, count, average etc.
numpy tile vs repeat
In this post, we will learn what is numpy tile and what’s the difference between numpy tiles and repeat
How to install tensorflow in a conda environment
In this post, we will learn how to install tensorflow 2 in a conda environment. I would be installing tensorflow in two steps, first we will create a conda p...
Overlay transparent polygons on image using python
In this post, we will see how to overlay a polygon on images using opencv and PIL, the polygon is defined as a series of vertices inside an array and we will...
Show images in grid inside jupyter notebook using matplotlib and numpy
In this post, we will see how to display the images in a grid using matplotlib imshow and Image grid. Alternatively, we will see the visualization of image b...
matplotlib fill color between vertical, horizontal lines and polygon
In this post, we will learn how to fill color in the matplotlib charts between two vertical or Horizaontal lines and also inside the Polygons
Matplotlib Scatter Plot Color by Category in Python
Scatter plot are useful to analyze the data typically along two axis for a set of data. It shows the relationship between two sets of data
Find nearest neighbor using KD Tree
KD Tree is a modified Binary Search Tree(BST) that can perform search in multi-dimensions and that’s why K-dimensional.
Vectors, Matrix And Tensors
In this post we will see how large data is stored in multi-dimensional arrays, which is also called as tensors.
Python API Documentation using Flask and Swagger
What is Swagger?
Create Interactive Dashboard in Python using Streamlit
Dashboard gives a graphical interface to visualize the key indicators and trends of your data. However, Creating Dashboard is always been a tedious task for ...
Compare two Numpy arrays for equality
In this post we will compare elements of two arrays for equality. This would be really helpful when you wanted to compare if two similar arrays coming out th...
How to split Numpy Arrays
In this post we will see how to split a 2D numpy array using split, array_split , hsplit, vsplit and dsplit.
Dataframe groupby date and time
In this post we will see how to group a timeseries dataframe by Year,Month, Weeks or days. Additionally, we will also see how to groupby time objects like ho...
Decision Tree in Sklearn
In this post we are going to see how to build a basic decision tree classifier using scikit-learn package and how to use it for doing multi-class classificat...
Time Series Analysis and Forecasting with ARIMA
In the previous post we have seen how to visualize a time series data. In this post we will discuss how to do a time series modelling using ARMA and ARIMA mo...
Time Series Data Visualization
Visualizing Time Series data with Python
How to Remove Outliers in Python
Introduction
How to create bins in pandas using cut and qcut
Data Binning
Resample and Interpolate time series data
Resampling is a method of frequency conversion of time series data. You can use resample function to convert your data into the desired frequency.
Convert Pandas dataframe to dictionary
In my this blog we will discover what are the different ways to convert a Dataframe into a Python Dictionary or Key/Value Pair
How to use Pandas Count and Value_Counts
Counting number of Values in a Row or Columns is important to know the Frequency or Occurrence of your data.
Parallelize pandas apply using dask and swifter
Using Pandas apply function to run a method along all the rows of a dataframe is slow and if you have a huge data to apply thru a CPU intensive function then...
Sort Pandas Dataframe and Series
Sorting a dataframe by row and column values or by index is easy a task if you know how to do it using the pandas and numpy built-in functions
Pandas dataframe filter with Multiple conditions
Selecting or filtering rows from a dataframe can be sometime tedious if you don’t know the exact methods and how to filter rows with multiple conditions
Find K smallest and largest values and its indices in a numpy array
To find the maximum and minimum value in an array you can use numpy argmax and argmin function
Concatenating arrays in Numpy
We will be discussing about merging numpy arrays and different functions that are available in the toolbox to perform this job
Reshaping numpy arrays in python
Reshape is an important feature which lets you to change the shape of your array without changing its data
How to work with numpy.where()
What is numpy.where()
How to create interactive data visualization using plotly
Visualization is the graphical representation of your data and it let you paint your data into a canvas in a way you want to see it. There are lot of amazing...
How to calculate Distance in Python and Pandas using Scipy spatial and distance functions
Working with Geo data is really fun and exciting especially when you clean up all the data and loaded it to a dataframe or to an array. The real works starts...
How to work with JSON in Pandas
JSON is widely used format for storing the data and exchanging. Many of the API’s response are JSON and being light weight it’s used almost everywhere
How to iterate through a python dictionary
A python Dictionary is one of the important data structure which is extensively used in data science and elsewhere when you want to store the data as a key-v...
A primer on Python Regular Expression
Regex is a group of characters which helps to find pattern within a string. Regex is used in lot of applications including the search engines, search and for...
Pandas apply, map and applymap
In this post we will see how to apply a function along the axis of a dataframe using apply and applymap and how to map the values of a Series from one domain...
How to create dataframe for testing?
Did you ever wanted to create dataframes for testing and find it hard to fill the dataframe with dummy values then DO NOT Worry there are functions that are ...
How to use Regex in Pandas
There are several pandas methods which accept the regex in pandas to find the pattern in a String within a Series or Dataframe object. These methods works on...
Python Detect and Translate language
The internet is flooded with articles and posts for translating the language using Machine Learning or Deep Learning LSTM models and building a deep neural n...
How to remove duplicate data from python dataframe
Not all data are perfect and we really need to get duplicate data removed from our dataset most of the time. it looks easy to clean up the duplicate data but...
Python Logging
Log is an important tool for any developer. it helps in debugging and log important information or exceptions that emits while the code executes
Working with Pandas datetime
In this post we will explore the Pandas datetime methods which can be used instantaneously to work with datetime in Pandas.
How to find Percentage Change in pandas
So you are interested to find the percentage change in your data. Well it is a way to express the change in a variable over the period of time and it is heav...
Building a Web app using Python and Mongodb
Introduction
Dataframe Visualization with Pandas Plot
Visualization has always been challenging task but with the advent of dataframe plot() function it is quite easy to create decent looking plots with your dat...
How to shift a column in Pandas
If you want to shift your columns without re-writing the whole dataframe or you want to subtract the column value with the previous row value or if you want ...
Pandas Groupby Tutorial
Hope if you are reading this post then you know what is groupby in SQL and how it is being used to aggregate the data of the rows with the same value in one ...
Pandas Dataframe Align function
Pandas Align basically helps to align the two dataframes have the same row and/or column configuration and as per their documentation it Align two objects on...
Pandas Transform and Filter
In this blog we will see how to use Transform and filter on a groupby object. We all know about aggregate and apply and their usage in pandas dataframe but h...
Pandas Coalesce - How to Replace NaN values in a dataframe
In this post we will discuss on how to use fillna function and how to use SQL coalesce function with Pandas, For those who doesn’t know about coalesce functi...
Add new rows and columns to Pandas dataframe
We often get into a situation where we want to add a new row or column to a dataframe after creating it. A quick and dirty solution which all of us have trie...
How to create Pandas Pivot Table
Pivot table lets you calculate, summarize and aggregate your data. MS Excel has this feature built-in and provides an elegant way to create the pivot table f...
Pandas How to replace values based on Conditions
Using these methods either you can replace a single cell or all the values of a row and column in a dataframe based on conditions .
Pandas Difference Between two Dataframes
There are often cases where we need to find out the common rows between the two dataframes or find the rows which are in one dataframe and missing from secon...
Pandas how to get a cell value and update it
Accessing a single value or setting up the value of single row is sometime required when we doesn’t want to create a new Dataframe for just updating that sin...
Pandas Map Dictionary values with Dataframe Columns
Pandas has a cool feature called Map which let you create a new column by mapping the dataframe column values with the Dictionary Key. Let’s understand this ...
Pandas Select rows by condition and String Operations
There are instances where we have to select the rows from a Pandas dataframe by multiple conditions. Especially, when we are dealing with the text data then ...
How to Rename Columns in Pandas
Pandas has two ways to rename their Dataframe columns:
Text Data Visualization in Python
The best way to understand any data is by visualizing it. if I give you a table load of data and Charts then the latter is more easier way to get insight fro...
Sentence Similarity in Python using Doc2Vec
Introduction
Compare two excel files for difference using Python
Comparing two excel spreadsheets and writing difference to a new excel was always a tedious task and Long Ago, I was doing the same thing and the objective t...
Named Entity Recognition: How to Automate Customer Support
Customer support is one of the complex and most important part of any business. This area of business stands to benefit from the machine learning as it is he...
How to find distance between two Points based on Latitude and Longitude using Python and SQL
if you are working with GIS or POI data then you must be dealing with lat/long values and there would be use cases to calculate the distance between two poin...
How to Tame a Python
All of those out there who claim that you can learn python in 3,6 or 9 days or 1 month are just fooling around and f** up with your mind. Get it straight you...
Google Takeout: How to download your personal google data
I always wondered how our life would have been if Google hadn’t been there. We depend on most of things from our personal to professional life on Google and ...
Color Columns, Rows & Cells of Pandas Dataframe
I always wanted to highlight the rows,cells and columns which contains some specific kind of data for my Data Analysis. I wanted to Know which cells contains...
Text Matching: Cosine Similarity
Recently I was working on a project where I have to cluster all the words which have a similar name. For a novice it looks a pretty simple job of using some ...
Read Google Spreadsheet data into Pandas Dataframe
Many a times it happens that we have our data stored on a Google drive and to analyze that data we have to export the data as csv or xlsx and store it on a d...
Draw Pencil Sketches and Play with Photos
During my childhood, I was always fascinated to have my Pencil sketch and impress my school mates. However at that time it wasn’t an easy job for school goer...
Web Scraping Made Easy
Introduction
Pandas in a nutshell
Introduction
A Glimpse of Jupyterlab
Introduction
Merge Images with Python
Introduction
Get Started with Matplotlib - Data Visualization for Python
Why Visualization?
Exploratory Analysis of H1B Visa
The H-1B is a non-immigrant visa in the United States, it is designed to bring foreign professionals with college degrees and specialized skills to fill jobs...
Data Analysis of IMDB Data
[youtube https://www.youtube.com/watch?v=mS3dzczv1ZQ?version=3&rel=1&fs=1&autohide=2&showsearch=0&showinfo=1&iv_load_policy=1&sta...
Data Science
Dataframe groupby date and time
In this post we will see how to group a timeseries dataframe by Year,Month, Weeks or days. Additionally, we will also see how to groupby time objects like ho...
Decision Tree in Sklearn
In this post we are going to see how to build a basic decision tree classifier using scikit-learn package and how to use it for doing multi-class classificat...
Time Series Analysis and Forecasting with ARIMA
In the previous post we have seen how to visualize a time series data. In this post we will discuss how to do a time series modelling using ARMA and ARIMA mo...
Time Series Data Visualization
Visualizing Time Series data with Python
Resample and Interpolate time series data
Resampling is a method of frequency conversion of time series data. You can use resample function to convert your data into the desired frequency.
Reshaping numpy arrays in python
Reshape is an important feature which lets you to change the shape of your array without changing its data
How to work with numpy.where()
What is numpy.where()
How to create interactive data visualization using plotly
Visualization is the graphical representation of your data and it let you paint your data into a canvas in a way you want to see it. There are lot of amazing...
How to calculate Distance in Python and Pandas using Scipy spatial and distance functions
Working with Geo data is really fun and exciting especially when you clean up all the data and loaded it to a dataframe or to an array. The real works starts...
How to work with JSON in Pandas
JSON is widely used format for storing the data and exchanging. Many of the API’s response are JSON and being light weight it’s used almost everywhere
Pandas apply, map and applymap
In this post we will see how to apply a function along the axis of a dataframe using apply and applymap and how to map the values of a Series from one domain...
How to create dataframe for testing?
Did you ever wanted to create dataframes for testing and find it hard to fill the dataframe with dummy values then DO NOT Worry there are functions that are ...
How to use Regex in Pandas
There are several pandas methods which accept the regex in pandas to find the pattern in a String within a Series or Dataframe object. These methods works on...
Python Detect and Translate language
The internet is flooded with articles and posts for translating the language using Machine Learning or Deep Learning LSTM models and building a deep neural n...
How to remove duplicate data from python dataframe
Not all data are perfect and we really need to get duplicate data removed from our dataset most of the time. it looks easy to clean up the duplicate data but...
Working with Pandas datetime
In this post we will explore the Pandas datetime methods which can be used instantaneously to work with datetime in Pandas.
How to find Percentage Change in pandas
So you are interested to find the percentage change in your data. Well it is a way to express the change in a variable over the period of time and it is heav...
Building a Web app using Python and Mongodb
Introduction
Dataframe Visualization with Pandas Plot
Visualization has always been challenging task but with the advent of dataframe plot() function it is quite easy to create decent looking plots with your dat...
How to shift a column in Pandas
If you want to shift your columns without re-writing the whole dataframe or you want to subtract the column value with the previous row value or if you want ...
Pandas Groupby Tutorial
Hope if you are reading this post then you know what is groupby in SQL and how it is being used to aggregate the data of the rows with the same value in one ...
Pandas Dataframe Align function
Pandas Align basically helps to align the two dataframes have the same row and/or column configuration and as per their documentation it Align two objects on...
Pandas Transform and Filter
In this blog we will see how to use Transform and filter on a groupby object. We all know about aggregate and apply and their usage in pandas dataframe but h...
Pandas Coalesce - How to Replace NaN values in a dataframe
In this post we will discuss on how to use fillna function and how to use SQL coalesce function with Pandas, For those who doesn’t know about coalesce functi...
Add new rows and columns to Pandas dataframe
We often get into a situation where we want to add a new row or column to a dataframe after creating it. A quick and dirty solution which all of us have trie...
How to create Pandas Pivot Table
Pivot table lets you calculate, summarize and aggregate your data. MS Excel has this feature built-in and provides an elegant way to create the pivot table f...
Pandas How to replace values based on Conditions
Using these methods either you can replace a single cell or all the values of a row and column in a dataframe based on conditions .
Pandas Difference Between two Dataframes
There are often cases where we need to find out the common rows between the two dataframes or find the rows which are in one dataframe and missing from secon...
Pandas how to get a cell value and update it
Accessing a single value or setting up the value of single row is sometime required when we doesn’t want to create a new Dataframe for just updating that sin...
Pandas Map Dictionary values with Dataframe Columns
Pandas has a cool feature called Map which let you create a new column by mapping the dataframe column values with the Dictionary Key. Let’s understand this ...
Pandas Select rows by condition and String Operations
There are instances where we have to select the rows from a Pandas dataframe by multiple conditions. Especially, when we are dealing with the text data then ...
How to Rename Columns in Pandas
Pandas has two ways to rename their Dataframe columns:
Text Data Visualization in Python
The best way to understand any data is by visualizing it. if I give you a table load of data and Charts then the latter is more easier way to get insight fro...
Sentence Similarity in Python using Doc2Vec
Introduction
Named Entity Recognition: How to Automate Customer Support
Customer support is one of the complex and most important part of any business. This area of business stands to benefit from the machine learning as it is he...
How to find distance between two Points based on Latitude and Longitude using Python and SQL
if you are working with GIS or POI data then you must be dealing with lat/long values and there would be use cases to calculate the distance between two poin...
Python Itertools: For a faster and memory efficient code
The reason python stands out from many other languages is because of it’s simplicity and easy to work with, and the data science community has put the work i...
Learn SQL for Data Science
SQL is important as it is generally one of the first step needed to get your data from a database or data warehouse. SQL is how you query data from databases...
How to Tame a Python
All of those out there who claim that you can learn python in 3,6 or 9 days or 1 month are just fooling around and f** up with your mind. Get it straight you...
Google Takeout: How to download your personal google data
I always wondered how our life would have been if Google hadn’t been there. We depend on most of things from our personal to professional life on Google and ...
How to use AI powered Query in Google Spreadsheet
Dealing with data has always been a daunting task no matter how much data geek you are. Being a Data Scientist the moment I see a new dataset the first thing...
Color Columns, Rows & Cells of Pandas Dataframe
I always wanted to highlight the rows,cells and columns which contains some specific kind of data for my Data Analysis. I wanted to Know which cells contains...
Text Matching: Cosine Similarity
Recently I was working on a project where I have to cluster all the words which have a similar name. For a novice it looks a pretty simple job of using some ...
Read Google Spreadsheet data into Pandas Dataframe
Many a times it happens that we have our data stored on a Google drive and to analyze that data we have to export the data as csv or xlsx and store it on a d...
Google Facets - An Open Source Tool to Analyze & Visualize your data
The major challenge which a data scientists face today is to visualize or understand the data and spot the complexity within the given data set and which res...
Web Scraping Made Easy
Introduction
Pandas in a nutshell
Introduction
Exploratory Analysis of H1B Visa
The H-1B is a non-immigrant visa in the United States, it is designed to bring foreign professionals with college degrees and specialized skills to fill jobs...
Data Visualization with Excel - Part 1
There are abundant tool available for Data Analysis & Visualization but we all are using excel before we know what is data analytics & visualization.
Data Analysis of IMDB Data
[youtube https://www.youtube.com/watch?v=mS3dzczv1ZQ?version=3&rel=1&fs=1&autohide=2&showsearch=0&showinfo=1&iv_load_policy=1&sta...
python
pandas count duplicate rows
DataFrames are a powerful tool for working with data in Python, and Pandas provides a number of ways to count duplicate rows in a DataFrame. In this article...
Pandas value error while merging two dataframes with different data types
If you’re encountering a “value error” while merging Pandas data frames, this article has got you covered. Learn how to troubleshoot and solve common issues ...
How to get True Positive, False Positive, True Negative and False Negative from confusion matrix in scikit learn
In machine learning, we often use classification models to predict the class labels of a set of samples. The predicted labels may or may not match the true ...
Pandas how to use list of values to select rows from a dataframe
In this post we will see how to use a list of values to select rows from a pandas dataframe We will follow these steps to select rows based on list of value...
How to make two plots side by side and create different size subplots in python using matplotlib
In this post we will see how to plot multiple sub-plots in the same figure. We will follow the following steps to create matplotlib subplots: Plot ...
How to prevent xticks overlapping in matplotlib
In this article, we will explore how to prevent overlapping x-axis tick labels. When plotting data in a graph, the labels of the x and y axes may sometimes o...
pandas plot multiple columns bar chart - grouped and stacked chart
In this article, we will see how to create a grouped bar chart and stacked chart using multiple columns of a pandas dataframe Here are the steps that we wil...
How to Append dictionary to a dataframe in pandas
In this article, we will see how to append a dictionary to a dataframe, we could either append dictionary as rows or as a new column to the dataframe Here a...
Pandas drop rows with conditions, with conditions in list and with Index and rows having NaN values
In this article, we will see how to drop rows of a Pandas dataframe based on conditions. Additionally, we will also discuss on how to drop by index, by condi...
Matplotlib xticks every hour and every 15 or 30 minutes starting on the hour
In this post we will see how to use mataplotlib xticks locators and formatters to place the ticks every hour or every 15/30 minutes on a plot. we will use t...
How to implode in pandas - reverse of explode
In this post we are going to see how to perform reverse of explode We will be following the below steps to implode a column in the dataframe: Create a d...
Pandas Group dataframe by time interval - Hour, Month, Year, Number of days and start grouping at specific time
In this post we are going to see how to group a time-series dataframe by time interval such as Hour, Month, Year, Number of days and also see how to use para...
Pandas Merge two dataframes of different length but keep all rows in output data frame
In this post we will see how to merge two dataframes of unequal length on a common column or index and get all the rows from either or both the dataframes in...
Python Plotting latitude and longitude from csv on map using basemap, folium, geopandas and plotly
In this article, we will see how to plot latititude, longitude from csv using Python. Here are the most popular python libraries to plot geo data on a map. ...
Pandas rolling apply custom functions and statistical operations such as rolling mean(), max(), count(), sum(), median() agg()
Rolling window calculations are provided by Pandas rolling() function. The rolling() function is commonly used in finance, economics, and science. It is uti...
Pandas combine two text columns
In this post we will learn how to concatenate two or more string columns of a dataframe. You can use either + operator or the str.cat() function to combine t...
Pandas convert column into rows and transpose rows to column
In this post we will see how to convert the column into rows, rows into columns, transpose one column into multiple columns and how to Pivot/Unpivot the data...
Numpy get ith column and specific column and row data from an array
We want to select specific column and rows in a numpy array
Pandas select rows between two dates
We want to select/filter rows between two dates of a dataframe which has a date as column/index
Numpy to generate random number between 0 and 1
In this post we will discuss how to quickly generate random numbers and float between 0 and 1 or between a range using numpy.
Pandas merge on Index and merge dataframe on Index & column
In this post we will discuss how to merge two dataframes either on their Index or on Index & column, Pandas has a merge API with lot of parameters to mak...
Pandas split string into columns and split string with multiple delimiters
In this post we will discuss how to split a dataframe string into multiple columns and also split string with single and multiple delimiters, The most useful...
Python Dictionary Append
In this post we will discuss how to append a new key-value pair or update an existing key value in a dictionary. There are two ways you can append or update ...
python lambda if, else & elif with multiple conditions and filter list with conditions in lambda
lambda is an one-liner python functions used to quickly built a function with ease, In this post we would see how to use if-else or multiple conditions to ev...
Excel Libraries in Python
There are so many libraries available in Python language that could help to handle the spreadsheets and let you edit, modify, run formulas and help in data a...
Pandas extract numbers and floats from string
We want to extract the numbers, float and multiple numbers from a string column in a dataframe.
Pandas Add month, days, hours, minutes, Bday (business days) and year to date and also find out first day of Month and last day of month
We want to add Month, Days, Hours, Minutes or Year offset to a date column in a dataframe and also like to see how to find out the MonthBegin and MonthEnd da...
Tensorflow available GPU and it’s details
In this post we will see how to find all the available CPU and GPU devices on the host machine and get the device details and other info like it’s Memory usa...
Pandas compare columns in two data frames
We have two dataframes and a common column that we want to compare and find out the matching, missing values and sometimes the difference between the values ...
Pandas select rows and columns in MultiIndex dataframe
We want to select or slice the rows and columns of a MultiIndex dataframe. In this post we will take a look on how to slice the dataframe using the index at ...
Pandas filter dates by month, hour, day and last N days & weeks
We have dataframe with dates or timestamps columns and we would like to filter the rows by Month, Hour, day or by last n days from today’s date.
Pandas get row index that matches specific value in column or condition
We want to get index of rows that matches a specific column value or a condition based on multiple columns. The pandas index attribute get you the index of d...
How to change the date format and work with dates effectively in Matplotlib
We want to plot the data with date on x-axis but the dates are not in the required format for ex: your date column in the data is in YYYY-MM-DD format and yo...
Pandas define your own groupby aggregation functions
The .agg method does aggregation as it sounds and you can pass in the names of aggregation methods, Python aggregations, Numpy reduce functions and you can a...
Matplotlib add value labels on a bar chart using bar_label
We want to add the value labels in a bar chart, which is the value of each label on the top or center of a bar in a plot. We have bar_label() method in matpl...
Pandas create new column based on value in other column with multiple conditions
In this post we will see how to create a new column based on values in other columns with multiple if/else-if conditions. It’s bit straight forward to create...
Pandas sort dataframe by month name
In this post we will see how to sort a dataframe by month name(string) column. It’s bit straight forward to sort a number or string alphabetically using sort...
Pandas groupby and select first, last or nth row in each group
In this post we will see how to select specific rows in each group of pandas groupby object. There are certain cases where we are interested to select only t...
image dataset from directory in Tensorflow
In this post we will create tensorflow dataset(tf.data.Dataset) from MNIST image dataset using image_dataset_from_directory function
How to create tfrecord from image ndarray dataset
In this post we will create tfrecord files from images and the dataset that we will be using is google colab MNIST sample_data for training.
How to filter out the NaN values in a pandas dataframe
As data comes in many shapes and forms, Missing values in pandas are denoted as NaN, It is a special floating-point value. There are also other ways to repre...
How to crop the central region of image using python PIL
There is a crop function in PIL to crop the image if you know the crop area coordinates. How would you crop the central region of Image if you want certain f...
How to return multiple columns using pandas apply
In this article we are going to compare the performance of two approaches i.e. Apply method and Vectorization which can be used to apply a function on datafr...
How to find closest date in dataframe for a given date
In this article we are going to see how to search for the closest date in a dataframe for a given date
Pandas search a string in dataframe across all columns
In this article we are going to learn how to search a string in whole dataframe across multiple columns
Numpy check if elements of array belong to another array
In this article we are going to see what are the convenience functions that can be used to check if elements of a 1D array exists in another 2D array or not
Mataplotlib heatmap for correlation matrix using dataframe
Well most of the articles I found on google search page is about heatmap using seaborn, so this is something that motivated me to write this article about pl...
pandas groupby list of values
In this post, we will learn how to create list of values in a pandas groupby.
Pandas
How to normalize data in pandas crosstab
Pandas Crosstabs
Dataframe groupby date and time
In this post we will see how to group a timeseries dataframe by Year,Month, Weeks or days. Additionally, we will also see how to groupby time objects like ho...
How to create bins in pandas using cut and qcut
Data Binning
Resample and Interpolate time series data
Resampling is a method of frequency conversion of time series data. You can use resample function to convert your data into the desired frequency.
Convert Pandas dataframe to dictionary
In my this blog we will discover what are the different ways to convert a Dataframe into a Python Dictionary or Key/Value Pair
How to use Pandas Count and Value_Counts
Counting number of Values in a Row or Columns is important to know the Frequency or Occurrence of your data.
Parallelize pandas apply using dask and swifter
Using Pandas apply function to run a method along all the rows of a dataframe is slow and if you have a huge data to apply thru a CPU intensive function then...
Sort Pandas Dataframe and Series
Sorting a dataframe by row and column values or by index is easy a task if you know how to do it using the pandas and numpy built-in functions
Pandas dataframe filter with Multiple conditions
Selecting or filtering rows from a dataframe can be sometime tedious if you don’t know the exact methods and how to filter rows with multiple conditions
How to calculate Distance in Python and Pandas using Scipy spatial and distance functions
Working with Geo data is really fun and exciting especially when you clean up all the data and loaded it to a dataframe or to an array. The real works starts...
How to work with JSON in Pandas
JSON is widely used format for storing the data and exchanging. Many of the API’s response are JSON and being light weight it’s used almost everywhere
Pandas apply, map and applymap
In this post we will see how to apply a function along the axis of a dataframe using apply and applymap and how to map the values of a Series from one domain...
How to create dataframe for testing?
Did you ever wanted to create dataframes for testing and find it hard to fill the dataframe with dummy values then DO NOT Worry there are functions that are ...
How to remove duplicate data from python dataframe
Not all data are perfect and we really need to get duplicate data removed from our dataset most of the time. it looks easy to clean up the duplicate data but...
Working with Pandas datetime
In this post we will explore the Pandas datetime methods which can be used instantaneously to work with datetime in Pandas.
How to find Percentage Change in pandas
So you are interested to find the percentage change in your data. Well it is a way to express the change in a variable over the period of time and it is heav...
How to shift a column in Pandas
If you want to shift your columns without re-writing the whole dataframe or you want to subtract the column value with the previous row value or if you want ...
Pandas Groupby Tutorial
Hope if you are reading this post then you know what is groupby in SQL and how it is being used to aggregate the data of the rows with the same value in one ...
Pandas Dataframe Align function
Pandas Align basically helps to align the two dataframes have the same row and/or column configuration and as per their documentation it Align two objects on...
Pandas Transform and Filter
In this blog we will see how to use Transform and filter on a groupby object. We all know about aggregate and apply and their usage in pandas dataframe but h...
Pandas Coalesce - How to Replace NaN values in a dataframe
In this post we will discuss on how to use fillna function and how to use SQL coalesce function with Pandas, For those who doesn’t know about coalesce functi...
Add new rows and columns to Pandas dataframe
We often get into a situation where we want to add a new row or column to a dataframe after creating it. A quick and dirty solution which all of us have trie...
How to create Pandas Pivot Table
Pivot table lets you calculate, summarize and aggregate your data. MS Excel has this feature built-in and provides an elegant way to create the pivot table f...
Pandas How to replace values based on Conditions
Using these methods either you can replace a single cell or all the values of a row and column in a dataframe based on conditions .
Pandas Difference Between two Dataframes
There are often cases where we need to find out the common rows between the two dataframes or find the rows which are in one dataframe and missing from secon...
Pandas how to get a cell value and update it
Accessing a single value or setting up the value of single row is sometime required when we doesn’t want to create a new Dataframe for just updating that sin...
Pandas Map Dictionary values with Dataframe Columns
Pandas has a cool feature called Map which let you create a new column by mapping the dataframe column values with the Dictionary Key. Let’s understand this ...
Pandas Select rows by condition and String Operations
There are instances where we have to select the rows from a Pandas dataframe by multiple conditions. Especially, when we are dealing with the text data then ...
How to Rename Columns in Pandas
Pandas has two ways to rename their Dataframe columns:
Compare two excel files for difference using Python
Comparing two excel spreadsheets and writing difference to a new excel was always a tedious task and Long Ago, I was doing the same thing and the objective t...
Named Entity Recognition: How to Automate Customer Support
Customer support is one of the complex and most important part of any business. This area of business stands to benefit from the machine learning as it is he...
Google Takeout: How to download your personal google data
I always wondered how our life would have been if Google hadn’t been there. We depend on most of things from our personal to professional life on Google and ...
Color Columns, Rows & Cells of Pandas Dataframe
I always wanted to highlight the rows,cells and columns which contains some specific kind of data for my Data Analysis. I wanted to Know which cells contains...
Read Google Spreadsheet data into Pandas Dataframe
Many a times it happens that we have our data stored on a Google drive and to analyze that data we have to export the data as csv or xlsx and store it on a d...
pandas
pandas plot multiple columns bar chart - grouped and stacked chart
In this article, we will see how to create a grouped bar chart and stacked chart using multiple columns of a pandas dataframe Here are the steps that we wil...
How to Append dictionary to a dataframe in pandas
In this article, we will see how to append a dictionary to a dataframe, we could either append dictionary as rows or as a new column to the dataframe Here a...
Pandas drop rows with conditions, with conditions in list and with Index and rows having NaN values
In this article, we will see how to drop rows of a Pandas dataframe based on conditions. Additionally, we will also discuss on how to drop by index, by condi...
How to implode in pandas - reverse of explode
In this post we are going to see how to perform reverse of explode We will be following the below steps to implode a column in the dataframe: Create a d...
Pandas Group dataframe by time interval - Hour, Month, Year, Number of days and start grouping at specific time
In this post we are going to see how to group a time-series dataframe by time interval such as Hour, Month, Year, Number of days and also see how to use para...
Pandas Merge two dataframes of different length but keep all rows in output data frame
In this post we will see how to merge two dataframes of unequal length on a common column or index and get all the rows from either or both the dataframes in...
Pandas rolling apply custom functions and statistical operations such as rolling mean(), max(), count(), sum(), median() agg()
Rolling window calculations are provided by Pandas rolling() function. The rolling() function is commonly used in finance, economics, and science. It is uti...
Pandas combine two text columns
In this post we will learn how to concatenate two or more string columns of a dataframe. You can use either + operator or the str.cat() function to combine t...
Pandas convert column into rows and transpose rows to column
In this post we will see how to convert the column into rows, rows into columns, transpose one column into multiple columns and how to Pivot/Unpivot the data...
Pandas date difference in hours, minutes, days and business days
In this post we want to find the difference(Timedelta) to represent a duration, the difference between two dates or times.
Pandas iterate over rows and update or Update dataframe row values where certain condition is met
We want to iterate over the rows of a dataframe and update the values based on condition. There are three different pandas function available that let you it...
Pandas merge on Index and merge dataframe on Index & column
In this post we will discuss how to merge two dataframes either on their Index or on Index & column, Pandas has a merge API with lot of parameters to mak...
Pandas split string into columns and split string with multiple delimiters
In this post we will discuss how to split a dataframe string into multiple columns and also split string with single and multiple delimiters, The most useful...
Pandas extract numbers and floats from string
We want to extract the numbers, float and multiple numbers from a string column in a dataframe.
Pandas Add month, days, hours, minutes, Bday (business days) and year to date and also find out first day of Month and last day of month
We want to add Month, Days, Hours, Minutes or Year offset to a date column in a dataframe and also like to see how to find out the MonthBegin and MonthEnd da...
Pandas compare columns in two data frames
We have two dataframes and a common column that we want to compare and find out the matching, missing values and sometimes the difference between the values ...
Pandas select rows and columns in MultiIndex dataframe
We want to select or slice the rows and columns of a MultiIndex dataframe. In this post we will take a look on how to slice the dataframe using the index at ...
Pandas filter dates by month, hour, day and last N days & weeks
We have dataframe with dates or timestamps columns and we would like to filter the rows by Month, Hour, day or by last n days from today’s date.
Pandas get row index that matches specific value in column or condition
We want to get index of rows that matches a specific column value or a condition based on multiple columns. The pandas index attribute get you the index of d...
How to change the date format and work with dates effectively in Matplotlib
We want to plot the data with date on x-axis but the dates are not in the required format for ex: your date column in the data is in YYYY-MM-DD format and yo...
Pandas define your own groupby aggregation functions
The .agg method does aggregation as it sounds and you can pass in the names of aggregation methods, Python aggregations, Numpy reduce functions and you can a...
Matplotlib add value labels on a bar chart using bar_label
We want to add the value labels in a bar chart, which is the value of each label on the top or center of a bar in a plot. We have bar_label() method in matpl...
Pandas create new column based on value in other column with multiple conditions
In this post we will see how to create a new column based on values in other columns with multiple if/else-if conditions. It’s bit straight forward to create...
Pandas sort dataframe by month name
In this post we will see how to sort a dataframe by month name(string) column. It’s bit straight forward to sort a number or string alphabetically using sort...
Pandas groupby and select first, last or nth row in each group
In this post we will see how to select specific rows in each group of pandas groupby object. There are certain cases where we are interested to select only t...
How to filter out the NaN values in a pandas dataframe
As data comes in many shapes and forms, Missing values in pandas are denoted as NaN, It is a special floating-point value. There are also other ways to repre...
How to return multiple columns using pandas apply
In this article we are going to compare the performance of two approaches i.e. Apply method and Vectorization which can be used to apply a function on datafr...
How to find closest date in dataframe for a given date
In this article we are going to see how to search for the closest date in a dataframe for a given date
Pandas search a string in dataframe across all columns
In this article we are going to learn how to search a string in whole dataframe across multiple columns
pandas groupby list of values
In this post, we will learn how to create list of values in a pandas groupby.
pandas groupby filter by column values and conditional aggregation
In this post, we will learn how to filter column values in a pandas group by and apply conditional aggregations such as sum, count, average etc.
numpy
Numpy get ith column and specific column and row data from an array
We want to select specific column and rows in a numpy array
Pandas select rows between two dates
We want to select/filter rows between two dates of a dataframe which has a date as column/index
Numpy to generate random number between 0 and 1
In this post we will discuss how to quickly generate random numbers and float between 0 and 1 or between a range using numpy.
How to crop the central region of image using python PIL
There is a crop function in PIL to crop the image if you know the crop area coordinates. How would you crop the central region of Image if you want certain f...
Numpy check if elements of array belong to another array
In this article we are going to see what are the convenience functions that can be used to check if elements of a 1D array exists in another 2D array or not
numpy tile vs repeat
In this post, we will learn what is numpy tile and what’s the difference between numpy tiles and repeat
Index a Numpy Array by another Array
In this post we will see different ways to Index a Numpy array using another array of index
Compare two Numpy arrays for equality
In this post we will compare elements of two arrays for equality. This would be really helpful when you wanted to compare if two similar arrays coming out th...
How to split Numpy Arrays
In this post we will see how to split a 2D numpy array using split, array_split , hsplit, vsplit and dsplit.
Find K smallest and largest values and its indices in a numpy array
To find the maximum and minimum value in an array you can use numpy argmax and argmin function
Concatenating arrays in Numpy
We will be discussing about merging numpy arrays and different functions that are available in the toolbox to perform this job
Reshaping numpy arrays in python
Reshape is an important feature which lets you to change the shape of your array without changing its data
How to work with numpy.where()
What is numpy.where()
matplotlib
pandas count duplicate rows
DataFrames are a powerful tool for working with data in Python, and Pandas provides a number of ways to count duplicate rows in a DataFrame. In this article...
Pandas value error while merging two dataframes with different data types
If you’re encountering a “value error” while merging Pandas data frames, this article has got you covered. Learn how to troubleshoot and solve common issues ...
How to get True Positive, False Positive, True Negative and False Negative from confusion matrix in scikit learn
In machine learning, we often use classification models to predict the class labels of a set of samples. The predicted labels may or may not match the true ...
Pandas how to use list of values to select rows from a dataframe
In this post we will see how to use a list of values to select rows from a pandas dataframe We will follow these steps to select rows based on list of value...
How to make two plots side by side and create different size subplots in python using matplotlib
In this post we will see how to plot multiple sub-plots in the same figure. We will follow the following steps to create matplotlib subplots: Plot ...
How to prevent xticks overlapping in matplotlib
In this article, we will explore how to prevent overlapping x-axis tick labels. When plotting data in a graph, the labels of the x and y axes may sometimes o...
Matplotlib xticks every hour and every 15 or 30 minutes starting on the hour
In this post we will see how to use mataplotlib xticks locators and formatters to place the ticks every hour or every 15/30 minutes on a plot. we will use t...
How to create tfrecord from image ndarray dataset
In this post we will create tfrecord files from images and the dataset that we will be using is google colab MNIST sample_data for training.
Mataplotlib heatmap for correlation matrix using dataframe
Well most of the articles I found on google search page is about heatmap using seaborn, so this is something that motivated me to write this article about pl...
Tutorial
How to create interactive data visualization using plotly
Visualization is the graphical representation of your data and it let you paint your data into a canvas in a way you want to see it. There are lot of amazing...
How to use Regex in Pandas
There are several pandas methods which accept the regex in pandas to find the pattern in a String within a Series or Dataframe object. These methods works on...
Working with Pandas datetime
In this post we will explore the Pandas datetime methods which can be used instantaneously to work with datetime in Pandas.
Building a Web app using Python and Mongodb
Introduction
Dataframe Visualization with Pandas Plot
Visualization has always been challenging task but with the advent of dataframe plot() function it is quite easy to create decent looking plots with your dat...
Pandas Groupby Tutorial
Hope if you are reading this post then you know what is groupby in SQL and how it is being used to aggregate the data of the rows with the same value in one ...
tensorflow
Tensorflow available GPU and it’s details
In this post we will see how to find all the available CPU and GPU devices on the host machine and get the device details and other info like it’s Memory usa...
image dataset from directory in Tensorflow
In this post we will create tensorflow dataset(tf.data.Dataset) from MNIST image dataset using image_dataset_from_directory function
How to create tfrecord from image ndarray dataset
In this post we will create tfrecord files from images and the dataset that we will be using is google colab MNIST sample_data for training.
How to crop the central region of image using python PIL
There is a crop function in PIL to crop the image if you know the crop area coordinates. How would you crop the central region of Image if you want certain f...
How to install tensorflow in a conda environment
In this post, we will learn how to install tensorflow 2 in a conda environment. I would be installing tensorflow in two steps, first we will create a conda p...
Excel
Compare two excel files for difference using Python
Comparing two excel spreadsheets and writing difference to a new excel was always a tedious task and Long Ago, I was doing the same thing and the objective t...
How to use AI powered Query in Google Spreadsheet
Dealing with data has always been a daunting task no matter how much data geek you are. Being a Data Scientist the moment I see a new dataset the first thing...
Data Visualization with Excel - Part 1
There are abundant tool available for Data Analysis & Visualization but we all are using excel before we know what is data analytics & visualization.
google sheet
How to use AI powered Query in Google Spreadsheet
Dealing with data has always been a daunting task no matter how much data geek you are. Being a Data Scientist the moment I see a new dataset the first thing...
Read Google Spreadsheet data into Pandas Dataframe
Many a times it happens that we have our data stored on a Google drive and to analyze that data we have to export the data as csv or xlsx and store it on a d...
Reading Google Sheets data using Python
Google docs are one of the widely used tools across the industry and the spreadsheets are used to store lot of our data, which we would want to access anytim...
Data Privacy
Google Takeout: How to download your personal google data
I always wondered how our life would have been if Google hadn’t been there. We depend on most of things from our personal to professional life on Google and ...
Hey Google! When did I ask you to access my Purchase details?
Source: Google Image
Time Series Analysis
Time Series Analysis and Forecasting with ARIMA
In the previous post we have seen how to visualize a time series data. In this post we will discuss how to do a time series modelling using ARMA and ARIMA mo...
Time Series Data Visualization
Visualizing Time Series data with Python
Resample and Interpolate time series data
Resampling is a method of frequency conversion of time series data. You can use resample function to convert your data into the desired frequency.
Matplotlib
Show images in grid inside jupyter notebook using matplotlib and numpy
In this post, we will see how to display the images in a grid using matplotlib imshow and Image grid. Alternatively, we will see the visualization of image b...
matplotlib fill color between vertical, horizontal lines and polygon
In this post, we will learn how to fill color in the matplotlib charts between two vertical or Horizaontal lines and also inside the Polygons
Matplotlib Scatter Plot Color by Category in Python
Scatter plot are useful to analyze the data typically along two axis for a set of data. It shows the relationship between two sets of data
google-colab
Tensorflow available GPU and it’s details
In this post we will see how to find all the available CPU and GPU devices on the host machine and get the device details and other info like it’s Memory usa...
image dataset from directory in Tensorflow
In this post we will create tensorflow dataset(tf.data.Dataset) from MNIST image dataset using image_dataset_from_directory function
How to crop the central region of image using python PIL
There is a crop function in PIL to crop the image if you know the crop area coordinates. How would you crop the central region of Image if you want certain f...
Uncategorized
Hey Google! When did I ask you to access my Purchase details?
Source: Google Image
Data Visualization with Excel - Part 1
There are abundant tool available for Data Analysis & Visualization but we all are using excel before we know what is data analytics & visualization.
Flask
Python API Documentation using Flask and Swagger
What is Swagger?
Building a Web app using Python and Mongodb
Introduction
Scipy
How to Remove Outliers in Python
Introduction
How to calculate Distance in Python and Pandas using Scipy spatial and distance functions
Working with Geo data is really fun and exciting especially when you clean up all the data and loaded it to a dataframe or to an array. The real works starts...
scikit-learn
Sklearn data Pre-Processing using Standard and Minmax scaler
In Machine learning the variables that are measured at different scales can impact the numerical stability and precision of the estimators
Decision Tree in Sklearn
In this post we are going to see how to build a basic decision tree classifier using scikit-learn package and how to use it for doing multi-class classificat...
groupby
pandas groupby list of values
In this post, we will learn how to create list of values in a pandas groupby.
pandas groupby filter by column values and conditional aggregation
In this post, we will learn how to filter column values in a pandas group by and apply conditional aggregations such as sum, count, average etc.
Visualization
Dataframe Visualization with Pandas Plot
Visualization has always been challenging task but with the advent of dataframe plot() function it is quite easy to create decent looking plots with your dat...
Mongodb
Building a Web app using Python and Mongodb
Introduction
streamlit
Create Interactive Dashboard in Python using Streamlit
Dashboard gives a graphical interface to visualize the key indicators and trends of your data. However, Creating Dashboard is always been a tedious task for ...
Swagger
Python API Documentation using Flask and Swagger
What is Swagger?
Tensors
Vectors, Matrix And Tensors
In this post we will see how large data is stored in multi-dimensional arrays, which is also called as tensors.
NearestNeighbor
Find nearest neighbor using KD Tree
KD Tree is a modified Binary Search Tree(BST) that can perform search in multi-dimensions and that’s why K-dimensional.
Algorithm
Find nearest neighbor using KD Tree
KD Tree is a modified Binary Search Tree(BST) that can perform search in multi-dimensions and that’s why K-dimensional.
Numpy
Show images in grid inside jupyter notebook using matplotlib and numpy
In this post, we will see how to display the images in a grid using matplotlib imshow and Image grid. Alternatively, we will see the visualization of image b...
opencv
Overlay transparent polygons on image using python
In this post, we will see how to overlay a polygon on images using opencv and PIL, the polygon is defined as a series of vertices inside an array and we will...
PIL
Overlay transparent polygons on image using python
In this post, we will see how to overlay a polygon on images using opencv and PIL, the polygon is defined as a series of vertices inside an array and we will...
conda
How to install tensorflow in a conda environment
In this post, we will learn how to install tensorflow 2 in a conda environment. I would be installing tensorflow in two steps, first we will create a conda p...
pil
How to crop the central region of image using python PIL
There is a crop function in PIL to crop the image if you know the crop area coordinates. How would you crop the central region of Image if you want certain f...
seaborn
pandas plot multiple columns bar chart - grouped and stacked chart
In this article, we will see how to create a grouped bar chart and stacked chart using multiple columns of a pandas dataframe Here are the steps that we wil...