tip 9

Author: Dennis Bakhuis

B_Pandas_tips/9 - Query or loc.ipynb

@@ -0,0 +1,141 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Pandas tip #9: .loc[] or .query()?\n",
+    "When having tabular data, one of the most common things to do is filter that data. For me in Pandas this would mean using .loc[], .iloc[], or .at[]. These all work great (remember that they return views so be careful with chaining) but there is another kid on the block: .query().\n",
+    "\n",
+    "The .query() is a bit different as it takes a SQL-like where-clause string as an argument. The .loc[] and friends take a boolean mask as arguments. These boolean masks are often small logic operations that need to be calculated separately. When using .query() the mask is combined with the selection process and this has a small performance gain. This is only visible on large DataFrames.\n",
+    "\n",
+    "Another difference between .query() and the others is that it returns a copy and not a view. You have to be careful not to chain .loc[] as these can behave weird (and you see the famous warning). This is solved with copies and therefore, .query() is always save to chain.\n",
+    "\n",
+    "I still use .loc[] all the time, most probably because I am used to writing those but .query() can be shorter and faster in some occasions."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Lets generate some random data:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "\n",
+    "categories = list('ABCD') \n",
+    "n_samples = 100_000\n",
+    "\n",
+    "rng = np.random.default_rng()\n",
+    "df = pd.DataFrame({\n",
+    "    'client_id': rng.integers(0, 1000, size=n_samples),    \n",
+    "    'product_category': rng.choice(categories, size=n_samples),\n",
+    "})"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Lets time the selection using .loc[]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%timeit\n",
+    "df.loc[\n",
+    "    (df.product_category=='A') | (df.product_category == 'C')\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "And now using .query()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%timeit\n",
+    "df.query(\"product_category == 'A' | product_category == 'C'\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can use variables with the older @ notation or the new Python f-strings."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "category_name = 'C'\n",
+    "\n",
+    "cat_c = df.query(\"product_category == @category_name\")\n",
+    "cat_c = df.query(f\"product_category == '{category_name}'\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "If you have any questions, comments, or requests, feel free to [contact me on LinkedIn](https://linkedin.com/in/dennisbakhuis)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}