A Guide to Organized Data with Pandas DataFrames
This post is part 2/3 in the enDAQ in Python Series
Organized data can help save time and effort by filtering and manipulating large datasets easily. If you are new to Python and enDAQ recordings, my intention with this video and post is to guide your journey towards organized data with Pandas DataFrames.
While enDAQ Lab software offers visualization and access to sensor specifications, extracting data and obtaining sensor information can be a laborious and messy process. Manually exporting to CSV or navigating through various menus becomes time-consuming, especially when dealing with multiple devices, recordings, or even sensor channels.
Fortunately, Pandas DataFrames can alleviate these challenges. With our enDAQ open-source Python libraries, you can effortlessly create DataFrames to organize and store your signal data and/or your sensor and channel information. These DataFrames can be easily filtered, combined, and manipulated to consolidate all the necessary information in a concise and organized manner. By taking advantage of Python environments and Pandas’ efficient handling of large datasets, using DataFrames enables faster processing speeds and saves valuable computation and analysis time on your projects!
In this post, I'll cover:
Recap to Get Started
To start organizing information from your enDAQ sensor's IDE files, we’ll use our endaq.ide library again to grab our file and collect some sample channels to work with.
To understand the ide and channels objects, check out the previous video and blog: Introduction to IDEs for Data Acquisition in Python.
Importing IDE Data into a DataFrame
The endaq.ide.to_pandas() function allows you to generate a table of information for either a channel OR a subchannel, the time_mode can be changed to ‘seconds,’ ‘datetime,’ or ‘timedelta.’ The DataFrame returned by default has the indices set to the timestamp, this can be changed using Pandas’ DataFrame.reset_index() method. This might be useful if you’re looking to combine your DataFrame with more information that does not line up with the same timestamp index.
To see other possibilities with your DataFrames, look through the Pandas Documentation online.
The name of the subchannels can be used to grab their respective columns from this DataFrame, and can be done in the same format as keying a dictionary. The session length mentioned in the previous video should match up with the number of events (rows) recorded in this DataFrame! Pandas offers a few ways to save a DataFrame to a file that can stay on your computer, for example creating an excel file in the working directory with .to_excel( ).
Tables of Channels & Basic Info
Another DataFrame of information we can grab is a table of specifications about our channels from an IDE file. This can be narrowed via the start end end time to be looked at, or even the measurement_type which can be provided a string or a measurement object from endaq.ide.measurement (in our last video).enDAQ in Python Series
This post is part of the enDAQ in Python Series. Feel free to check out my other related posts;
- Intro to IDE's, Channels, and Subchannels - Part 1/3
- Pandas DataFrames - Part 2/3
- Applying Transforms to Your Data - Part 3/3
- Recording Data Hands-Free
