Write a simple parquet file with a categorical data type

Oct 13, 2025 by magicalceylontours.com 75 views

Introduction

Polars is a fast and efficient library for data manipulation and analysis. It provides a wide range of features, including support for various data types, such as categorical and enum datatypes. However, when trying to parse dataframes with these datatypes using Nu, a command-line shell and scripting language, an error occurs. In this article, we will explore this issue and provide a solution.

Describe the Bug

The bug occurs when trying to collect polars dataframes that have categorical datatypes. The error message indicates that the value "cat" is not supported in Nu.

Step 1: Reproduce the Bug

To reproduce the bug, follow these steps:

Write a simple parquet file with a categorical data type:

import polars as pl

pl.DataFrame({"foo": ['a', 'b']}, schema_overrides={'foo': pl.Categorical}).write_parquet('example.parquet')

This code creates a dataframe with a categorical column "foo" and writes it to a parquet file named "example.parquet".

Attempt to read the file:

❯ polars open example.parquet | polars collect
Error:
  × Error creating Dataframe
  help: Value not supported in nushell: cat

This command attempts to read the parquet file and collect the data into a dataframe. However, it results in an error message indicating that the value "cat" is not supported in Nu.

Show that the file is valid by casting to string:

❯ polars open example.parquet | polars cast str foo | polars collect
╭───┬─────╮
│ # │ foo │
├───┼─────┤
│ 0 │ a   │
│ 1 │ b   │
╰───┴─────╯

This command casts the categorical column "foo" to a string and collects the data into a dataframe. The resulting dataframe is valid, indicating that the issue is specific to categorical datatypes.

Expected Behavior

The expected behavior is that Nu dataframes should be able to handle categorical data types and other types that are part of the parquet standard/native polars data types.

Configuration

The following configuration was used to reproduce the bug:

key	value
version	0.101.0
major	0
minor	101
patch	0
branch
commit_hash
build_os	macos-aarch64
build_target	aarch64-apple-darwin
rust_version	rustc 1.83.0 (90b35a623 2024-11-26) (Homebrew)
cargo_version	cargo 1.83.0
build_time	2024-12-22 14:10:19 +00:00
build_rust_channel	release
allocator	mimalloc
features	default, sqlite, trash
installed_plugins	polars 0.101.0

Conclusion

In conclusion, the bug occurs when trying to collect polars dataframes with categorical datatypes using Nu. The error message indicates that the value "cat" is not supported in Nu. However, casting the categorical column to a string results in a valid dataframe, indicating that the issue is specific to categorical datatypes. The expected behavior is that Nu dataframes should be able to handle categorical data types and other types that are part of the parquet standard/native polars data types.

Workaround

A workaround for this issue is to cast the categorical column to a string before collecting the data into a dataframe. This can be done using the polars cast function, as shown in the example above.

Future Development

To resolve this issue, the Nu team can consider adding support for categorical datatypes in the future. This would involve implementing the necessary logic to handle categorical data types and other types that are part of the parquet standard/native polars data types.

References

Introduction

In our previous article, we explored the issue of categorical/enum datatypes not being parsed in Polars when using Nu. In this article, we will provide a Q&A section to address some common questions related to this issue.

Q: What is the cause of this issue?

A: The cause of this issue is that Nu does not currently support categorical datatypes. This is a limitation of the Nu language and is not specific to Polars.

Q: Why is this issue specific to categorical datatypes?

A: This issue is specific to categorical datatypes because they are not a native data type in Nu. Categorical datatypes are a feature of Polars and are not supported by Nu.

Q: Can I use a workaround to resolve this issue?

A: Yes, you can use a workaround to resolve this issue. One possible workaround is to cast the categorical column to a string before collecting the data into a dataframe. This can be done using the polars cast function.

Q: Will this issue be resolved in future versions of Nu?

A: It is possible that this issue will be resolved in future versions of Nu. However, there is no guarantee that this will happen. The Nu team is constantly working to improve the language and add new features, but it is difficult to predict when specific issues will be resolved.

Q: Can I report this issue to the Nu team?

A: Yes, you can report this issue to the Nu team. The Nu team is actively seeking feedback from users and is committed to resolving issues as quickly as possible. You can report this issue by opening a new issue on the Nu GitHub repository.

Q: What are some alternative solutions to using categorical datatypes in Nu?

A: There are several alternative solutions to using categorical datatypes in Nu. One possible solution is to use a different data type, such as a string or an integer. Another possible solution is to use a different library or framework that supports categorical datatypes.

Q: Can I use Polars with other languages that support categorical datatypes?

A: Yes, you can use Polars with other languages that support categorical datatypes. Polars is a language-agnostic library and can be used with a variety of languages, including Python, Rust, and Julia.

Q: What are some best practices for working with categorical datatypes in Polars?

A: Here are some best practices for working with categorical datatypes in Polars:

Use the polars cast function to cast categorical columns to a string or other data type as needed.
Use the polars categorize function to create categorical columns from string or other data types.
Use the polars groupby function to group data by categorical columns.
Use the polars pivot function to pivot data based on categorical columns.

Conclusion

In conclusion, the issue of categorical/enum datatypes not being parsed in Polars when using Nu is a limitation of the Nu language. However, there are workarounds and alternative solutions available. By following best practices and using the right tools and techniques, you can work effectively with categorical datatypes in Polars.