Skip to content

Use serialized metadata size to calculate the cache entry cell #1484

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zvonand merged 2 commits into from
Mar 8, 2026
+17 −2
Merged

Use serialized metadata size to calculate the cache entry cell #1484

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1ea2a90
workaround leak on metadata cache size
arthurpassos Mar 6, 2026
20b1d70
Merge branch 'antalya-26.1' into fix_parquet_metadata_cache_max_size
zvonand Mar 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion src/Processors/Formats/Impl/ParquetFileMetaDataCache.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,21 @@

#if USE_PARQUET

#include <parquet/metadata.h>

namespace DB
{

size_t ParquetFileMetaDataWeightFunction::operator()(const parquet::FileMetaData & metadata) const
{
/// TODO fix-me: using the size on disk is not ideal, but it is the simplest and best we can do for now.
Copy link

@ianton-ru ianton-ru Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not ideal? As comment in CacheBase says, functor must return "approximate size" of the data.
Or it is mean that file size is not equal to used disk size and need increase to the file system block size?

Copy link
Collaborator Author

@arthurpassos arthurpassos Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the size matches the size on disk. I say not ideal because imo the ideal would be the in memory data structure sizes

/// this implementation is only used by the v1 reader, which is going to be deprecated and a new implementation for the v3
/// reader will be added in the future.
return metadata.size();
}

ParquetFileMetaDataCache::ParquetFileMetaDataCache()
: CacheBase<String, parquet::FileMetaData>(CurrentMetrics::end(), CurrentMetrics::end(), 0)
: CacheBase<String, parquet::FileMetaData, std::hash<String>, ParquetFileMetaDataWeightFunction>(CurrentMetrics::end(), CurrentMetrics::end(), 0)
{}

ParquetFileMetaDataCache * ParquetFileMetaDataCache::instance()
Expand Down
7 changes: 6 additions & 1 deletion src/Processors/Formats/Impl/ParquetFileMetaDataCache.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,12 @@ class FileMetaData;
namespace DB
{

class ParquetFileMetaDataCache : public CacheBase<String, parquet::FileMetaData>
struct ParquetFileMetaDataWeightFunction
{
size_t operator()(const parquet::FileMetaData & metadata) const;
};

class ParquetFileMetaDataCache : public CacheBase<String, parquet::FileMetaData, std::hash<String>, ParquetFileMetaDataWeightFunction>
{
public:
static ParquetFileMetaDataCache * instance();
Expand Down
Loading