GNODEVAE: a graph-based ODE-VAE enhances clustering for single-cell data.

Background: Single-cell RNA sequencing analysis faces critical challenges including high dimensionality, sparsity, and complex topological relationships between cells. Current methods struggle to simultaneously preserve global structure, model cellular dynamics, and handle technical noise effectively.

Results: We present GNODEVAE, a novel architecture integrating Graph Attention Networks (GAT), Neural Ordinary Differential Equations (NODE), and Variational Autoencoders (VAE) for comprehensive single-cell analysis. Through systematic evaluation across 10 graph convolutional layers, GAT demonstrated optimal performance, achieving average ARI advantages of 0.108 and 0.112 over alternative graph convolutional layers in VGAE and GNODEVAE architectures respectively, along with ASW advantages of 0.047 and 0.098. Extensive comparison across 50 diverse single cell datasets against 18 existing methods demonstrates that GNODEVAE consistently outperforms three major categories of benchmark methods: 8 machine learning dimensionality reduction techniques, 7 deep generative VAE variants, and 3 graph-based and contrastive learning deep predictive models. GNODEVAE achieved average advantages of 0.112 in reconstruction clustering quality (ARI) and 0.113 in clustering geometry quality (ASW) over standard VGAE, with an average ASW advantage of 0.286 over all benchmark methods in clustering geometry quality. In gene dynamics clustering evaluation, GNODEVAE outperformed Diffusion map and Palantir methods across all geometric metrics.

Conclusions: GNODEVAE establishes a robust computational framework that synergistically combines neighborhood-awareness, dynamic modeling, and probabilistic expressiveness for single-cell multi-omics analysis. The consistent superior performance across diverse datasets demonstrates its effectiveness as a versatile tool for cell clustering, dimensionality reduction, and pseudotime trajectory analysis in both scRNA-seq and scATAC-seq data mining.