Sitemap

A list of all the posts and pages found on the site. For you robots out there is an XML version available for digesting as well.

Posts

publications

Image hallucination at different times of day using locally affine model and kNN template matching from time-lapse images.

Published in Proceedings of the Tenth Indian Conference on Computer Vision,Graphics and Image Processing. ACM, 2016., 2016

Image Hallucination has many applications in areas such as image processing, computational photography and image fusion. In this paper, we present an image Hallucination technique based on the template (patch) matching from the database of time lapse images and learned locally affine model. Template based techniques suffer from blocky artifacts. So, we propose two approaches for imposing consistency criteria across neighbouring patches in the form of regularization. We validate our Color transfer technique by hallucinating a variety of natural images at different times the day. We compare the proposed approach with other state of the art techniques of example image based color transfer and show that the images obtained using our approach look more plausible and natural.

Recommended citation: Patel, Nikunj, and Tushar Kataria. "Image hallucination at different times of day using locally affine model and kNN template matching from time-lapse images." In Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image Processing, pp. 1-8. 2016. https://dl.acm.org/doi/10.1145/3009977.3010038

Ringing artifacts in wavelet based image fusion: Analysis, measurement and remedies

Published in Information Fusion, 2020

A thorough analysis of the ringing phenomenon, by experimenting with different types of images and different wavelet families, with varying lengths of filters and varying levels of decomposition is performed to obtain deeper insights of the ringing artifacts. It is experimentally shown that wavelet based fusion results in the modification of the intra- and inter-scale dependencies, with the inter-scale dependency being the dominating factor causing the ringing artifacts. Also, these ringing artifacts are localized in the Fourier domain. Subsequently, a quantitative measure using structural dissimilarity is proposed to measure the ringing artifacts due to wavelet based fusion. Two possible solutions to compensate for the ringing artifacts are then proposed. In the first strategy, a filtering based method is proposed to reduce these ringing artifacts. It takes advantage of the localized nature of the ringing artifacts. Furthermore, the intra- and inter-scale dependencies are modeled using order-zero entropy. A second strategy using the inter-scale dependency is then proposed to reduce the ringing artifacts. Experimental results show that both these methods are able to reduce the ringing artifacts significantly and have further scope for improvement. Another critical finding of this work is selection of the wavelet filter and its levels of decomposition for the process of fusion.

Recommended citation: Vanmali, Ashish V., Tushar Kataria, Samrudha G. Kelkar, and Vikram M. Gadre. "Ringing artifacts in wavelet based image fusion: Analysis, measurement and remedies." Information Fusion 56 (2020): 39-69. https://www.sciencedirect.com/science/article/pii/S1566253517304748

Analysis of Ringing Artifact in Image Fusion Using Directional Wavelet Transforms

Published in IJERT, 2020

In the field of multi-data analysis and fusion, image fusion plays a vital role for many applications. With inventions of new sensors, the demand of high quality image fusion algorithms has seen tremendous growth. Wavelet based fusion is a popular choice for many image fusion algorithms, because of its ability to decouple different features of information. However, it suffers from ringing artifacts generated in the output. This paper presents an analysis of ringing artifacts in application of image fusion using directional wavelets (curvelets, contourlets, non-subsampled contourlets etc.). We compare the performance of various fusion rules for directional wavelets available in literature. The experimental results suggest that the ringing artifacts are present in all types of wavelets with the extent of artifact varying with type of the wavelet, fusion rule used and levels of decomposition.

Recommended citation: Ashish V. Vanmali, Tushar Kataria, Samrudha G. Kelkar, Vikram M. Gadre, “Analysis of Ringing Artifact In Image Fusion Using Directional Wavelet Transforms, ” in Proceedings of Vidyavardhini’s National Conference 2020 (Technical Advancements for Social Upliftments), International Journal of Engineering Research & Technology (IJERT), Vol. 9, Issue 3, Feb 2021, pp. 495-502, ISSN: 2278-0181. https://www.ijert.org/analysis-of-ringing-artifact-in-image-fusion-using-directional-wavelet-transforms

InfoSync: Information Synchronization across Multilingual Semi-structured Tables

Published in Findings of the Association for Computational Linguistics: ACL 2023, 2023

Information Synchronization of semi-structured data across languages is challenging. For example, Wikipedia tables in one language need to be synchronized with others. To address this problem, we introduce a new dataset InfoSync and a two-step method for tabular synchronization. InfoSync contains 100K entity-centric tables (Wikipedia Infoboxes) across 14 languages, of which a subset (~3.5K pairs) are manually annotated. The proposed method includes 1) Information Alignment to map rows and 2) Information Update for updating missing/outdated information for aligned tables across multilingual tables. When evaluated on InfoSync, information alignment achieves an F1 score of 87.91 (en <-> non-en). To evaluate information updation, we perform human-assisted Wikipedia edits on Infoboxes for 532 table pairs. Our approach obtains an acceptance rate of 77.28% on Wikipedia, showing the effectiveness of the proposed method.

Recommended citation: Khincha, Siddharth, Chelsi Jain, Vivek Gupta, Tushar Kataria, and Shuo Zhang. "InfoSync: Information Synchronization across Multilingual Semi-structured Tables." In Findings of the Association for Computational Linguistics: ACL 2023, pp. 2536-2559. 2023. https://aclanthology.org/2023.findings-acl.159/

Automating Ground Truth Annotations for Gland Segmentation Through Immunohistochemistry

Published in Modern Pathology, 2023

The microscopic evaluation of glands in the colon is of utmost importance in the diagnosis of inflammatory bowel disease (IBD) and cancer. When properly trained, deep learning pipelines can provide a systematic, reproducible, and quantitative assessment of disease-related changes in glandular tissue architecture. The training and testing of deep learning models require large amounts of manual annotations, which are difficult, time-consuming, and expensive to obtain. Here, we propose a method for the automated generation of ground truth in digital hematoxylin and eosin (H&E) stained slides using immunohistochemistry (IHC) labels. The image processing pipeline generates annotations of glands in H&E histopathology images from colon biopsies by transfer of gland masks from CK8/18, CDX2, or EpCAM IHC. The IHC gland outlines are transferred to co-registered H&E images for the training of deep learning models. We compare the performance of the deep learning models to manual annotations using an internal held out set of biopsies as well as two public datasets. Our results show that EpCAM IHC provides gland outlines that closely match manual gland annotations (Dice = 0.89) and are robust to damage by inflammation. In addition, we propose a simple data sampling technique that allows models trained on data from several sources to be adapted to a new data source using just a few newly annotated samples. The best performing models achieved average Dice scores of 0.902 and 0.89, respectively, on GLAS and CRAG colon cancer public datasets when trained with only 10% of annotated cases from either public cohort. Altogether, the performances of our models indicate that automated annotations using cell type specific IHC markers can safely replace manual annotations. Automated IHC labels from single institution cohorts can be combined with small numbers of hand-annotated cases from multi-institutional cohorts to train models that generalize well to diverse data sources.

Recommended citation: Kataria, Tushar, Saradha Rajamani, Abdul Bari Ayubi, Mary Bronner, Jolanta Jedrzkiewicz, Beatrice Knudsen, and Shireen Y. Elhabian. "Automating Ground Truth Annotations for Gland Segmentation Through Immunohistochemistry." Modern Pathology (2023): 100331. https://www.sciencedirect.com/science/article/abs/pii/S0893395223002363

ADASSM: Adversarial Data Augmentation in Statistical Shape Models from Images

Published in Shape in Medical Imaging, 2023

Statistical shape models (SSM) have been well-established as an excellent tool for identifying variations in the morphology of anatomy across the underlying population. Shape models use consistent shape representation across all the samples in a given cohort, which helps to compare shapes and identify the variations that can detect pathologies and help in formulating treatment plans. In medical imaging, computing these shape representations from CT/MRI scans requires time-intensive preprocessing operations, including but not limited to anatomy segmentation annotations, registration, and texture denoising. Deep learning models have demonstrated exceptional capabilities in learning shape representations directly from volumetric images, giving rise to highly effective and efficient Image-to-SSM networks. Nevertheless, these models are data-hungry and due to the limited availability of medical data, deep learning models tend to overfit. Offline data augmentation techniques, that use kernel density estimation based (KDE) methods for generating shape-augmented samples, have successfully aided Image-to-SSM networks in achieving comparable accuracy to traditional SSM methods. However, these augmentation methods focus on shape augmentation, whereas deep learning models exhibit image-based texture bias resulting in sub-optimal models. This paper introduces a novel strategy for on-the-fly data augmentation for the Image-to-SSM framework by leveraging data-dependent noise generation or texture augmentation. The proposed framework is trained as an adversary to the Image-to-SSM network, augmenting diverse and challenging noisy samples. Our approach achieves improved accuracy by encouraging the model to focus on the underlying geometry rather than relying solely on pixel values.

Recommended citation: Karanam, M.S.T., Kataria, T., Iyer, K., Elhabian, S.Y. (2023). ADASSM: Adversarial Data Augmentation in Statistical Shape Models from Images. In: Wachinger, C., Paniagua, B., Elhabian, S., Li, J., Egger, J. (eds) Shape in Medical Imaging. ShapeMI 2023. Lecture Notes in Computer Science, vol 14350. Springer, Cham. https://doi.org/10.1007/978-3-031-46914-5_8 https://link.springer.com/chapter/10.1007/978-3-031-46914-5_8

To Pretrain or Not to Pretrain? A Case Study of Domain-Specific Pretraining for Semantic Segmentation in Histopathology

Published in Medical Image Learning with Limited and Noisy Data, 2023

Annotating medical imaging datasets is costly, so fine-tuning (or transfer learning) is the most effective method for digital pathology vision applications such as disease classification and semantic segmentation. However, due to texture bias in models trained on real-world images, transfer learning for histopathology applications might result in underperforming models, which necessitates the need for using unlabeled histopathology data and self-supervised methods to discover domain-specific characteristics. Here, we tested the premise that histopathology-specific pretrained models provide better initializations for pathology vision tasks, i.e., gland and cell segmentation. In this study, we compare the performance of gland and cell segmentation tasks with histopathology domain-specific and non-domain-specific (real-world images) pretrained weights. Moreover, we investigate the dataset size at which domain-specific pretraining produces significant gains in performance. In addition, we investigated whether domain-specific initialization improves the effectiveness of out-of-distribution testing on distinct datasets but the same task. The results indicate that performance gain using domain-specific pretrained weights depends on both the task and the size of the training dataset. In instances with limited dataset sizes, a significant improvement in gland segmentation performance was also observed, whereas models trained on cell segmentation datasets exhibit no improvement

Recommended citation: Kataria, T., Knudsen, B., Elhabian, S. (2023). To Pretrain or Not to Pretrain? A Case Study of Domain-Specific Pretraining for Semantic Segmentation in Histopathology. In: Xue, Z., et al. Medical Image Learning with Limited and Noisy Data. MILLanD 2023. Lecture Notes in Computer Science, vol 14307. Springer, Cham. https://doi.org/10.1007/978-3-031-44917-8_24 https://link.springer.com/chapter/10.1007/978-3-031-44917-8_24

Structural Cycle GAN for Virtual Immunohistochemistry Staining of Gland Markers in the Colon

Published in Machine Learning in Medical Imaging, 2023

With the advent of digital scanners and deep learning, diagnostic operations may move from a microscope to a desktop. Hematoxylin and Eosin (H &E) staining is one of the most frequently used stains for disease analysis, diagnosis, and grading, but pathologists do need different immunohistochemical (IHC) stains to analyze specific structures or cells. Obtaining all of these stains (H &E and different IHCs) on a single specimen is a tedious and time-consuming task. Consequently, virtual staining has emerged as an essential research direction. Here, we propose a novel generative model, Structural Cycle-GAN (SC-GAN), for synthesizing IHC stains from H &E images, and vice versa. Our method expressly incorporates structural information in the form of edges (in addition to color data) and employs attention modules exclusively in the decoder of the proposed generator model. This integration enhances feature localization and preserves contextual information during the generation process. In addition, a structural loss is incorporated to ensure accurate structure alignment between the generated and input markers. To demonstrate the efficacy of the proposed model, experiments are conducted with two IHC markers emphasizing distinct structures of glands in the colon: the nucleus of epithelial cells (CDX2) and the cytoplasm (CK818). Quantitative metrics such as FID and SSIM are frequently used for the analysis of generative models, but they do not correlate explicitly with higher-quality virtual staining results. Therefore, we propose two new quantitative metrics that correlate directly with the virtual staining specificity of IHC markers.

Recommended citation: Dubey, S., Kataria, T., Knudsen, B., Elhabian, S.Y. (2024). Structural Cycle GAN for Virtual Immunohistochemistry Staining of Gland Markers in the Colon. In: Cao, X., Xu, X., Rekik, I., Cui, Z., Ouyang, X. (eds) Machine Learning in Medical Imaging. MLMI 2023. Lecture Notes in Computer Science, vol 14349. Springer, Cham. https://doi.org/10.1007/978-3-031-45676-3_45 https://link.springer.com/chapter/10.1007/978-3-031-45676-3_45

An NSF REU Site Based on Trust and Reproducibility of Intelligent Computation: Experience Report

Published in EduHPC23 Workshop on Education for High Performance Computing, 2023

This paper presents an overview of an NSF Research Experience for Undergraduate (REU) Site on Trust and Reproducibility of Intelligent Computation, delivered by faculty and graduate students in the Kahlert School of Computing at University of Utah. The chosen themes bring together several concerns for the future in producing computational results that can be trusted: secure, reproducible, based on sound algorithmic foundations, and developed in the context of ethical considerations. The research areas represented by student projects include machine learning, high-performance computing, algorithms and applications, computer security, data science, and human-centered computing. In the first four weeks of the program, the entire student cohort spent their mornings in lessons from experts in these crosscutting topics, and used one-of-a-kind research platforms operated by the University of Utah, namely NSF-funded CloudLab and POWDER facilities; reading assignments, quizzes, and hands-on exercises reinforced the lessons. In the subsequent five weeks, lectures were less frequent, as students branched into small groups to develop their research projects. The final week focused on a poster presentation and final report. Through describing our experiences, this program can serve as a model for preparing a future workforce to integrate machine learning into trustworthy and reproducible applications.

Recommended citation: Hall, Mary, Ganesh Gopalakrishnan, Eric Eide, Johanna Cohoon, Jeff Phillips, Mu Zhang, Shireen Elhabian et al. "An NSF REU Site Based on Trust and Reproducibility of Intelligent Computation: Experience Report." In Proceedings of the SC23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, pp. 343-349. 2023. https://dl.acm.org/doi/abs/10.1145/3624062.3624100

Unsupervised Domain Adaptation for Semantic Segmentation Under Target Data Scarcity

Published in IEEE International Symposium on Biomedical Imaging, 2024

Deep learning approaches for semantic segmentation rely on harnessing the power of annotated images to learn features indicative of semantic classes. However, they often fail to generalize when there is a significant domain (i.e., distributional) shift between the training (i.e., source) data and the dataset(s) encountered when deployed (i.e., target), necessitating manual annotations for the target data to achieve acceptable performance. Domain shift is especially important in medical imaging because different image modalities have significant intra- and intersite variations due to protocol and vendor variability. Current techniques are sensitive to hyperparameter tuning and target dataset size. This paper introduces an unsupervised domain adaptation method for semantic segmentation that eliminates the requirement for a substantial quantity of target data samples and their associated annotations. Using kernel density estimation, we match the target data distribution to the source in the feature space. We demonstrate the efficacy of our proposed approach on two datasets, multisite prostate MRI and histopathology images, particularly when the number of unannotated target samples available is limited.

Recommended citation: T. Kataria, B. S. Knudsen and S. Y. Elhabian, "Unsupervised Domain Adaptation for Semantic Segmentation Under Target Data Scarcity," 2024 IEEE International Symposium on Biomedical Imaging (ISBI), Athens, Greece, 2024, pp. 1-5, doi: 10.1109/ISBI56570.2024.10635646. https://ieeexplore.ieee.org/abstract/document/10635646

MASSM: An End-to-End Deep Learning Framework for Multi-Anatomy Statistical Shape Modeling Directly From Images

Published in Shape in Medical Imaging, MICCAI, 2024

Statistical Shape Modeling (SSM) effectively analyzes anatomical variations within populations but is limited by the need for manual localization and segmentation, which relies on scarce medical expertise. Recent advances in deep learning have provided a promising approach that automatically generates statistical representations (as point distribution models or PDMs) from unsegmented images. Once trained, these deep learning-based models eliminate the need for manual segmentation for new subjects. Most deep learning methods still require manual pre-alignment of image volumes and bounding box specification around the target anatomy, leading to a partially manual inference process. Recent approaches facilitate anatomy localization but only estimate population-level statistical representations and cannot directly delineate anatomy in images. Additionally, they are limited to modeling a single anatomy. We introduce MASSM, a novel end-to-end deep learning framework that simultaneously localizes multiple anatomies, estimates population-level statistical representations, and delineates shape representations directly in image space. Our results show that MASSM, which delineates anatomy in image space and handles multiple anatomies through a multitask network, provides superior shape information compared to segmentation networks for medical imaging tasks. Estimating Statistical Shape Models (SSM) is a stronger task than segmentation, as it encodes a more robust statistical prior for the objects to be detected and delineated. MASSM allows for more accurate and comprehensive shape representations, surpassing the capabilities of traditional pixel-wise segmentation.

Recommended citation: Ukey, Janmesh, Tushar Kataria, and Shireen Y. Elhabian. "MASSM: An End-to-End Deep Learning Framework for Multi-Anatomy Statistical Shape Modeling Directly From Images." arXiv preprint arXiv:2403.11008 (2024). https://arxiv.org/abs/2403.11008

Estimation and Analysis of Slice Propagation Uncertainty in 3D Anatomy Segmentation

Published in International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI, 2024

Supervised methods for 3D anatomy segmentation demonstrate superior performance but are often limited by the availability of annotated data. This limitation has led to a growing interest in self-supervised approaches in tandem with the abundance of available unannotated data. Slice propagation has emerged as a self-supervised approach that leverages slice registration as a self-supervised task to achieve full anatomy segmentation with minimal supervision. This approach significantly reduces the need for domain expertise, time, and the cost associated with building fully annotated datasets required for training segmentation networks. However, this shift toward reduced supervision via deterministic networks raises concerns about the trustworthiness and reliability of predictions, especially when compared with more accurate supervised approaches. To address this concern, we propose integrating calibrated uncertainty quantification (UQ) into slice propagation methods, which would provide insights into the model’s predictive reliability and confidence levels. Incorporating uncertainty measures enhances user confidence in self-supervised approaches, thereby improving their practical applicability. We conducted experiments on three datasets for 3D abdominal segmentation using five UQ methods. The results illustrate that incorporating UQ improves not only model trustworthiness but also segmentation accuracy. Furthermore, our analysis reveals various failure modes of slice propagation methods that might not be immediately apparent to end-users. This study opens up new research avenues to improve the accuracy and trustworthiness of slice propagation methods.

Recommended citation: Nihalaani, R., Kataria, T., Adams, J. and Elhabian, S.Y., 2024, October. Estimation and Analysis of Slice Propagation Uncertainty in 3D Anatomy Segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 273-285). Cham: Springer Nature Switzerland. https://link.springer.com/chapter/10.1007/978-3-031-72117-5_26

Evaluating concurrent robustness of language models across diverse challenge sets

Published in Empirical Methods in Natural Language Processing , 2024

Language models, characterized by their black-box nature, often hallucinate and display sensitivity to input perturbations, causing concerns about trust. To enhance trust, it is imperative to gain a comprehensive understanding of the model’s failure modes and develop effective strategies to improve their performance. In this study, we introduce a methodology designed to examine how input perturbations affect language models across various scales, including pre-trained models and large language models (LLMs). Utilizing fine-tuning, we enhance the model’s robustness to input perturbations. Additionally, we investigate whether exposure to one perturbation enhances or diminishes the model’s performance with respect to other perturbations. To address robustness against multiple perturbations, we present three distinct fine-tuning strategies. Furthermore, we broaden the scope of our methodology to encompass large language models (LLMs) by leveraging a chain of thought (CoT) prompting approach augmented with exemplars. We employ the Tabular-NLI task to showcase how our proposed strategies adeptly train a robust model, enabling it to address diverse perturbations while maintaining accuracy on the original dataset.

Recommended citation: Gupta, Vatsal, Pranshu Pandya, Tushar Kataria, Vivek Gupta, and Dan Roth. "Evaluating concurrent robustness of language models across diverse challenge sets." In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 22162-22184. 2024. https://aclanthology.org/2024.emnlp-main.1237/

EFFICIENTMORPH: Parameter-Efficient Transformer-Based Architecture for 3D Image Registration

Published in IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2025

Transformers have emerged as the state-of-the-art architecture in medical image registration, outperforming convolutional neural networks (CNNs) by addressing their limited receptive fields and overcoming gradient instability in deeper models. Despite their success, transformer-based models require substantial resources for training, including data, memory, and computational power, which may restrict their applicability for end users with limited resources. In particular, existing transformer-based 3D image registration architectures face two critical gaps that challenge their efficiency and effectiveness. Firstly, although window-based attention mechanisms reduce the quadratic complexity of full attention by focusing on local regions, they often struggle to effectively integrate both local and global information. Secondly, the granularity of tokenization, a crucial factor in registration accuracy, presents a performance tradeoff: smaller voxel-size tokens enhance detail capture but come with increased computational complexity, higher memory usage, and a greater risk of over-fitting. We present EFFICIENTMORPH, a transformer-based architecture for unsupervised 3D image registration that balances local and global attention in 3D volumes through a plane-based attention mechanism and employs a Hi-Res tokenization strategy with merging operations, thus capturing finer details without compromising computational efficiency. Notably, EFFICIENTMORPH sets a new benchmark for performance on the OASIS dataset with ~16-27x fewer parameters. https://github.com/MedVIC-Lab/Efficient.Morph.Registration

Recommended citation: Aziz, Abu Zahid Bin, Mokshagna Sai Teja Karanam, Tushar Kataria, and Shireen Y. Elhabian. "EFFICIENTMORPH: Parameter-Efficient Transformer-Based Architecture for 3D Image Registration." In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1330-1341. IEEE, 2025. https://ieeexplore.ieee.org/abstract/document/10943319

Ntsebench: Cognitive reasoning benchmark for vision language models

Published in Nations of the Americas Chapter of the Association for Computational Linguistics, 2025

Cognitive textual and visual reasoning tasks, including puzzles, series, and analogies, demand the ability to quickly reason, decipher, and evaluate patterns both textually and spatially. Due to extensive training on vast amounts of human-curated data, large language models (LLMs) and vision language models (VLMs) excel in common-sense reasoning tasks, but still struggle with more complex reasoning that demands deeper cognitive understanding. We introduce NTSEBENCH, a new dataset designed to evaluate cognitive multimodal reasoning and problem-solving skills of large models. The dataset contains 2,728 multiple-choice questions, accompanied by a total of 4,642 images, spanning 26 categories. These questions are drawn from the nationwide NTSE examination in India and feature a mix of visual and textual general aptitude challenges, designed to assess intelligence and critical thinking skills beyond mere rote learning. We establish baselines on the dataset using state-of-the-art LLMs and VLMs. To facilitate a comparison between open-source and propriety models, we propose four distinct modeling strategies to handle different modalities—text and images—in the dataset instances.

Recommended citation: Pandya, Pranshu, Vatsal Gupta, Agney S. Talwarr, Tushar Kataria, Dan Roth, and Vivek Gupta. "Ntsebench: Cognitive reasoning benchmark for vision language models." In Findings of the Association for Computational Linguistics: NAACL 2025, pp. 3680-3708. 2025. https://aclanthology.org/2025.findings-naacl.204/

TransientTables: Evaluating LLMs’ Reasoning on Temporally Evolving Semi-structured Tables

Published in Nations of the Americas Chapter of the Association for Computational Linguistics, 2025

Humans continuously make new discoveries, and understanding temporal sequence of events leading to these breakthroughs is essential for advancing science and society. This ability to reason over time allows us to identify future steps and understand the effects of financial and political decisions on our lives. However, large language models (LLMs) are typically trained on static datasets, limiting their ability to perform effective temporal reasoning. To assess the temporal reasoning capabilities of LLMs, we present the TRANSIENTTABLES dataset, which comprises 3,971 questions derived from over 14,000 tables, spanning 1,238 entities across multiple time periods. We introduce a template-based question-generation pipeline that harnesses LLMs to refine both templates and questions. Additionally, we establish baseline results using state-of-the-art LLMs to create a benchmark. We also introduce novel modeling strategies centered around task decomposition, enhancing LLM performance.

Recommended citation: Shankarampeta, Abhilash, Harsh Mahajan, Tushar Kataria, Dan Roth, and Vivek Gupta. "TransientTables: Evaluating LLMs Reasoning on Temporally Evolving Semi-structured Tables." arXiv preprint arXiv:2504.01879 (2025). https://aclanthology.org/2025.naacl-long.332/

Leveraging LLM For Synchronizing Information Across Multilingual Tables

Published in Nations of the Americas Chapter of the Association for Computational Linguistics, 2025

Recommended citation: Khincha, Siddharth, Tushar Kataria, Ankita Anand, Dan Roth, and Vivek Gupta. "Leveraging LLM For Synchronizing Information Across Multilingual Tables." arXiv preprint arXiv:2504.02559 (2025). https://aclanthology.org/2025.naacl-long.329/

StainDiffuser: MultiTask Dual Diffusion Model for Virtual Staining

Published in arXiv preprint arXiv:2403.11340, 2025

Hematoxylin and Eosin (H&E) staining is widely regarded as the standard in pathology for diagnosing diseases and tracking tumor recurrence. While H&E staining shows tissue structures, it lacks the ability to reveal specific proteins that are associated with disease severity and treatment response. Immunohistochemical (IHC) stains use antibodies to highlight the expression of these proteins on their respective cell types, improving diagnostic accuracy, and assisting with drug selection for treatment. Despite their value, IHC stains require additional time and resources, limiting their utilization in some clinical settings. Recent advances in deep learning have positioned Image-to-Image (I2I) translation as a computational, cost-effective alternative for IHC. I2I generates high fidelity stain transformations digitally, potentially replacing manual staining in IHC. Diffusion models, the current state of the art in image generation and conditional tasks, are particularly well suited for virtual IHC due to their ability to produce high quality images and resilience to mode collapse. However, these models require extensive and diverse datasets (often millions of samples) to achieve a robust performance, a challenge in virtual staining applications where only thousands of samples are typically available. Inspired by the success of multitask deep learning models in scenarios with limited data, we introduce STAINDIFFUSER, a novel multitask diffusion architecture tailored to virtual staining that achieves convergence with smaller datasets. STAINDIFFUSER simultaneously trains two diffusion processes: (a) generating cell specific IHC stains from H&E images and (b) performing H&E based cell segmentation, utilizing coarse segmentation labels exclusively during training. STAINDIFFUSER generates high-quality virtual stains for two markers, outperforming over twenty I2I baselines.

Recommended citation: Tushar Kataria, Beatrice Knudsen, Shireen Y. Elhabian. "StainDiffuser: MultiTask Dual Diffusion Model for Virtual Staining." arXiv preprint arXiv:2403.11340 (2025).

Building Trust in Virtual Immunohistochemistry: Automated Assessment of Image Quality

Published in arXiv preprint arXiv:2511.04615, 2025

Deep learning models can generate virtual immunohistochemistry (IHC) stains from hematoxylin and eosin (H&E) images, offering a scalable and low-cost alternative to laboratory IHC. However, reliable evaluation of image quality remains a challenge as current texture- and distribution-based metrics quantify image fidelity rather than the accuracy of IHC staining. Here, we introduce an automated and accuracy grounded framework to determine image quality across sixteen paired or unpaired image translation models. Using color deconvolution, we generate masks of pixels stained brown (i.e., IHC-positive) as predicted by each virtual IHC model. We use the segmented masks of real and virtual IHC to compute stain accuracy metrics (Dice, IoU, Hausdorff distance) that directly quantify correct pixel - level labeling without needing expert manual annotations. Our results demonstrate that conventional image fidelity metrics, including Frechet Inception Distance (FID), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM), correlate poorly with stain accuracy and pathologist assessment. Paired models such as PyramidPix2Pix and AdaptiveNCE achieve the highest stain accuracy, whereas unpaired diffusion- and GAN-based models are less reliable in providing accurate IHC positive pixel labels. Moreover, whole-slide images (WSI) reveal performance declines that are invisible in patch-based evaluations, emphasizing the need for WSI-level benchmarks. Together, this framework defines a reproducible approach for assessing the quality of virtual IHC models, a critical step to accelerate translation towards routine use by pathologists.

Recommended citation: Tushar Kataria, Shikha Dubey, Mary Bronner, Jolanta Jedrzkiewicz, Ben J. Brintz, Shireen Y. Elhabian*, Beatrice S. Knudsen*. "Building Trust in Virtual Immunohistochemistry: Automated Assessment of Image Quality." arXiv preprint arXiv:2511.04615 (2025).

MapVerse: A Benchmark for Geospatial Question Answering on Diverse Real-World Maps

Published in IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2026

Maps are powerful carriers of structured and contextual knowledge, encompassing geography, demographics, infrastructure, and environmental patterns. Reasoning over such knowledge requires models to integrate spatial relationships, visual cues, real-world context, and domain-specific expertise-capabilities that current large language models (LLMs) and vision-language models (VLMs) still struggle to exhibit consistently. Yet, datasets used to benchmark VLMs on map-based reasoning remain narrow in scope, restricted to specific domains, and heavily reliant on artificially generated content (outputs from LLMs or pipeline-based methods), offering limited depth for evaluating genuine geospatial reasoning. To address this gap, we present MapVerse, a large-scale benchmark built on real-world maps. It comprises 11,837 human-authored question-answer pairs across 1,025 maps, spanning ten diverse map categories and multiple question categories for each. The dataset provides a rich setting for evaluating map reading, interpretation, and multimodal reasoning. We evaluate ten state-of-the-art models against our benchmark to establish baselines and quantify reasoning gaps. Beyond overall performance, we conduct fine-grained categorical analyses to assess model inference across multiple dimensions and investigate the visual factors shaping reasoning outcomes. Our findings reveal that while current VLMs perform competitively on classification-style tasks, both open- and closed-source models fall short on advanced tasks requiring complex spatial reasoning.

Recommended citation: Sharat Bhat, Harshita Khandelwal, Tushar Kataria, Vivek Gupta. "MapVerse: A Benchmark for Geospatial Question Answering on Diverse Real-World Maps." In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2026.

MedConcept: Unsupervised Concept Discovery for Interpretability in Medical VLMs

Published in International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2026

While medical Vision-Language models (VLMs) achieve strong performance on tasks such as tumor or organ segmentation and diagnosis prediction, their opaque latent representations limit clinical trust and the ability to explain predictions. Interpretability of these multimodal representations are therefore essential for the trustworthy clinical deployment of pretrained medical VLMs. However, current interpretability methods, such as gradient- or attention-based visualizations, are often limited to specific tasks such as classification. Moreover, they do not provide concept-level explanations derived from shared pretrained representations that can be reused across downstream tasks. We introduce MedConcept, a framework that uncovers latent medical concepts in a fully unsupervised manner and grounds them in clinically verifiable textual semantics. MedConcept identifies sparse neuron-level concept activations from pretrained VLM representations and translates them into pseudo-report-style summaries, enabling physician-level inspection of internal model reasoning. To address the lack of quantitative evaluation in concept-based interpretability, we introduce a quantitative semantic verification protocol that leverages an independent pretrained medical LLM as a frozen external evaluator to assess concept alignment with radiology reports. We define three concept scores, Aligned, Unaligned, and Uncertain, to quantify semantic support, contradiction, or ambiguity relative to radiology reports and use them exclusively for post hoc evaluation. These scores provide a quantitative baseline for assessing interpretability in medical VLMs. All codes, prompt and data to be released on acceptance. Ke

Recommended citation: Md Rakibul Haque, KM Arefeen Sultan, Tushar Kataria, Shireen Elhabian. "MedConcept: Unsupervised Concept Discovery for Interpretability in Medical VLMs." In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2026.

IMPLICITSTAINER: Resolution Agnostic Data-Efficient Virtual Staining Using Neural Implicit Functions

Published in arXiv preprint arXiv:2505.09831, 2026

Hematoxylin and eosin (H&E)-stained slides are central to cancer diagnosis and monitoring, visualizing tissue architecture and cellular morphology. However, H&E lacks the molecular specificity needed to distinguish cell states and functional activation. Antibody-based stains, such as immunohistochemistry (IHC), are therefore required to identify specific phenotypes (e.g., CD3$^+$ T cells or HER2-positive tumor cells) but are costly, time-consuming, and not universally available. Deep learning-based image translation methods, often termed virtual staining, offer a complementary alternative by generating virtual immunostains directly from H&E images. Most existing virtual staining methods are patch-based and operate at fixed resolutions, often requiring large datasets and additional post-hoc super-resolution models to generate high-resolution images. Furthermore, GAN- and diffusion-based approaches introduce stochasticity into generated stains which, although beneficial for visual realism in natural images, can lead to hallucinations and structural distortions that affect the accuracy and reliability required for clinical use. We propose IMPLICITSTAINER, a deterministic framework that reformulates virtual staining as a continuous pixel-level translation problem. In contrast to existing patch-based approaches, IMPLICITSTAINER formulates image translation as a continuous spatial mapping using neural implicit deep learning models. Each target-domain (IHC) pixel is predicted from a high-dimensional embedding of the corresponding source-domain H&E pixel, its local spatial neighborhood, and explicit coordinate information. IMPLICITSTAINER enables resolution-agnostic inference, improves robustness in low-data regimes, and yields deterministic, reproducible outputs. Across more than twenty baselines, IMPLICITSTAINER achieves SOTA performance on virtual staining tasks, including IHC and mIF.

Recommended citation: Tushar Kataria, Beatrice Knudsen, Shireen Y. Elhabian. "IMPLICITSTAINER: Resolution Agnostic Data-Efficient Virtual Staining Using Neural Implicit Functions." arXiv preprint arXiv:2505.09831 (2026).

On the Viability of Semi-Supervised Segmentation Methods for Statistical Shape Modeling

Published in Shape in Medical Imaging (ShapeMI), MICCAI, 2026

Statistical Shape Models (SSMs) excel at identifying population level anatomical variations, which is at the core of various clinical and biomedical applications, including morphology-based diagnostics and surgical planning. However, the effectiveness of SSMs is often constrained by the necessity for expert-driven manual segmentation, a time-intensive and expensive process that restricts their broader utility. While deep learning approaches offer a potential workaround by directly estimating SSMs from unsegmented images, they merely shift the burden. Although these models do not require segmentation during deployment, they still fail to address the challenge of acquiring the manual annotations needed for training, particularly in resource-limited settings. Semi-supervised models for anatomy segmentation present a logical solution to the annotation burden. However, the lack of established guidelines leaves end-users uncertain about the actual effectiveness of these approaches for the downstream task of constructing SSMs. In this study, we bridge this gap by systematically evaluating semi-supervised methods as viable alternatives to manual segmentation. By applying these methods under low-annotation settings and utilizing the predicted segmentations for SSM generation, we establish a comprehensive new performance benchmark. Our findings reveal a clear divide in performance: while certain methods yield noisy segmentations that degrade SSM quality, others accurately capture the population’s modes of variation comparable to those obtained from manual-segmentation SSMs despite a 60-80% reduction in manual annotation requirements.

Recommended citation: Asma Khan*, Tushar Kataria*, Janmesh Ukey, Shireen Y. Elhabian. "On the Viability of Semi-Supervised Segmentation Methods for Statistical Shape Modeling." In Shape in Medical Imaging (ShapeMI), MICCAI 2026.

Inference-Time Orthogonal Seeding Enables Geometry-Aligned 3D Organ Segmentation for Slice-Propagation Methods

Published in Machine Learning in Medical Imaging (MLMI), MICCAI, 2026

Dense voxel-level annotation remains a major bottleneck in 3D medical image segmentation. Single-slice propagation methods such as Sli2Vol reduce this burden by propagating one annotated seed slice through a volume using label-free registration. However, axial-only propagation accumulates errors with distance from the seed, especially in surface-distance metrics, because it ignores coronal and sagittal evidence and therefore underuses the 3D information available in CT/MRI volumes. To better leverage volumetric geometry, we study how key training and inference choices affect slice-propagation models, including single-axis versus multi-axis label-free registration, single-seed versus multi-seed propagation, and orthogonal seed configurations. Instead of propagating from a single axial seed, we use three orthogonal seeds—one axial, one coronal, and one sagittal—and fuse their propagated labels with a simple label-free rule. Our results show that the training paradigm has limited impact: an axially trained network applied to off-axis seeds captures nearly all the improvement, while explicit three-axis training adds little. Instead, performance is driven by inference-time seed geometry, especially orthogonality rather than the number of annotated slices, as a budget-matched three-axial control provides no benefit and can even degrade performance. On a multi-organ CT cohort, orthogonal seeding with the axial Sli2Vol backbone improves Dice by 21.9%, Normalized Surface Dice by 25.5%, and reduces Average Hausdorff Distance by 53.5% over the single-axis baseline.

Recommended citation: Md Rakibul Haque, Tushar Kataria, Shireen Y. Elhabian. "Inference-Time Orthogonal Seeding Enables Geometry-Aligned 3D Organ Segmentation for Slice-Propagation Methods." In Machine Learning in Medical Imaging (MLMI), MICCAI 2026.

BoundarySeg: Boundary-Aware Multi-Task Learning for Semi-Supervised 3D Medical Image Segmentation with Limited Unlabeled Data

Published in arXiv preprint arXiv:2505.09829, 2026

Data scarcity is a primary bottleneck in 3D medical image segmentation: expert voxel-level annotations are expensive, and privacy regulations often severely restrict access to unannotated clinical volumes. While semi-supervised learning (SSL) aims to alleviate annotation burdens, existing paradigms implicitly rely on large pools of unlabeled data ($U$) relative to labeled data ($L$), assuming $U \gg L$ to filter noise and enforce consistency. When restricted to the data-constrained regime where unlabeled data are limited ($U \le L$), standard SSL methods suffer from severe pseudo-label degeneracy, performance degradation, and high seed-to-seed variance. In this work, we propose BoundarySeg, a unified multi-task semi-supervised framework specifically designed for the limited unlabeled data regime ($U \le L$). BoundarySeg formulates volumetric segmentation as a joint learning task that predicts whole-organ geometry alongside auxiliary organ boundaries. Rather than relying on large cohorts to average out pseudo-label noise, BoundarySeg employs the auxiliary boundary branch as an internal geometric anchor. Coupled with uncertainty-weighted mean-teacher consistency across both tasks alongside axis-flip equivariance and copy-paste regularization, this joint supervision stabilizes spatial representations and prevents segmentation collapse when $U \le L$. Across multiple 3D segmentation datasets, BoundarySeg maintains high performance stability as unlabeled pool size shrinks, consistently outperforming state-of-the-art semi-supervised frameworks in low-data settings. \href{https://github.com/tushaarkataria/BoundarySeg}{https://github.com/tushaarkataria/BoundarySeg}

Recommended citation: Tushar Kataria, Shireen Y. Elhabian. "BoundarySeg: Boundary-Aware Multi-Task Learning for Semi-Supervised 3D Medical Image Segmentation with Limited Unlabeled Data." arXiv preprint arXiv:2505.09829 (2026).

MorphoFlow: Sparse-Supervised Generative Shape Modeling with Adaptive Latent Relevance

Published in IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2026

Statistical shape modeling (SSM) is central to population level analysis of anatomical variability, yet most existing approaches rely on densely annotated segmentations and fixed latent representations. These requirements limit scalability and reduce flexibility when modeling complex anatomical variation. We introduce MorphoFlow, a sparse supervised generative shape modeling framework that learns compact probabilistic shape representations directly from sparse surface annotations. MorphoFlow integrates neural implicit shape representations with an autodecoder formulation and autoregressive normalizing flows to learn an expressive probabilistic density over the latent shape space. The neural implicit representation enables resolution-agnostic modeling of 3D anatomy, while the autodecoder formulation supports direct optimization of per-instance latent codes under sparse supervision. The autoregressive flow captures the distribution of latent anatomical variability providing a tractable, likelihood-based generative model of shapes. To promote compact and structured latent representations, we incorporate adaptive latent relevance weighting through sparsity-inducing priors, enabling the model to regulate the contribution of individual latent dimensions according to their relevance to the underlying anatomical variation while preserving generative expressivity. The resulting latent space supports uncertainty quantification and anatomically plausible shape synthesis without manual latent dimensionality tuning. Evaluation on publicly available lumbar vertebrae and femur datasets demonstrates accurate high-resolution reconstruction from sparse inputs and recovery of structured modes of anatomical variation consistent with population level trends.

Recommended citation: Mokshagna Sai Teja Karanam, Tushar Kataria, Shireen Elhabian. "MorphoFlow: Sparse-Supervised Generative Shape Modeling with Adaptive Latent Relevance." In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2027.

FLASHSWIN: Unlocking Large Windows and Dense Tokens in Swin Vision Transformers with Memory Efficient Attention

Published in arXiv preprint arXiv:2610.04664, 2026

High-resolution vision backbones have long been forced to trade away local token density to afford larger receptive fields. Hierarchical Swin transformers impose this compromise because standard windowed attention materializes an $M^2\times M^2$ score matrix per window, incurring $O(M^4)$ memory as windows or token grids grow. Furthermore, Swin adds a learned relative-position bias elementwise to attention scores, requiring full materialization of the score matrix and its gradient. This keeps Swin and SwinV2 trapped in a small-window($M=8,16$), coarse-token regime with patch size $4\times4$ ($p=4$), limiting performance for fine-grained tasks. We introduce FLASHSWIN, which replaces standard windowed attention with a FlashAttention implementation that computes exact softmax attention without materializing the score matrix, reducing per-window memory from $O(M^4)$ to $O(M^2)$. This enables higher token density and larger receptive fields without inflating memory overhead. Training memory is flat across window sizes: at a $32\times32$ window, FLASHSWIN-T requires only $12.4$\,GB, unchanged from $8\times8$, compared to $70/90$\,GB for SwinV2/V1-T. However, applying FlashAttention directly to Swin creates a trade-off: bypassing the score matrix precludes Swin’s additive relative-position bias, forfeiting spatial information in exchange for memory efficiency. FLASHSWIN restores position information as window-local learnable 2D RoPE, making large windows and dense token grids both affordable and accurate. At matched scale, FLASHSWIN-T outperforms Swin variants. With dense tokens and wide windows ($p=2,M=32$), the same Tiny model reaches $84.1\%$ ImageNet-1K, $44.1$ COCO box AP, and $47.28$ ADE20K mIoU—gains of $+1.3$, $+5.1$, and $+1.82$ over SwinV2-T at $M=16$, respectively. At fixed $M=32$, halving the patch size yields roughly $3\times$ larger gains in boundary quality than in mIoU.

Recommended citation: Tushar Kataria, Gerald Sabin, Ponnuswamy Sadayappan, Shireen Y. Elhabian. "FLASHSWIN: Unlocking Large Windows and Dense Tokens in Swin Vision Transformers with Memory Efficient Attention." arXiv preprint arXiv:2610.04664 (2026).

Tushar Kataria

Sitemap

Pages

Posts

publications

teaching