One of the major health problems, brain stroke, is listed as a top cause of mortality and disability. Hence, it is crucial that the diagnosis of brain stroke, as depicted in CT images, is done correctly and as early as possible. In this work, we are proposing a novel framework called "Dual Branch CNNTransformer." This framework combines the power of CNNs with the capabilities of the Transformer architecture. The proposed system is composed of a Vision Transformer and a Graph Attention Network, which are combined using a novel "Cross-Attention" fusion mechanism. This ensures that the proposed system has the discriminative power to detect and classify brain stroke accurately, as it can capture fine spatial details as well as region-region dependencies in the CT image. It has been observed in the experimental results that the proposed system has achieved an impressive accuracy of 97.8% in detecting and classifying brain stroke, thus establishing it as a novel and effective system for brain stroke diagnosis. Not only does the proposed framework attain state-of-the-art accuracy, but it does so with strong generalization capabilities across various CT image datasets. The model benefits from the crossattention fusion mechanism, as it balances the learning of local CNN features with global transformer representations. The addition of the graph attention mechanism allows for relational reasoning, enabling the model to detect nuanced differences in brain structures. The attention mechanism allows for interpretability, providing clinicians with visual cues that align with clinical practice. This research provides an innovative and reliable solution to the integration of deep learning into clinical stroke assessment.