Welcome to the forum, Bhx7!
You are correct. The Rose & Crown Inn was located to the west of the viaduct and situated between Keelmans Row and Rose & Crown Row.
(Ignore the red star)
Census enumerators were creatures of habit who followed a strict route. In the 1901 census, having left the Sea View/Paradise Row area (Bottom right corner of map) he entered the Iron Works area and followed his route including Fergusons Row (where my father was born, 1900) proceding through Old Factory Yard and its offices, he then makes his way downhill to the riverside where he records in the following order as he walks towards the bridge at the foot of the hairpin bend:
Green – Rose & Crown Inn (marked P.H. ’Public House’ on the map)
Orange – Rose & Crown Row (nrs 1-6)
Pink – Bridge House
Yellow – Clock House
Blue – Rose Cottage, the last dwelling in his district (Cowpen 16)'
The Rose & Crown Inn is actually named on this second map from 1897 just 4 years before the census.
Thanks to Eggy for the art-work on this last map. As you see, he's much better at it than I am!