In our theme demo we use slider instead of Custom Header Image and it would be so muh easier for you to do the same instead of trying to change element place via code. For more information on hot to setup slider, please see theme documentation.
To add caption on images you can use code like this. Other customizations might be required but this should get the most of the job done. Add this code to Theme Options - Other - Custom CSS
That code is exactly what I was looking for. There is only one little problem: when I use a full width image, everything is perfect(see here), but with smaller images, caption line is about 3mm longer than image canvas(example1, example2). What should I change in the code to fix this?