To make the banjitar sound, well, more banjo-like, it's a pretty common thing to play around with string gauges. Players can get pretty creative, which is a good thing. Playing the banjitar means you're not locked into a certain set of rules or expectations. I'm not saying the ideas I'll present here are recommended or even best practices. I'm just throwing them out there to give you some ideas you can try. If you're happy with the way your stock banjitar sounds, then you can just ignore these suggestions.
I have three banjitars and I keep them all strung up and tuned somewhat differently. My "standard" tuning banjo is tuned just like a guitar and I use regular Ernie Ball electric guitar strings on it. This one has a wooden rim and a resonator and it has a very mellow tone. The lighter the strings, the more the banjo head can vibrate. It's almost the opposite of a guitar where heavier strings drive the top harder to produce more volume and tone. The head of a banjo vibrates better with lighter strings. I use the .009-.042 gauge on this one. I've never tried going lighter, but I might give it a try in the future, just to see how it sounds. So far, the regular .009 gauge works fine. I typically capo this instrument between the 2nd to 4th fret, just because that's the sound I like. Using the capo moderates the low frequencies of that last octave below the five-string banjo's range.
My open back banjo has a distinctly different sound that the ones with closed backs. It is more open sounding and it's not as loud as the others. On this one I tried several combinations. I started with extra-light acoustic guitar strings, which greatly improved the sound from the lights. Then, seeing the difference the lighter gauge strings made, I went even lighter. Now I use Ernie Ball .009-.042 gauge also, but I tune this one to open A tuning (E A E A C# E). This is closer to a traditional five-string's chords, but it has low-sounding strings on the top. I have played guitar in open G tuning, which is like a five-string banjo's tuning, except for the lowest-tuned strings. The fifth string is a G, but it is a low G, tuned two octaves down from that annoying half-string on the top of a banjo's courses. Additionally, the sixth string is a low D, tuned a whole step below a guitar's low E string. You'd think open G would be a no-brainer on the banjitar, but those two low strings are a deal-breaker. If you thought the low E and A strings sounded bad, tuning them each down a whole step is even worse. The string tension on those two strings and the low frequency combine for an unpleasant sound. However, if you use light gauge electric strings and tune the whole thing up a whole step, it just works somehow.
My most "banjo-sounding" banjitar is a closed-back resonator model with a cast aluminum rim and flange. This one is strung up with D'Addario Nashville Tuning strings. If you are unfamiliar with Nashville tuning, it's a trick studio players use to help acoustic guitar parts cut through in a dense mix. Originally, it came about by taking the high octave strings on a twelve-string guitar set and putting them on a six-string guitar. That puts the top four strings up an octave from their customary pitch, leaving the first and second strings tuned to their usual pitch. This creates a jangle that rings through, without having to do a lot of EQ to get the acoustic guitar to sit right in the mix.
I originally tried a setup I saw in a video by Banjo Bill, which I have in a blog post earlier on this site. Banjo Bill changed out just the top three strings with the low E string two octaves higher. I modified the gauges to use a .010, .017, .013, .017, .013, .010 set. I liked the sound OK and the whole thing was unwound strings, which made for a pretty uniform banjo jangle. I found though, over time, that the string tension just felt kind of unbalanced and I started to experiment with different gauges. I saw another video by a guy who puts a .010 on the sixth, to give him a high drone like you'd have on a five-string banjo while all the rest of the strings were tuned like a guitar. He had "railroad spikes" installed, which act like a fifth-string capo on a banjo. Then he can capo that one string to whatever note he wants so he can have a high drone note in several keys. Interesting idea. I tried that, but I think it would only be useful if you do the spikes also.
Finally, I decided to try the D'Addario Nashville sets and I really liked the sound. With them, the third, fourth, fifth, and sixth strings are all an octave above normal. This gives you a high G on the third string, which is the same as that high octave on a 12-string. I have played twelve-strings a lot over the past forty years and that high G string pops pretty easily on a guitar with a 25 1/2 inch scale length. Stretching it to pitch on a banjo with a 26 1/2 scale length seemed pretty dicey. It would tune to that pitch, but the banjitar felt pretty tight all the way across the strings. What I decided to do was tune down a whole step (D G D F A D) and put capo on the 2nd fret to play in standard tuning. This was actually quite comfortable. The string tension feels very much like a five-string banjo and, when fingerpicking, you get some delightful tones. The long scale length makes first-position chords kind of a long stretch with small hands, so I like the feel with the capo. If I want to play without the capo, it still sounds good all the way across the whole range of the instrument. This is the instrument I use when I want to do some complicated-sounding Bela Fleck-like stuff.
The banjitar seems to invite innovation and creativity as players think "outside the box" to find the sounds they are hearing inside their heads. If you have a special tuning, string set, or other invention that you use with your banjitar, please feel free to share!