---"I'm currently using a TG-5 in Olympus housing with 2 Inon S2000 strobes, a focus light, and the AOI UWL-400 wide angle lens. After taking 3 dive trips with the camera I've decided to upgrade to a dslr setup since I was pretty frustrated with the TG-5's lack of manual control, shutter speed, and low light performance. The deciding point was when I went to Malapascua and couldn't take a single good picture of the thresher shark over 3 days, as no lights or strobes were allowed. I also had a lot of trouble taking good pictures of the mandarin fish mating since they moved so fast and I couldn't shine a focus light on it."
I hear you. I went to a DSLR after years of point-n-shoots, and a year of a Sony RX100 with dual strobes. Lack of autofocus speed and precision was the primary driver, but since then (using both D810 and D850) I'd add in dynamic range and resolution as extremely important factors.
---"So I've decided on a Nikon D7500 since it seemed to be a good mid-range option and bought the Nauticam housing. I bought 2 lenses, the Tokina 10-17mm which seems to be widely recommended for wide angle, and the Nikon AF-S 85mm which seems like a good compromise between the typical 60mm and 105mm recommendations. I bought the recommended zoom gear for Tokina + the 4.33" dome port, and the compact port base + macro port 30 per Nauticam's port chart."
Sounds good so far, though I have only FX experience here and not DX. However, DX cameras have some lens advantages that make be a bit jealous. DX cameras have a lot more options. My FX cameras don't even have a mid-range option.
Some macro comments: I've shot (FX) with the 60 and 105 Nikkor macros. The 60 was an experiment to see if I could get both macro AND somewhat of a mid-range option. With the 105, you almost can't get a whole diver pic - the strobes are too far away and water clarity becomes a big issue. With the 60, macro suddenly became an issue - I had to be so close to my subjects that it just didn't work for me. That experiment lasted one dive.
So I've been shooting the 105 macro for 3 years now, and the biggest complaint I have is that it's a bit short for my usual macro subjects. Some divers can get closer, but I tend to shoot from about 3 feet away. I tried a 1.4TC next, and while it gave me pretty much exactly focal length that worked best (150mm), it robbed enough sharpness that I was dissatisfied with the results.
Which makes me think you 85DX macro might be just right. That said, the 60mm is no slouch and has been used successfully by many photographers underwater. The lack of VR on it is a bit of an issue.
---"All in that already cost me over US$6000 and I'm debating whether or not I want to continue to purchase the below items, which based on my research seems to be "nice to haves" but not totally critical:"
Yes, I hear you. I just went ahead and bought pretty much everything I wanted, with the idea that "I'm not getting any younger, and it's now or never". I bought a 180 degree viewfinder with my first DSLR rig, along with a leak detection option (Nauticam).
---"-Nauticam vacuum pump- I haven't had any issues with my TG5, and no issues with my Go Pro Hero 5 before that. We've been diving for 3 years and never had a flood so questioning whether it's worth the $200?"
Oh, it's TOTALLY worth it! Not even a question. Not only will it give you great peace of mind, it may very well save your dive rig sooner than you think. I had a couple of warnings over the years on pump-down, and one definitely saved the rig by alerting me the dome port wasn't properly attached before diving.
These are also great for those hurried-on-board-fixes or battery changes one does on a dive boat. I always carry my vacuum pump on board with me, even if not intending to change batteries between dives.
---"-Nauticam external viewfinder - either 45 or 180. I read the 45 is really good for macro but it takes a long time to get used to, and not recommended if you only take a few dive trips a year? I wouldn't mind paying for it if it's easy to get used to but I wouldn't want to pay $1000 for it if I need to waste a few dives at the beginning of every trip getting used to it"
I thought about the 45 degree, but opted for the 180. I don't remember the logic at the time for not choosing the 45 degree. I bought my first rig having never used (or even seen) another FX DSLR rig.
With the 180 viewfinder, I can easily see the whole viewfinder through my mask. It's just a high-eyepoint viewfinder. If it magnifies too I never noticed. I need to be able to read the viewfinder information displays as well as the corners of my subject.
As an added benefit (?) it gives me a third stabilization point, along with both hands, as I press my mask against the external viewfinder. Probably the same without it, but totally different than shooting Live View or mirrorless and holding the camera out in front of you.
---"-a macro diopter, thinking through Subsee+10, Nauticam SMC-1, and Aquako III. My next trip is to Anilao, but wondering if I will take a few days to adjust to the 85mm macro lens before I can move on to supermacro anyway, so if I can push this purchase further down the line? The Aquako appeals to me the most because of its small size and price, but it's a 52mm thread. If I get a swing mount, then use a 52 to 67mm setdown ring, then attach the Aquako, will there be too much distance between the lens and the diopter to be effective?"
I don't know the answer, but am intrigued with the idea. I've gone back and forth with the thought of adding diopters, and I'll probably try something at one point. I had wet lenses on my RX100 rig, along with a flip holder, but I could never get a decent shot with the macro adapter. (I was probably not close enough). And I actually never once used my wet dome lens for wide angle.
Given my TC experience with macro, I'm a bit suspicious of loss of IQ, and I'm actually looking into a 150mm Sigma macro lens, if I can marry it with a proper port and extension.
---"My next trips will be 1) Anilao, 2) French Polynesia, 3) Maldives - so thinking macro lens for 1) and just the wide angle for 2) and 3)?"
Personally I always bring both wide and macro when diving. So far I've mostly been doing macro for morning dives, then wide angle for afternoon dives (assuming clear water) when the sun should mostly give better light for ambient.
I've not been to Analao or Maldives, but I have been to French Polynesia, and I'd say there are both wide and macro shots there in abundance.
---"I will also have the standard 18-55mm kit lens, should I try to use it with the 4.33" dome port or the macro port?"
No idea here - I don't have a standard zoom option with my D810/D850 - but I suspect the dome port is what you'll want. No idea on extension length.
You know that dome ports do not enlarge while macro ports do?
"Lastly I'm debating what to do with my TG5. I can 1) give it to my husband who isn't really a photographer, and who also has the go pro hero 5, 2) try to sell it- but seems like a waste for something that's only been on 30 dives, or 3) take it with me as a backup camera?"
Take it as a backup camera.