F(x) là một nguyên hàm của hàm (x - 1) căn bậc hai của x^2 - 2x - 3

Ta có $F\left(x\right)=\underset{}{\overset{}{\int }}\left(x-1\right)\sqrt{x^2-2x-3}dx$

Đặt $t=\sqrt{x^2-2x-3}\Rightarrow t^2=x^2-2x-3\Rightarrow tdt=\left(x-1\right)dx.$

$\Rightarrow F\left(x\right)=\underset{}{\overset{}{\int }}f\left(x\right)dx=\underset{}{\overset{}{\int }}{t}^{2}dt=\frac{t^3}{3}+C$

$\Rightarrow F\left(x\right)=\frac{\left(x^2-2x-3\right)\sqrt{x^2-2x-3}}{3}+C$

ĐKXĐ: $x^2-2x-3\ge 0\iff \left[\begin{array}{l}x\ge 3\\ x\le -1\end{array}\right..$

Khi đó ta có: $F\left(x\right)=\left[\begin{array}{l}\frac{\left(x^2-2x-3\right)\sqrt{x^2-2x-3}}{3}+C_{1}khix\le -1\\ \frac{\left(x^2-2x-3\right)\sqrt{x^2-2x-3}}{3}+C_{2}khix\ge 3\end{array}\right.$

Ta có: $\left\{\begin{array}{l}F\left(-2\right)=\frac{5\sqrt{5}}{3}+C_1=\frac{5\sqrt{5}}{3}\Rightarrow C_1=0\\ F\left(4\right)-1=\frac{5\sqrt{5}}{3}+C_2-1=\frac{5\sqrt{5}}{3}\Rightarrow C_2=1\end{array}\right.$

$\Rightarrow F\left(x\right)=\left[\begin{array}{l}\frac{\left(x^2-2x-3\right)\sqrt{x^2-2x-3}}{3}khix\le -1\\ \frac{\left(x^2-2x-3\right)\sqrt{x^2-2x-3}}{3}\text{+1 }khix\ge 3\end{array}\right.$

$\Rightarrow \left\{\begin{array}{l}F\left(-3\right)=\frac{12\sqrt{12}}{3}=8\sqrt{3}\\ F\left(5\right)=\frac{12\sqrt{12}}{3}+1=8\sqrt{3}+1\end{array}\right.$

Vậy $F\left(-3\right)+F\left(5\right)=16\sqrt{3}+1\Rightarrow \left\{\begin{array}{l}a=16\\ b=1\end{array}\right.\Rightarrow a+b=17.$

Chọn A.