Two of the victims were Europeans, a Zanzibar minister said. Officials say 145 people have been rescued, but dozens are still missing.
The boat had left the city of Dar es Salaam, mainland Tanzania, earlier in the day.
The navy said it got into difficulty because of strong winds.
"We have so far received 24 bodies, including two Europeans," Zanzibar's Transport Minister Hamad Masoud Hamad told reporters.
The journey usually takes about two hours.
Thirty-one children are believed to have been on board.Last September, nearly 200 people died when an overcrowded boat with 800 people aboard sank off Zanzibar.