How long have you been married, and how long have you lived in Japan?
Count continuous residence, not total visits. Short trips do not break it; long absences, or leaving without a re-entry permit, can. Your residence card history and certificate of residence show the dates.