Yagami-kun's Family Affairs follows Yūji Yagami, a high school student with a problem: his mother, Nomi, looks very young and Yūji has a crush on her. To complicate matters, his high school homeroom teacher is also infatuated with her due to how young she looks. Nomi is oblivious to all of this and is head over heels in love with her husband, Yōji, and frequently displays this affection very publicly, which causes more embarrassment for Yūji.
Yūji's high school friends constantly tease him about his "mother complex". When Valentine's Day comes around, though, Yūji receives a box of handmade chocolates. The name on card can be read multiple ways, the most common way being Masayuki Ikari (五十里 真幸 ,Ikari Masayuki?), a male name, so his friends tease him throughout the day until it's time to meet the person who sent the chocolates at the front gate after school.
The girl who sent the chocolates (the name is actually read "Mayuki") starts meeting him after school so they can walk home together, and they eventually become an item. The remainder of the story follows Yūji and Mayuki as their love grows and as Yūji works to resolve his "mother complex" issues.