"The dog lived in the garden, but the cat, who was smarter, lived inside the house" is an example of—

This sentence contains two independent clauses ('The dog lived in the garden' and 'the cat lived inside the house') joined by the coordinating conjunction 'but'—making it compound. It also contains a dependent relative clause 'who was smarter' modifying 'the cat'—making it complex. A sentence with two or more independent clauses AND at least one dependent clause is called a complex-compound (or compound-complex) sentence. Hence (2) is correct.