Polyphonic pitch estimation and musical instrument identification are some of the most challenging tasks in the field of Music Information Retrieval (MIR). While existing approaches have focused on the modeling of harmonic partials, we design a joint Gaussian mixture model of the harmonic partials and the inharmonic attack of each note. This model encodes the power of each partial over time as well as the spectral envelope of the attack part. We derive an Expectation-Maximization (EM) algorithm to estimate the pitch and the parameters of the notes. We then extract timbre features both from the harmonic and the attack part via Principal Component Analysis (PCA) over the estimated model parameters. Musical instrument recognition for each estimated note is finally carried out with a Support Vector Machine (SVM) classifier. Experiments conducted on mixtures of isolated notes as well as real-world polyphonic music show higher accuracy over state-of-the-art approaches based on the modeling of harmonic partials only.