Dan Stevens is out as Matthew. So is Siobhan Finneran as O'Brien.

But SIX new characters are on their way to Downton Abbey Season 4, as teased by Masterpiece and Executive Producer Gareth Neame, who tells PBS:

"Downton Abbey has seen many great characters visit the house over the years and we couldn't be more thrilled to welcome the new faces that will be joining the regular cast of Downton Abbey in Season 4. We hope audiences will be as excited about them as we are."

Along with Shirley MacLaine making a return for next season's finale, the following actors/characters are joining the cast:

• Tom Cullen as Lord Gillingham, an old family friend of the Crawleys who visits the family as a guest for a house party and connects with Mary.
• Nigel Harman as a valet named Green.
• Dame Harriet Walter as Lady Shackleton, a long-time friend of the Dowager.
• Joanne David in a guest role as the Duchess of Yeovil.
• Julian Ovenden as an aristocrat named Charles Blake.
• Dame Kiri Te Kanawa as a guest who sings in the home.

Source:
http://www.tvfanatic.com/2013/03/downton-abbey-season-4-first-photo-casting-news/