In August, the Sox pitching staff has combined for a 5.19 ERA and 1.49 WHIP. That's third-worst in the AL, beating only Texas and Kansas City. Even across the entire season, Sox pitching is fourth-worst in the AL, with Tampa Bay also behind them. They have the worst bullpen in the AL, by a fairly wide margin (ERA of 5.36; next worst is TB, with 4.98).
There's been a lot of discussion the last couple weeks on this site as to whether a team can survive the postseason with such glaring problems (for example, here, here, and here). But, with the Yanks only 2.5 games out, making it to the postseason is hardly a forgone conclusion.
If the Sox do make the playoffs, it'll be a pretty rare feat: in 2002-04, all 24 teams making the playoffs were in at least the top 8 in terms of team ERA; most (including the 2004 Sox) were in the top 5 or 6.