Part A

$\mu$ large implies that the wedge does not move. Therefore the two forces on the small masses along their direction of movement are:$$F_1 = m_1 g, \quad F_2 = m_2 g \sin(\alpha).$$

They pull in different directions on the pulley, so the net acceleration (with respect to $m_1$ going down) is:$$F = F_1 - F_2 = g [m_1 - m_2 \sin(\alpha)].$$Then the acceleration of both masses is:$$ a = \frac Fm = \frac{g}{m_1 + m_2}[m_1 - m_2 \sin(\alpha)].$$

You should not be able to introduce a minus error just by choosing a convention. I assume that you just did not flip all the minus signs.

Part B

The wedge has to be at rest, so the force of friction has to compensate the tangential forces exactly. This means$$ \mu F_\text{down} \overset != F_\text{tangential}. $$

I now construct the forces that act on the masses:

Image may be NSFW.
Clik here to view.

The tensions will change with the acceleration:$$ T_1 = m_1[g-a], \quad T_2 = m_2[g\sin(\alpha) + a].$$To digest this, set $a =0$, the static case. Then the tensions are just the forces that gravity exerts on the masses. The extreme case, where $m_1$ is free falling is $a = g$. Then $T_1$ has to be zero, since the object is free falling.

The mass $m_2$ exerts a normal force onto the wedge that will push it to the left. The tension $T_2$ will drag on the pulley which will then push on the wedge to the right. The force to the left is given by$$ m_2 g \cos(\alpha)\sin(\alpha)$$whereas the force to the right (just the $x$-component) is given by$$ m_2[g \sin(\alpha) + a] \cos(\alpha).$$

In the static case $a = 0$, those two forces are equal and do not make the wedge move. In the dynamic case, they try to move the wedge. This is what you meant by conservation of linear momentum, just written down with the forces.

The net force to the right (assuming $a > 0$ then is$$ m_2 a \cos(\alpha) = F_\text{tangential}.$$

Now the forces that push the wedge downward and give it a frictional force. The mass $M$ will drag downward. Then $T_1$ will push the pulley down, so does the $y$-component of $T_2$. The last thing is the $y$-component of the normal force. The last two mostly add up to just $m_2 g$, there is still a $m_2 a \sin(\alpha)$ left from the dynamics in $T_2$:$$\underbrace{m_1[g-a]}_{T_1} + \underbrace{m_2[g\sin(\alpha) + a]\sin(\alpha)}_{{T_2}_y} + m_2 g \cos(\alpha)^2$$

Together, this is:$$F_\text{down} = Mg + m_1[g-a] + m_2[g + a\sin(\alpha)].$$

Now set them equal, plug in $a$ and solve for $\mu$. You might want to omit all the terms with $a$ here, since that makes it really complicated. If $M$ is large, you can do that without a large error.