Yes, it's absolutely fine. Just because the constructor can only be called by derived classes doesn't mean it won't be useful. For example, you might have an abstract class which represents a named entity of some kind - it would make sense to take the name as a constructor parameter.
It would probably be worth making the constructor protected, to make it even more obvious that you can't just call it from elsewhere.
Note that there being a constructor (or multiple constructors) in an abstract class does force derived class constructors to go through it, but it doesn't force the derived classes to have the same constructor signatures. For example:
public abstract class NamedFoo
{
private readonly string name;
public string Name { get { return name; } }
protected NamedFoo(string name)
{
this.name = name;
}
}
public class DerivedFooWithConstantName
{
public DerivedFooWithConstantName() : base("constant name")
{
}
}
In this case the derived class constructor is "removing" a parameter (by providing a constant value as the argument to the abstract class constructor) but in other cases it could "add" parameters that it required, or have a mixture.