Here are some random and possibly off the wall changes that I would consider trying to include in an Angband variant/fork. More details may be added as I flesh out concepts more. Specifically if others thing they are interesting or promising. Certainly if anyone else likes one of these and has a strong desire to code it themselves, they should certainly go for it.
Split AC into absorption/evasion
This was already attempted once but fell a bit flat. I think I can get it right the second time. The basic idea is that monster AC would be split into a difficulty to hit and a difficulty to damage. For example something like a tick would be easy to hit but difficult to damage. While something like a will o the wisp would be difficult to hit but easy to damage. (CC: very keen to see this happen as part of NewCombat.)
Allow monsters to climb over rubble
Give monsters and players the probability of climbing over rubble instead of getting stopped by it (cf. DaJAngband). The probability of success would depend on player dexterity and monster level or possibly evasion. Failure would result in a loss of a turn.
Implement flying monsters
Sort of a necessity for the previous point. Monsters that have the flying ability can easily pass over rubble with no trouble. (CC: this has very interesting possibilities, esp. if combined with others' ideas about new terrains.)
Split monster awakeness further
Also an idea poached from DJA. Monsters can be either asleep, awake, or alert. Asleep and alert monsters are what we currently are used to. Awake monsters are ones that are moving around but do not know where @ is. They become alert if @ moves into their line of sight (and they can see @, more on that later).
Let awake monsters wander
Monsters that are awake but not alert should wander about the level. Exactly how to do this is not clear. One option is to pick a random point in the dungeon and have the monster move towards that point. However, this is probably not a good idea because it may be extremely time consuming to compute these paths. Another option is to have a structured random walk where you save the last move, and give the monster preference to move in the same direction it just did, and really dislike moving in the opposite direction. I'm not clear how well this will work.
Pits/nests need to be handled carefully. These monsters shouldn't break out and start wandering around the level immediately. (CC: IMO one possibility is to have different types of doors/gates, rather than special-casing the wandering code for monsters in pits/nests.)
Groups are also a bit tricky to deal with. One possibility is to determine a group leader and have it wander around (handing off the scepter if necessary) and have everyone else follow it.
Let awake monsters cast certain spells
Specifically, phase, teleport, haste. Heal-self is a bit hard to consider. At the very least, any monster with reasonable HP and heal-self + teleport of any form is far too annoying. (CC: once we do #1279, these problems go away. Also, would you introduce S/NPP's TELE_SELF_TO?)
Allow monsters to wake up/alert friends
If a monster is alert and it has other monsters in the room with it that are awake but not alert, it can use its turn to yell, in which case all the awake monsters become alert. In this way if a hound sees @ approach a room, all the hounds immediately become aware. A monster should always choose this action.
If a monster is alert and is adjacent to a monster that is asleep, it can use its turn to wake up the adjacent sleeping monster, who becomes alert. Only one monster can be woken up at any time. A monster should always choose this action.
Allow player hiding/invisibility
Players with lights will always be seen immediately by monsters. Players without lights (or unlights) will only be seen by monsters that can detect infravision (most). Players can also shield infravision (by becoming invisible, or using other means) in which case the player can only be seen by monsters that can see invisible.
Also, players should be able to hide and ambush monsters. A well-hidden player is only detectable by monsters with ESP. Hidden players cannot move but can ambush wandering monsters. Getting a damage bonus on the first round of combat (maybe a significant damage bonus for rogues) This would require quite a bit of work, and I'm not convinced that the payoff is worth it. (CC: it works quite well in S, so I think it might be worth it.)
Allow the player to hear monsters
I've wanted to do this forever, but haven't gotten anywhere near it...
(CC: this is great list, with the potential to completely change the feel of the AI, and make the game much more interesting/challenging without actually needing to boost any difficulty. Once you've decided on specifics, maybe open some tickets for individual changes?)