Judging by the speed of each experiment, the Pacman Pop Const 30 takes roughly as long as the 50, which is interesting as they share the same results. The question is, which approach is best? The Pop Const 10 does eventually match the other two performances, and will likely level out just as the others did. But it can reach the goal in less time, but more learning iterations. I suppose in the end, all are judged by time, as the number of iterations can be infinite, if necessary. So perhaps this Pop 10 works, but should have a larger number of iterations to learn over.

Note that these results are using purely pre-goal specialisation only, no general rule specialisation. That still needs some work before I am ready to launch an experiment for that.

I have so far got around this by using existing background knowledge to check rules, and also introducing a new form of background knowledge (well it is still the same) which isn’t evaluated by the JESS compiler, but is still a valid and legal rule for the environment. However, as I wrote up these rules, I realised that I am essentially telling the agent the dynamics of the environment, which a smarter planning agent could use to achieve its needs. Which isn’t really a bad idea at all, perhaps something I’ll check out later. Anyway, I thought of a new agent measure to learn the environment,

Because the agent can spend so much time in the environment, it should be able to learn the dynamics of the environment by itself, and learn which conditions are always together, and which are apart. By allowing the agent to learn the environment, this means that every background knowledge rule doesn’t have to be declared by the environment designer, only the ones that are required for asserting predicates automatically need be.

The problem this task faces is firstly extra overhead, but as all of the data about the state has already been collected, the agent need only sort and check it against its current beliefs of the environment’s structure. The second problem is deciding when to stop checking the environment. Because the agent only checks the environment for the first few episodes (for covering purposes), it may find its belief’s of the environment to be short-sighted. Like many learning mechanisms of the agent, it may just have to be settled after a number of episodes or something and only forcefully checked when the agent covers new rules and maybe for the first few pre-goal states.

This learning mechanism may have the capability of learning major shifts in the environment, but for now it can just learn constant rules for the entire environment.

The mechanism operates by maintaining three lists for each (non-type) condition: sometimes true, never true, and always true (Both, False, True). Whenever the agent encounters a state, it evaluates the conditions and their relation against other conditions (with each condition in simplified variable form). So initially, after one state has been seen, all currently true conditions are in the True list and all other conditions in the environment are in the False list. For each state seen after this, conditions in either list can either stay where they are, or shift to Both list. Eventually, after a number of states, the observed behaviour will be stabilised for X steps (and have seen Y pre-goals), so the agent can actively scanning the state and focus on learning which rules work.

There is a natural bonus to this system as well. I’m still not 100% sure if it’s foolproof (I’m sure any logical programming book will be able to confirm it for me), but the system of implication allows conditions to be more quickly spread across other conditions. For example, highest(X) -> clear(X), and clear(X) -> !on(?,X). Therefore, highest(X) -> !on(?,X). This may just be learned automatically anyway, but it may be beneficial to be aware of this.

This mechanism can be built into the existing covering class (as a separate class) which could be merged with the known ranges member. Ranges could be a problem too…

I probably should have saved the image for future reference, but I forgot to. Ah well. A possible strategy to pursue is to have a changing population constant, based on the maturity of the experiment. Perhaps a min of 10 and max of 50. Something for later experimentation. The reason for even thinking of it is that smaller population constants are more likely to have large variances. So a form of annealling, I suppose.

Here is the results of the experiment 62.78% (78.39%) into the first run. Note: this has been updated to show later results, also of two experiments.

While these are promising, they come at a high (possibly negotiable) price: time. The experiment has so far taken 94 hours and a half hours. Out of that time, 75 hours is learning time. That’s almost 4 days of runtime and just over 3 days of learning time, and it’s not even half complete on the first run. As said in the previous post: this is likely a combination of the changes to the learning rate and the fact that the better PacMan does, the longer it takes. According to the ETA, the first run will be complete in 5 more days… Assuming 10 days of learning, that’s 100 days to simply produce some reliable results. 3 and a third months!

So something needs to be done. Symphony, shorter learning algorithm, optimised Ms. PacMan environment, any of them. Otherwise my PhD will be spent waiting on possibly unstable experiments to complete.

The readable policy goes like this:
A typical policy:
(distanceGhost player ?X ?__Num7&:(betweenRange ?__Num7 0.0 10.0)) (pacman player) => (fromGhost ?X ?__Num7) / (distanceGhost player ?X ?__Num7&:(betweenRange ?__Num7 0.0 52.0)) (pacman player) => (fromGhost ?X ?__Num7)
(distancePowerDot player ?X ?__Num1&:(betweenRange ?__Num1 0.0 1.0)) (pacman player) => (toPowerDot ?X ?__Num1) / (distancePowerDot player ?X ?__Num1&:(betweenRange ?__Num1 0.0 50.0)) (pacman player) => (toPowerDot ?X ?__Num1)
(distanceDot player ?X ?__Num3&:(betweenRange ?__Num3 1.0 13.0)) (pacman player) => (toDot ?X ?__Num3) / (distanceDot player ?X ?__Num3&:(betweenRange ?__Num3 0.0 52.0)) (pacman player) => (toDot ?X ?__Num3) / (distanceDot player ?X ?__Num3&:(betweenRange ?__Num3 0.0 1.0)) (pacman player) => (toDot ?X ?__Num3)
(distanceDot player ?X ?__Num3&:(betweenRange ?__Num3 1.0 13.0)) (pacman player) => (toDot ?X ?__Num3) / (distanceDot player ?X ?__Num3&:(betweenRange ?__Num3 0.0 52.0)) (pacman player) => (toDot ?X ?__Num3) / (distanceDot player ?X ?__Num3&:(betweenRange ?__Num3 0.0 1.0)) (pacman player) => (toDot ?X ?__Num3)
(distanceDot player ?X ?__Num3&:(betweenRange ?__Num3 1.0 13.0)) (pacman player) => (toDot ?X ?__Num3) / (distanceDot player ?X ?__Num3&:(betweenRange ?__Num3 0.0 52.0)) (pacman player) => (toDot ?X ?__Num3) / (distanceDot player ?X ?__Num3&:(betweenRange ?__Num3 0.0 1.0)) (pacman player) => (toDot ?X ?__Num3)
(distancePowerDot player ?X ?__Num2&:(betweenRange ?__Num2 43.0 50.0)) (pacman player) => (fromPowerDot ?X ?__Num2) / (distancePowerDot player ?X ?__Num2&:(betweenRange ?__Num2 32.75 43.0)) (pacman player) => (fromPowerDot ?X ?__Num2) / (distancePowerDot player ?X ?__Num2&:(betweenRange ?__Num2 12.25 22.5)) (pacman player) => (fromPowerDot ?X ?__Num2) / (distancePowerDot player ?X ?__Num2&:(betweenRange ?__Num2 2.0 12.25)) (pacman player) => (fromPowerDot ?X ?__Num2)
(junctionSafety ?X ?__Num4&:(betweenRange ?__Num4 -11.0 0.0)) => (toJunction ?X ?__Num4) / (junctionSafety ?X ?__Num4&:(betweenRange ?__Num4 -16.0 -11.0)) => (toJunction ?X ?__Num4)

Note that some rules are not present. Primarily the agent runs from ghosts, though only ghosts 10 units away (smart move, if they are hostile). Then the agent eats powerdots to keep the ghosts pacified. Though the first rule of this slot seems a little useless, as the agent will rarely be 1 unit from a powerdot, and when it is, it’ll probably be eating it anyway. Perhaps it is used to counterbalance an overarching fromPowerDot rule. Following this is the toDot slot, times three. The agent clearly likes to stack that rule with all three rules of the slot being active. This will result in the agent always pursuing dots, but when dots are 0-13 units away, the agent pursues them with gusto. The fromPowerDot rule seems a little useless; running from distant powerdots, but perhaps it is just how the agent copes with being forced to use that rule (it is likely to disappear with slot removal as it has a selection chance of 0.58). The junctionSafety one doesn’t seem to be useful either; perhaps the same reason for the previous slot.

When I checked in on the agent previously on the weekend, the fromGhost rule was not the top rule; I think toPowerDot was. Or perhaps toFruit (which is strangely missing – it’s selection ratio was too low, probably caused by the fact the rule rarely triggers). So I’m picking the point in the graph where performance takes off is where fromGhost was most actively used.

The pre-goal model is useless for these sorts of heuristic mutations, so I will likely have to introduce a new algorithm for creating mutations: either with or without the existing pre-goal one. Perhaps maintaining a tree of rules, with the covered rule as the base rule and more specialised rules running down from it. This tree needs to be prunable too. This is beginning to look a lot like TG, though.

A secondary problem is the fact that actions not necessarily directly concerned with objects in the domain may still perform better with conditions linked to those objects (e.g. toPowerDot only when ghosts are not edible). If actions didn’t have delayed effects, this could easily be found by watching the post-action state, but alas, toPowerDot could be active at any point, regardless of PacMan’s distance to a powerdot. As far as I am concerned at the moment, I cannot see a solution to this problem without expensive state watching. Perhaps just the to/fromGhost actions will have to suffice for now.

How irritating. Well, I guess I’ll just have to restart it. But I will do that tomorrow, when I have found a solution to decreasing the elite policy size (and hopefully increasing the update value).

As for Ms. PacMan, it is still slow, but after 12 (learning) hours, the experiment is 2.7125% along (27.125% iteration). The ruleset is beginning to shape itself to something useful, with handy fromGhost rules (which include conditions for ghost state: aggressive) and toDot rules near the top, and other, less useful rules near the bottom/disappearing. Assuming it continues at this speed, one iteration will take ~45 hours (2 days). Hence the entire experiment takes 20 days, but these can be split up.

Speaking of splitting up, Eibe brought up the possibility of splitting the learning across multiple machines (i.e. Symphony). This could be easily achieved, as the very nature of the experiment allows it to be split. Simply send out X agents operating in their own environments and when they return, sort them in order and update generator. Then repeat. Of course this current system operates in an iterative manner, but the learning should be roughly equal if the update parameter is proprotional to the number of samples.

An alternative to that method is a much larger one which only requires 10 machines, each running entire experiments. Then the results are averaged and stored. But that takes much longer and doesn’t make full use of the number of processors Symphony has available.

Seems there is still the problem of statistical pre-goal unification which hasn’t upset the PacMan experiment yet, but is likely to when a pre-goal is created with edible ghosts. I’ll have top give more thought to sorting that out later.

A testing policy from PacMan:
Policy:
(distanceGhost player ?X ?__Num7&:(betweenRange ?__Num7 2.0 12.666666666666666)) (nonblinking ?X) (aggressive ?X) (pacman player) => (fromGhost ?X ?__Num7)
(distanceFruit player fruit ?__Num9&:(betweenRange ?__Num9 2.0 14.5)) (pacman player) => (toFruit fruit ?__Num9)
(distancePowerDot player ?X ?__Num1&:(betweenRange ?__Num1 0.0 51.0)) (pacman player) => (toPowerDot ?X ?__Num1)
(distanceDot player ?X ?__Num3&:(betweenRange ?__Num3 0.0 52.0)) (pacman player) => (toDot ?X ?__Num3)
(distancePowerDot player ?X ?__Num2&:(betweenRange ?__Num2 19.5 29.25)) (pacman player) => (fromPowerDot ?X ?__Num2)
(distanceGhostCentre player ?X ?__Num8&:(betweenRange ?__Num8 0.0 13.0)) (pacman player) => (toGhostCentre ?X ?__Num8)
(distanceGhost player ?X ?__Num6&:(betweenRange ?__Num6 34.0 43.0)) (pacman player) => (toGhost ?X ?__Num6)
(junctionSafety ?X ?__Num4&:(betweenRange ?__Num4 -8.0 0.0)) => (toJunction ?X ?__Num4)
(distanceGhostCentre player ?X ?__Num5&:(betweenRange ?__Num5 0.0 52.0)) (pacman player) => (fromGhostCentre ?X ?__Num5)

Clearly from ghost behaviour is most important, along with eating fruit and using the powerdot to keep the ghosts placid. The toDot is all encompassing and will always be active. The fromPowerDot rule will only trigger at a distance, so it has no value. The toGhost rules have little value currently, as they don’t include the edible attribute, but that is likely because they can’t be included. The last 2 are practically useless, though the all-compassing fromGhostCentre does lend some defensive behaviour.

Just had a thought about pre-goal mutation and such. Haven’t fleshed out the possibilities yet, but mutate rules based on what constant elements are present and mutate in relevant conditions seen in conjunction with said elements. So a rule concerning a ghost would mutate in the 4 attributes concerning the ghost: edible, aggressive, blinking or nonblinking. This could also create an opening for negation, allowing me to remove half of those attributes. I’ll think on it.

And in a simple case, Blocks World has an action without use: moveFloor. Yes, it is used in modules, but when using modules, it is useless. So for the sake of these environments and possibly future environments, I will implement an optional slot addition/removal system.

The algorithm is something like this:
- Each slot maintains an M value for the base chance of it being used (initially M = 1, M >= 0) and an S value for variance (initially S = say 0.5, 0 <= S <= 0.5).
- The chance of a slot being used depends on a normally sampled value of m = M +- S.
- If m is above 1.0, the slot will be used but not removed from the slot distribution. The value for the not removed slot is equal to m – 1.0.
- If m is below 1.0, the slot will only be used (and removed from the distribution) if a random number is below m. Otherwise the slot is not used but is removed from the distribution.

For example, a slot has M = 1.0 and S = 0.5. It is drawn with m = 1.4. Hence, the slot is used at least once and a 40% chance of being used twice.

Another example, a slot is drawn with m = 2.3. The slot is used at least twice and a 30% chance of being used thrice.

Another example, a slot is drawn with m = 0.8. The slot only has an 80% chance of being used at all.

During the update procedure, these slot values M and S are updated as well, with M’s values being updated in a standard step-wise manner using the elite solution count. S’s value needs to slowly decrease over time if M remains relatively stable. Perhaps a standard deviation for the elite samples can be calculated and used to update S in a step-wise manner. That should suffice.

The algorithm can be changed to modify the probabilities of individual rules being selected by taking into account how many times a rule has been previously used. So given a distribution of rules D, where each rule has probability P(R) and each rule records how many times it has been used U(R) (this includes times ruls are used but never fired), the probabilities of D can be updated at each step using the following formula:

D’ = forall(R): P’(R) = if (U(R) < #(R in D) * alpha) ? P(R) * (#(R in D) + 1 - U(R)) * beta : P(R)

This states, if a rule's uses are less than the number of rules in the distribution * alpha (a coefficient), then modify the probability of being selected by the probability * the number of rules in the distribution + 1 - the number of uses * beta (another coefficient).

This would probably work well enough with alpha and beta both set to 1, but some modifications may be needed to properly test each little used rule.

An example distribution:
Rule A: 0.4, U = 5 => 0.4
Rule B: 0.17, U = 0 => 0.17 * 5 = 0.85
Rule C: 0.26, U = 1 => 0.26 * 4 = 1.04
Rule D: 0.17, U = 0 => 0.17 * 5 = 0.85

Normalised:
Rule A: 0.13
Rule B: 0.27
Rule C: 0.33
Rule D: 0.27

This system attempts to maintain the probabilities of the original distribution but offsets them by the number of uses, which will eventually decrease to the threshold, resulting in a fair and stable distribution again.

I suppose I could ignore existing probabilities and just set the distribution to reflect the inverse uses (until all thresholds are met). But this could result in unfair balancing.

A Ms. PacMan experiment was run recently (40 hours in, though half of that was just testing) which has the following results (about 62% completion):
2064.0
3259.2
3441.8
3574.0334
3836.2334
3582.9666
3420.2
3421.8333
3715.6667
3890.8667
3555.1333
3555.1667
3542.5
3654.1667
3400.3667
3484.4
3443.6667
3809.3333
3616.8667
3061.9
2986.3
3770.3667
3762.3667
3142.9333
3247.0334
3266.5
4481.4
4112.433
4163.967
3327.4333
3934.3
4198.533

This is significantly worse than the previous experiment over the regular environment. While there is some initial improvement in the rules, it doesn’t appear to be increasing very fast and the last few results could simply have been flukes.

The readable generator is:
A typical policy:
(distanceDot player ?X ?__Num3&:(betweenRange ?__Num3 0.0 52.0)) (pacman player) => (toDot ?X ?__Num3)
(distanceGhost player ?X ?__Num7&:(betweenRange ?__Num7 0.0 52.0)) (pacman player) => (fromGhost ?X ?__Num7)
(distancePowerDot player ?X ?__Num1&:(betweenRange ?__Num1 0.0 50.0)) (pacman player) => (toPowerDot ?X ?__Num1)
(distanceGhostCentre player ?X ?__Num5&:(betweenRange ?__Num5 0.0 51.0)) (pacman player) => (fromGhostCentre ?X ?__Num5)
(junctionSafety ?X ?__Num4&:(betweenRange ?__Num4 -16.0 28.0)) => (toJunction ?X ?__Num4)
(distancePowerDot player ?X ?__Num2&:(betweenRange ?__Num2 0.0 50.0)) (pacman player) => (fromPowerDot ?X ?__Num2)
(distanceGhostCentre player ?X ?__Num8&:(betweenRange ?__Num8 0.0 51.0)) (pacman player) => (toGhostCentre ?X ?__Num8)
(distanceGhost player ?X ?__Num6&:(betweenRange ?__Num6 0.0 52.0)) (pacman player) => (toGhost ?X ?__Num6)
(distanceFruit player fruit ?__Num9&:(betweenRange ?__Num9 0.0 52.0)) (pacman player) => (toFruit fruit ?__Num9)

Obviously, toDot behaviour is best, followed by fromGhost behaviour. Normally, I’d prefer fromGhost to be highest weighted, but because it only contains one (or two) all-encompassing rules, it would mean the agent spends most of its time cowering in the corner. ToPowerDot is above fromPowerDot, generally a good choice, and toGhost and toGhostCentre are both lowly weighted. Strangely, toFruit is quite low. Possibly because it doesn’t turn up so much? You know, the fruit may be making all the difference between the scores from this run and the previous one.

On the bright side, the faster optimisation seems to be working, with this run due to be completed in about 24 hours, making a total time of 63 hours.

I need to fix this range mutation and also to fix the mutation towards useful to/fromGhost rules that include the binary attributes edible/aggressive. Furthermore, the modularisation for learning clear still hasn’t been 100% solved, as sub-optimal rules are being chosen as the best.