Guild of Project Controls: Compendium | Roles | Assessment | Certifications | Membership

Tips on using this forum..

(1) Explain your problem, don't simply post "This isn't working". What were you doing when you faced the problem? What have you tried to resolve - did you look for a solution using "Search" ? Has it happened just once or several times?

(2) It's also good to get feedback when a solution is found, return to the original post to explain how it was resolved so that more people can also use the results.

Comparison of resource levelling algorithms of different Project Management tools

34 replies [Last post]
Evgeny Z.
User offline. Last seen 38 weeks 5 days ago. Offline
Joined: 13 Jan 2008
Posts: 442
Groups: None

Dear all,

Few years ago  I posted here an articles on brief comparison of resource levelling engines of different tools:

In addition to this I started an initiative to develop requirements to conduct more scientific resource levelling engine comparison of different tools. That initiative did not really proceed back then.

Now a couple of years later I still did not find any report on comparing of resource levelling of different project management programs, therefore I want to restart my 2 years ago initiative but with much more flexible rules. 

So, what I suggest:

Anybody can post here a comparison results of resource levelling outcomes of at least any 2 tools using any comparison method he has chosen for himself. The only hard requirement would be: that the test would be repeatable.

In such way we can build a sort of database of results, from  where anybody can track conclusions independently.  

I also plan this 1st post in this thread with the comulative results table, so that it is all in one place.

 

Brief digest on the comparison results so far (updated 04.09.2015 01:27 CET)

======================================================================

 

Microsoft Project 2010 SP2

Spider Project Optimization Plus

Primavera P6 v 15.1

Comparison criteria

Average ratio leveled / not leveled MSP 2010 SP2

Average ratio Leveled / not leveled Spider

Average Spider Project duration relative to MSP 2010 SP2

Average ratio leveled / not leveled Primavera P6 V15.1

Average Primavera P6 V15.1 relative to MSP 2010 SP2

25 problems from RCSP library

2,81

2,46

88%

2,95

105%

 

Graphical representation of results, relative to Microsoft Project 2010

 

 photo PM_SW_tools_comp_2015.04.09_zpsj4f2l8nu.png

Replies

Evgeny Z.
User offline. Last seen 38 weeks 5 days ago. Offline
Joined: 13 Jan 2008
Posts: 442
Groups: None

This is an old thread, but just to follow up:

Together with Vladimir Liberzon we have used the work, which originated from this thread and put together presentation for the year 2017 Construction CPM Conference

Resulting slides can be found here:

Resource Constrained Scheduling Capabilities of PM Software

http://www.fplotnick.com/constructioncpm/2017Presentations/THU33-ResourceConstrainedScheduling.pdf

Comparison of resource levelling is from the slide 24

Conclusion was the same: Spider Project was the best.

Rafael Davila
User offline. Last seen 13 hours 38 min ago. Offline
Joined: 1 Mar 2004
Posts: 5229

I was told that when sample projects were leveled by Asta resources assigned to activities were scheduled to work at different time. As the result activity durations become longer. It is like the car is used separately from its driver, the car was assigned without a driver and the driver was used without his car.

My contact did not find Asta option to assign resources together as the team (crew).

Mike, is it true? What did they miss? I don't believe that Asta is not capable of modeling simple team work.

Rafael Davila
User offline. Last seen 13 hours 38 min ago. Offline
Joined: 1 Mar 2004
Posts: 5229

Patrick

You said - The problem with most software is the assumption the duration is fixed - it is not, the quantity of work is fixed, the duration is a function of the resources allocated (but the relationship is complex).

Spider project can tackle the issue very easily with the use of production type activities whose duration is determined by the allocated resources, their production rates and the total volume of work.  

  1. Production can be done by individual teams working on separate shifts and can be done by a combination of several teams.
  2. Teams selection can be done among a pool of several teams that share common skills with or without same production rates.
  3. For some basic software, skills and materials are just an ID label different to the functionality Spider associate with these names. 
  4. As things change with activity progress the original resource assignments can cahnge, the team you use for the same activity on winter might be different to the team you use on summer and their production rates per unit time can be substantially different. Usually these are standard crews with different labor and equipment composition each with different output per day.
  5. The team you use at night migh be less productive than the one working during the day.

I do not know of software other than Spider Project that can model activity duration as a function of the assigned resources productivity, it is a no brainer for us but not available everywhere. 

I was reluctant to go with very simple models but it was agreed to go low because most commercially available software cannot dream about modeling duration as a function of the resources allocated, a function that in many cases is linear but at other times it is not and for this we use skills for whole crews.

So we agreed to compare resource leveling capabilities without considering the other resource assignment variables Spider can handle very easily.

Maybe we can agree that software unable to model activity duration as a function of assigned resources is not good enough.

Best Regards

Rafael

Rafael Davila
User offline. Last seen 13 hours 38 min ago. Offline
Joined: 1 Mar 2004
Posts: 5229

Regarding the schedule GENETIC01, when I download it, then Spider complaints that file is corrupted and I can't open it. Can you reload it?

I downloaded myself the file and it works so it should not make any difference if I upload the same thing, I suspect it might be due to the Spider version you are using.  Try downloading again, try updating your Spider version and let me know, even if a demo version try updating it.

Thank you very much for testing Project Easy and letting me know, I do not have P6 nor MSP.

Best Regards,

Rafael

Patrick,

it looks like we live and work at different markets.

First problem that you mentioned does not exist and is not directly linked with resource leveling capabilities. 

And every owner wants to save time and money by proper use of available resources.

Those who are not interested in this loose time and money of somebody else.

Patrick Weaver
User offline. Last seen 6 days 7 hours ago. Offline
Joined: 18 Jan 2001
Posts: 373
Groups: None

The problem with most software is the assumption the duration is fixed - it is not, the quantity of work is fixed, the duration is a function of the resources allocated (but the relationship is complex). No one in the general commercial market is looking at optimisation, see the presentations at: http://www.mosaicprojects.com.au/Resources_Papers_152.html

Evgeny Z.
User offline. Last seen 38 weeks 5 days ago. Offline
Joined: 13 Jan 2008
Posts: 442
Groups: None

Rafael,

for the project Easy both MSP2010  and  P6 v15.1 come with the same result as Spider (12 days)

So, may be it is too easy.

 photo Easy_mpp_zpsbifybbxa.png  photo Easy_PrimaveraP6v15.1_zpsqjkh7w2e.png

 

Regarding the schedule GENETIC01, when I download it, then Spider complaints that file is corrupted and I can't open it. Can you reload it?

What I am still looking for is the following:

1) may be more of very simple intuitively-understood schedule, which can be assigned a pass/fail criteria, rather then duration.  At the same time it shall not be that simple, that every tool would come with pass result.

So, if the solution is optimal - pass, otherwise - fail.

The best example I know so far is:

 photo VladimirLiberzon_leveling test MS Project 2007_before leveling_zpsjjnmovgu.jpg

2) Example of some real construction (or any other) project, which can be used for resource leveling comparison. It would be very good if such schedule is in *.mpp format and in general publicly available. So, that people cannot say, that we specially crafted such schedule to give an advantage to one or the other tool. The reason  I want to have such schedule is because we shall not be using only academic-created schedules. At the end of the day we are practical people.

Regards.

Rafael Davila
User offline. Last seen 13 hours 38 min ago. Offline
Joined: 1 Mar 2004
Posts: 5229

Evgeny,

I want to take my time to congratulate you for your effort.  For some mysterious reason software comparison has been absent in recent years.

I am curious about your results for Project Easy with MSP and P6.  Is it possible you are still looking for MSP and P6 prioritization rules?  I thought you was to use default rules. I only have and use Spider Project.

Best Regards,

Rafael

Rafael Davila
User offline. Last seen 13 hours 38 min ago. Offline
Joined: 1 Mar 2004
Posts: 5229

Johannes,

Thank you.

Best Regards,

Rafael

Johannes Vandenberg
User offline. Last seen 8 weeks 5 days ago. Offline
Joined: 21 Jan 2010
Posts: 234

Hi Raphael and Evgeny

I have ran the the leveling test GENETIC01 on P6 R8.4. I have 32 days without leveling and 49 days with leveling. The standard leveling option, activity leveling to "normal", was the shortest. I could not find an leveling criteria that could improve the 49 days.

<span>New Gallery 2015/4/11</span><span>originaldate</span><span> 4/11/2015 8:52:04 PM</span><span>width</span><span> 1152</span><span>height</span><span> 815</span>

regards Johannes

Rafael Davila
User offline. Last seen 13 hours 38 min ago. Offline
Joined: 1 Mar 2004
Posts: 5229

You can try Project EASY.

EASY photo EASY_zpsz1a10jiw.jpg
Rafael Davila
User offline. Last seen 13 hours 38 min ago. Offline
Joined: 1 Mar 2004
Posts: 5229

Thank Evgeny,

I cannot wait to see how P6, MSP and Spider results for GENETIC01 compare.  A small job, not very complicated but good as a jump start.

Best Regards,

Rafael

Evgeny Z.
User offline. Last seen 38 weeks 5 days ago. Offline
Joined: 13 Jan 2008
Posts: 442
Groups: None

Rafael,

thank you very much for your examples. I will look at them. Meanwhile I have downloaded Primavera P6 v15.1, which is apparently very nicely made available for download for trial purposes by Oracle.

See updated table with Primavera results below. I will also put the digest of results in the 1st post of this thread.So, in the same way as in my tests from 2 years ago MSP is actually better, than Primavera, but worse, than Spider.

Just for record:

 

Settings used for Primavera scheduling: I used default settings (Activity levelling priority). However since there are no actively priorities, assigned to activities, this effectively means, that no activity has a priority over any other activity. In this way I am asking Primavera exactly what I asked MS Project and Spider: I do not care about activity priorities, I do not want to influence you, just deliver me the shortest schedule you can. I also verified, that if I remove this “Activity Levelling Priority”, this does not change anything, primavera produces the same schedule.

 

I can imagine, that by playing around with many possible priorities settings, available at Primavera it is possible to achieve a shorted schedule, but then probably these priority setting would be different for every schedule. However this would be similar to manual scheduling in my view.  

 photo PrimaveraLevellingSettings_zps79a1fevc.png

  

Microsoft Project 2010 SP2

Spider Project Optimization Plus

Primavera P5 v 15.1

Comparison criteria

Duration not resource leveled

Duration resource leveled MSP

Ratio leveled / not leveled MSP 2010 SP2

Duration resource leveled Spider

Ratio Leveled / not leveled Spider

Spider Project  relative to MSP 2010 SP2

Resource leveled Primavera P6 v. 15.1

Ratio leveled / not leveled Primavera P6 V15.1

Primavera P6 V15.1 relative to MSP 2010 SP2

RCPSP   j3013_2

32

75

2,34

68

2,13

91%

91

2,84

121%

RCPSP   j3013_8

48

134

2,79

114

2,38

85%

140

2,92

104%

RCPSP   j3029_4

54

128

2,37

109

2,02

85%

128

2,37

100%

RCPSP   j3029_6

43

103

2,40

98

2,28

95%

108

2,51

105%

RCPSP   j3029_9

52

121

2,33

112

2,15

93%

132

2,54

109%

RCPSP   j6013_10

63

143

2,27

133

2,11

93%

163

2,59

114%

RCPSP   j6029_6

77

184

2,39

174

2,26

95%

219

2,84

119%

RCPSP   j6045_10

61

141

2,31

126

2,07

89%

148

2,43

105%

RCPSP   j6045_2

78

181

2,32

159

2,04

88%

174

2,23

96%

RCPSP   j6045_6

80

183

2,29

157

1,96

86%

187

2,34

102%

RCPSP   j9013_8

59

142

2,41

130

2,20

92%

154

2,61

108%

RCPSP   j9025_7

77

164

2,13

140

1,82

85%

172

2,23

105%

RCPSP   j9041_4

85

191

2,25

170

2,00

89%

199

2,34

104%

RCPSP   j9045_10

88

204

2,32

183

2,08

90%

218

2,48

107%

RCPSP   j9045_5

98

215

2,19

196

2,00

91%

232

2,37

108%

RCPSP   j12016_1

71

253

3,56

224

3,15

89%

266

3,75

105%

RCPSP   j12016_2

85

303

3,56

261

3,07

86%

304

3,58

100%

RCPSP   j12031_10

90

312

3,47

265

2,94

85%

321

3,57

103%

RCPSP   j12051_2

92

304

3,30

259

2,82

85%

327

3,55

108%

RCPSP   j12051_8

85

297

3,49

236

2,78

79%

287

3,38

97%

RCPSP   j12056_3

93

316

3,40

272

2,92

86%

339

3,65

107%

RCPSP   j12056_4

86

294

3,42

253

2,94

86%

285

3,31

97%

RCPSP   j12056_7

117

389

3,32

326

2,79

84%

398

3,40

102%

RCPSP   j12056_8

99

392

3,96

329

3,32

84%

396

4,00

101%

RCPSP   j12056_9

101

368

3,64

330

3,27

90%

390

3,86

106%

          

Average

  

2,81

 

2,46

88%

 

2,95

105%

 

 photo PM_SW_tools_comp_2015.04.09_zpsj4f2l8nu.png

Stephen Devaux
User offline. Last seen 17 weeks 2 days ago. Offline
Joined: 23 Mar 2005
Posts: 667

Ok, Raphael. So how much are those six days that are the difference between MSP and Spider worth? $60,000? $600,000? 18 human lives?

Y'know, it's bad enough that people die because it akesmore time than expected to develop a new drug. or build a new hospital, or respond to a disaster or dig a potable water well. But when the delay is caused by a computer algorithm, that is truly sad.

Fraternally in project management,

Steve the Bajan 

Rafael Davila
User offline. Last seen 13 hours 38 min ago. Offline
Joined: 1 Mar 2004
Posts: 5229

I got a duration of 43 days at a single click, no need to mess with prioritization rules.

GENO5 photo 05_zpse79sbrrz.jpg

http://www.mediafire.com/download/o4p6vludc93l3pq/RD603.001.sprj

Rafael Davila
User offline. Last seen 13 hours 38 min ago. Offline
Joined: 1 Mar 2004
Posts: 5229

Goinjg back to Basics, my next test schedule I call

GENETIC01

GEN02 photo 02_zpsa87pdg8r.jpgGEN01 photo 01_zpskz3jccuj.jpgGEN03 photo 03_zpskjvotquz.jpg

GEN04 photo 04_zpswnuq7dw4.jpg

What is the best you can you get with MSP, P6 and Spider?

Best Regards,

Rafael

Rafael Davila
User offline. Last seen 13 hours 38 min ago. Offline
Joined: 1 Mar 2004
Posts: 5229

Vladimir,

I agree, but wanted to point out resource leveling is not just about renewable resources by challenging others daring to try the impossible, to try modeling resource leveling of other resource types using renewable resources.

I believe some people do not even know the difference between consumable and renewable resources and both matter.  Even fewer realize that spatial resources can be modeled via consumable resources, most do not know the concept and believe they will never have a need to model spatial resources.  

A year ago I was involved on a claim that was finally agreed using common sense but it was a perfect case for modeling spatial resources.  It was about a vocational school and because of unforeseen underground conditions a building was relocated causing all sequence to be changed.  Scheduling was tied to prefabricated elevated slab forms that must remain on each building until concrete work is finished, these are spatial resources.  Changing [justifying] the logic was a nightmare while spatial resource modeling takes care of it at a single click of the mouse, well two clicks.  It happened twice that a change in building sequence was in order.

Focusing only on renewable resources is myopic.

Best Regards

Rafael

Rafael,

at https://www.linkedin.com/groups/Resource-Leveling-Very-Interesting-Thread-45942.S.5990883937872809985?view=&gid=45942&item=5990883937872809985&type=member&commentID=discussion%3A5990883937872809985%3Agroup%3A45942&trk=hb_ntf_COMMENTED_ON_GROUP_DISCUSSION_YOU_FOLLOWED#commentID_discussion%3A5990883937872809985%3Agroup%3A45942 Paul Harris suggested to consider different leveling options comparing software leveling capabilities. It could be an interesting but separate discussion. Paul listed some options that may be considered and his list is not full.

But it is still interesting to compare resource consrained schedules created by different packages when leveling setting and requirements are the same. Evgeny suggested to try different software for most simple settings (FS links, no splittings, no materials, etc.), the settings that every scheduling software shall support. Later we may discuss other leveling requirements and how different packages meet them. Creating a full list of potential requirements is an interesting task.

The package that produces better (shorter) schedules can save a lot to its users. Let's start from the most simple case and then consider other leveling options, each one in the separate discussion.

Rafael Davila
User offline. Last seen 13 hours 38 min ago. Offline
Joined: 1 Mar 2004
Posts: 5229

Because of the delivery schedule you cannot do all activities starting from day one, at some point you will run out of brick resources so after Adjust activity schedules to remove overallocations the duration will be 29 days.

Feel free to make an excel table and calculate bricks availability day by day.  BTW in this case allowing splitting does not makes any difference although frequently it does. 

What did you got with MSP after resource leveling?  

XXBRICKS photo 01_zpsn34sfehd.jpg

Hi Steve,

Spider Project calculates the costs of delays. But I agree with you that this test shall focus on #1. Cost of time deserves separate discussion.

Best Regards,

Vladimir

Rafael Davila
User offline. Last seen 13 hours 38 min ago. Offline
Joined: 1 Mar 2004
Posts: 5229

I suppose we can agree ancient Egyptians would have to tackle resource leveling issues so the following problem might have been a regular routine.

Pyramids photo Pyramids_zpsmachp2le.jpg

Depending on the Pharaoh Dynasty bricks installation would not be allowed to be split, more advanced dynasties would realize it makes sense.

How long it would take to finish the three brick installation activities using MSP, during the two dynasties? Nothing new, perhaps ancient, they just had the abacus and could figure it out.

I would limit my use of such benchmark schedules to develop portions of overall resource leveling and then would look to see how it fits within the overall tool, but in modern times, after the abacus they are not good enough by themselves to evaluate the overall performance of resource modeling tools.

Stephen Devaux
User offline. Last seen 17 weeks 2 days ago. Offline
Joined: 23 Mar 2005
Posts: 667

Hi, all.

I tend to agree with Vladimir that it's good that the problems are relatively simple. It's also good that they are relatively "standard" for testing algorithms. If a s/w package simply doesn't have sufficient functionality to digest the schedule, that's an important shortcoming -- but it's also a separate issue. I think that the "cleaner" you can keep this discussion by limiting it to a standard set of problems that any s/w should be able to do, the better.

When it comes to resource leveling, there are IMO three things that it's important for any s/w to be able to do:

  1. Adjust activity schedules to remove overallocations. The judgement about how well it does this, in terms of delaying completion, is very important and what this test would be about.
  2. Turn the time units of the delay into value/cost units, i.e., the impact of the delay on the ROI/NPV/EMV for measure ofdrag and drag cost of thedelaying factors. This would be very simple with the (a) the computation of resource availability drag (RAD) and (b) inclusion of an input field that allows the user to designate how much varying the completon date wil limpact the project's expected value. But does any s/w allow this? Vladimir, does even Spider allow this? Because it seems to me that including the functionality to monetize the cost of the lost time is the ultimate format for showing the importance of a superior leveling algorithm. That's why I urged Evgeny to include a column with a dollar amount based on a $1,000/day in time cost (or $10,000/day or whatever you like).
  3. Finally, the s/w must include the ability to EASILY AND QUICKLY tie the delays, and the time cost for each delay, to the specific factor causing it. If I had a smallish project (say, 500 activities), a s/w package whose leveling algorithm produces an inferior (i.e., longer) schedule BUT THAT EASILY AND QUICKLY showed the delaying factors and the cost of each might be preferable to me to one that initially produced a shorter schedule but didn't aid much with the analysis and quantification. 

For purposes of this test and any paper based on it, I think it is best to focus just on #1 -- but I also think it would be good to include data showing the potential impact on ROI, just for those ironheads who don't get the point. ("Oh, but it's good if the s/w gives me more time! I'll probably need topush back the deadline anyway!")

I'm quite excited about this.

Fraternally in project management,

Steve the Bajan 

Evgeny Z.
User offline. Last seen 38 weeks 5 days ago. Offline
Joined: 13 Jan 2008
Posts: 442
Groups: None

Rafael,

thanks. I will use it. Gargantuan 02  is a variant of what I called a ""Vladimir Liberzon Leveling test"" in my previouse test report:

I think for such simple tests we can have just pass-fail creteria.

Regards.

Rafael Davila
User offline. Last seen 13 hours 38 min ago. Offline
Joined: 1 Mar 2004
Posts: 5229

Evgeny,

Gargantuan 02 would be:

Task A - Evgeny - 10 days; 
Task B - Vlad - 10 days; 
Task C - Evgeny - 5 days; 
Task D - Vlad - 10 days;

Tasks A and C will have "Start" as predecessor. 
Task B will have A as predecessor. 
Task D will have C as predecessor. 
Tasks B and D will have "Finish" as successor.

Best Regards,

Rafael

Rafael Davila
User offline. Last seen 13 hours 38 min ago. Offline
Joined: 1 Mar 2004
Posts: 5229

Vladimir,

I am talking about simple things such as SS and FF relationships.  Does it meets the criteria or shall also be banned from such comparison just because MSP is so primitive you have to tweak the schedule?  I am not sure the tweaking to model SS and FF links to same activity using additional milestone dummies will yield same resource leveling within MSP, not sure about SS and FF relationships when not tied to the same activity, it happens.  What is complex about FF and SS relationships?

What is complex about the gargantuan I showed before?  It only have FS relationships, a single resource with availability of five across the board. Does it meets the criteria?

The results depend on how challenging the set is and based on the gargantuan it looks like the selection is not very challenging. 

Best Regards

Rafael

Evgeny Z.
User offline. Last seen 38 weeks 5 days ago. Offline
Joined: 13 Jan 2008
Posts: 442
Groups: None

Rafael,

I agree, that these are the most basic activity relations, one can imagine. But advantage of it that it shall be able to use it to compare any Project Management tool out there as any tool shall be able to support these types of relations. Hence there is no up to date comparison available even for such simple model, this is a good test to start with in my view.

Any way, as I mentioned, everybody  is welcome to post here any other comparisons.

May be you can post here comparison of resource levelling for some typical constriction project? (I know there can be some data protection issues here)

By the way, there are some  other problems available from the same site, but I didn’t look at them yet:

http://www.om-db.wi.tum.de/psplib/data.html

  • Resource-Constrained Project Scheduling Problem with Minimal and Maximal Time Lags (RCPSP/max)
  • Multi-Mode Resource Constrained Project Scheduling Problem (MRCPSP)
  • Multi-Mode Resource Constrained Project Scheduling Problem with Minimal and Maximal Time Lags (MRCPSP/max)
  • Resource Investment Problem with Minimal and Maximal Time Lags (RIP/max) 

Regards.

Evgeny

Rafael,.

these sets of projects are well known benchmarking tool. It is good that they do not include anything complex because any tool can be used with these simple examples. In other case it would be hard to compare resource leveling capabilities of different packages just because most of them do not include advanced features.

Just look at construction project in Spider Demo: it models working in two shifts, requires material leveling. I don't know other packages that include these features and so Spider Project schedule cannot be compared with the schedules created by other packages just because they cannot create any feasible schedule for this project. And there are other features like variable resource assignment, cost leveling, etc. that have no analogues in other tools.

So it is reasonable not to make a set of test projects too complex and include only those features that every PM software shall definitely support. This test does not try to compare functionality but just resource leveling algorithms. It would be nice if PP participants will try the packages they are using with the same projects adding data to Evgeny's table.

Rafael Davila
User offline. Last seen 13 hours 38 min ago. Offline
Joined: 1 Mar 2004
Posts: 5229

I am starting my own set and decided not to rule out schedules good for the Abacus, my first I call it the gargantuan.  See the following figure.

gargantuan photo gargantuan_zpsdv3zstyy.jpg

With Spider I can get out of the box the 7 days duration, MSP answer is 9 days or 28.6% longer than Spider.

Rafael Davila
User offline. Last seen 13 hours 38 min ago. Offline
Joined: 1 Mar 2004
Posts: 5229

Evgeny,

I downloaded J120 folder but it looks like the set is quite primitive, of the times where CPM relationships were limited to SF relationships, just take a look at the links table, also multiple relationships between two activities [usually SS and FF] are lacking, just look at the predecessors column an look for activities with multiple reference to same activity code.

These testing sets were developed decades ago for the limited functionality of CPM of the 60's, to me they are a joke, in order for the comparison to be valid as per today tools it must consider more complicated models, more complicated relationships. Good only to compare Ford Pinto models, a red Pinto versus a green Pinto.

Unfortunately some lazy academics are still stuck with decade old models and are not taking the time to develop something more up to date. Other simply do not want to rule out MSP limited functionality in fear of loosing some audience, for them it is all about publish or perish. 

Do you have something better, more up to date?

Best Regards,

Rafael

Stephen Devaux
User offline. Last seen 17 weeks 2 days ago. Offline
Joined: 23 Mar 2005
Posts: 667

Hi, Evgeny.

I think it will be very important to get similar data on as many s/w packages as possible --Primavera, Asta, PlanView, Safran, etc.

Since I am not really knowledgable of any of them, I cannot help. But good luck.

Fraternally in project management,

Steve the Bajan

Evgeny Z.
User offline. Last seen 38 weeks 5 days ago. Offline
Joined: 13 Jan 2008
Posts: 442
Groups: None

Stephen,

Thank you very much for your feedback. I really appreciate it! I definitely want to do more thorough work on this subject, but I really hope, that I am not the only contributor here. In fact the value of the work will be much higher, if there are several contributors and reviewers. That is why we all like Wikipedia, because it is contributed by many people and anybody can shout, if somebody else whites nonsense.

I never thought of publishing my humble work in a professional journal, but if you believe it is worth doing this, once more data is available, I will definitely consider it.

Any way, meanwhile I have decided to slightly change the choice of problems I have used, as  I noticed, that j60 and j90 series are not represented. Therefore I have made the following adjustment to my selection:

  • 10 most interesting from j120 series
  • 5 most interesting from j90 series
  • 5 most interesting from   j60 series
  • 5 most interesting from j30 series

I think I am going to use these problems going forward for comparisons of different tools. They are all available here for download as a single archive in the form of Microsoft Project 2010 .mpp and .xml files.

 

So, now the table looks like this

  

Microsoft Project 2010 SP2

Spider Project Optimization Plus

Comparison criteria

Duration not resource leveled

Duration resource leveled MSP

Ratio leveled / not leveled MSP 2010 SP2

Duration resource leveled Spider

Ratio Leveled / not leveled Spider

RCPSP   j12056_8

99

392

3,96

329

3,32

RCPSP   j12056_9

101

368

3,64

330

3,27

RCPSP   j12016_2

85

303

3,56

261

3,07

RCPSP   j12016_1

71

253

3,56

224

3,15

RCPSP   j12051_8

85

297

3,49

236

2,78

RCPSP   j12031_10

90

312

3,47

265

2,94

RCPSP   j12056_4

86

294

3,42

253

2,94

RCPSP   j12056_3

93

316

3,40

272

2,92

RCPSP   j12056_7

117

389

3,32

326

2,79

RCPSP   j12051_2

92

304

3,30

259

2,82

RCPSP   j3013_8

48

134

2,79

114

2,38

RCPSP   j9013_8

59

142

2,41

130

2,20

RCPSP   j3029_6

43

103

2,40

98

2,28

RCPSP   j6029_6

77

184

2,39

174

2,26

RCPSP   j3029_4

54

128

2,37

109

2,02

RCPSP   j3013_2

32

75

2,34

68

2,13

RCPSP   j3029_9

52

121

2,33

112

2,15

RCPSP   j6045_2

78

181

2,32

159

2,04

RCPSP   j9045_10

88

204

2,32

183

2,08

RCPSP   j6045_10

61

141

2,31

126

2,07

RCPSP   j6045_6

80

183

2,29

157

1,96

RCPSP   j6013_10

63

143

2,27

133

2,11

RCPSP   j9041_4

85

191

2,25

170

2,00

RCPSP   j9045_5

98

215

2,19

196

2,00

RCPSP   j9025_7

77

164

2,13

140

1,82

 

Since the problems selection has changed, the average schedule extension due levelling has also slightly changed.

Now average extension of schedule caused by Microsoft Project is 181% (2,81 times) and the one caused  by Spider Project is 146% (2,46 times).  Spider is still better on every single test.

I also found that the table with 25 entries is a bit difficult to grasp, therefore I decided to also show this graphically. Namely I decided to compare all tools with MS Project 2010 SP2 (no specific reason to chose MSP 2010, except that it is probably the most widely used software nowadays and it is installed on my PC at the moment).  Any way, this does not really matter what to chose as a baseline, as it is relative result, which is important.

So, on the below graph the lower the result lays, the better it is.  Line y=x belongs to Microsoft Project 2010 SP2 (as it is used as a baseline). Thus any result, which lays below the line y=x is better, than MS Project 2010 SP2, any result which lays above that line is worse than MS Project 2010 SP2 (nothing at the moment) photo leveling tools_zps2n3djxfh.png

Stephen Devaux
User offline. Last seen 17 weeks 2 days ago. Offline
Joined: 23 Mar 2005
Posts: 667

Evgeny, this is awesome!

My only issue is that, as wonderful as it is that you are doing it and publishing your results here, I'd love to see you do a more thorough study, testing more packages, and posting your results in Harvard Business Review, Project Management Journal, or some equally prestigious international publication.

I have some further comments:

  • I would love to see you include an additional column for delay cost, using a standard of $1,000 (or euros or roubles or rupees or yen or whatever) for every day the project is longer than the unleveled duration. (In Total Project Control (TPC) terms, this would be the resource availability drag cost (RAD Cost) for the resource insufficiencies causing the delay (and thus value reduction). That would seem to me to make the point of the study even more dramatic. ($1,000/day seems like a reasonable and minimal round number.)  
  • One of the important things that it has always seemed to me that a resource-leveling algorithm should do is to list, in descending order, the drag cost of the resource bottlenecks, i.e., how much each is costing. I would be curious if any leveler has that functionality, because your study would make the value of such a computation VERY clear!
  • If you are not going to do try to publish the results of this study in a prestigious journal (and I really,really hope you will consider this), would you be willing at least to publish it as guest blog at my TotalProjectControl website? :-)

Under any circumstances, I am going to a mention and link to this tread on PM discussion groups on LinkedIn.

Thanks again, Evgeny.

Fraternally in project management,

Steve the Bajan

Evgeny Z.
User offline. Last seen 38 weeks 5 days ago. Offline
Joined: 13 Jan 2008
Posts: 442
Groups: None

Ok, and since I started this the 1st comparison results would come from me.

I have used the Resource-Constrained Project Scheduling Problem (RCPSP) library, which is the part of the PROJECT SCHEDULING PROBLEM LIBRARY – PSPLIB, published by the   Prof. Dr. Rainer Kolisch from the Technical University of Munchen and by Arno Sprecher from Christian-Albrechts-Universität zu Kiel.

Briefly about RCPSP:

It contains a set of problems for resource levelling. Namely it has

·         480 problems with 30 tasks and 4 resources (j30 - series)

·         480 problems with 60 tasks and 4 resources (j60 series)

·         480 problems with 90 tasks and 4 resources (j90 series)

·         600 problems with 120 tasks and 4 resources (j120 series)

So, 2040 problems in total.

Authors of this library describe the purpose of it as following

<<

We present a set of benchmark instances for the evaluation of solution procedures for single- and multi-mode resource-constrained project scheduling problems. ... This should be a valuable and driving source for further improvements in the area of project type scheduling.

>>

In my view one can hardly find any more independent test criteria.

What I have done is following:

I have converted all of these 2040 problems in the from of Microsoft Project files. One can download them all here. After that I have run MS Project 2010 for all of these 2040 problems and selected 15 most interesting of them. The “most interesting” for me were the ones, which gave the biggest ratio of schedule duration increase due to levelling (duration resource levelled / duration not resource levelled). They all happen to be from J120 series.  

Since I want my test to be easily repeatable by anybody, I also then selected 5 most interesting of J30 series. The reason I have done it because one of the tools I used for comparison was Spider Project, which provides a fully-functional demo version, which is limited to 40 tasks. So, with j30 series anybody can repeat my tests.

After that for each of these selected 20 problems I have also run resource constrained schedule in Spider Project.

I have disallowed splitting of tasks by resource leveling in both tools to ensure level playing field.

Resource Levelling Mode:

  • In Ms Project – standard
  • In Spider – Optimisation Plus

So, by doing this I have compared 2 software tools available to me using the most independent test criteria I could find.  

So I am posting results here.

  

Microsoft Project 2010 SP2

Spider Project Optimization Plus

Comparison criteria

Duration not resource leveled

Duration resource leveled MSP

Ratio leveled / not leveled MSP

Duration resource leveled Spider

Ratio Leveled / not leveled Spider

RCPSP   j3013_2

32

75

2,34

68

2,13

RCPSP   j3013_8

48

134

2,79

114

2,38

RCPSP   j3029_4

54

128

2,37

109

2,02

RCPSP   j3029_6

43

103

2,40

98

2,28

RCPSP   j3029_9

52

121

2,33

112

2,15

RCPSP   j12011_6

91

297

3,26

238

2,62

RCPSP   j12016_1

71

253

3,56

224

3,15

RCPSP   j12016_2

85

303

3,56

261

3,07

RCPSP   j12016_6

79

261

3,30

228

2,89

RCPSP   j12031_10

90

312

3,47

265

2,94

RCPSP   j12051_2

92

304

3,30

259

2,82

RCPSP   j12051_8

85

297

3,49

236

2,78

RCPSP   j12056_1

95

313

3,29

272

2,86

RCPSP   j12056_10

102

333

3,26

292

2,86

RCPSP   j12056_3

93

316

3,40

272

2,92

RCPSP   j12056_4

86

294

3,42

253

2,94

RCPSP   j12056_5

117

385

3,29

313

2,68

RCPSP   j12056_7

117

389

3,32

326

2,79

RCPSP   j12056_8

99

392

3,96

329

3,32

RCPSP   j12056_9

101

368

3,64

330

3,27

Conclusion

The idea of my post is that I provide information and everybody can draw his own conclusion. But the simple matter of fact is the following:

  • On every single of selected problems Spider Project came up with shorter schedule, then MS Project 2010.
  • On the selected set of problems the average duration extension due to resource levelling of  MS Project was 219% (3,19 times) and of Spider Project was 174% (2,74 times).  Or in another words, for the selected set of problems the resource-leveled Spider Project schedule would be in average 14% shorter, then Microsoft Project one.

Regards.

Evgeny